Add group snapshot controller metrics

[yati1998]

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR introduces new operation like CreateGroupsnapshot, DeleteGroupsnapshots, etc to the existing metrics

Testing

• enable feature gate for groupvolumesnapshot
• pass --http-endpoint to include the port to expose the metrics
• curl using the port and metrics path

TODO

Add output for deleteGroupSnapshot operation

Which issue(s) this PR fixes:

Fixes # 1081

Special notes for your reviewer: n/a Does this PR introduce a user-facing change?:

it exposes the metrics for volumegroupsnapshot operations on passing flags like --http-endpoint and --metrics-path to the snapshot controller. These are the command line arguments for the users to enable metrics and start the sever.
The metric `snapshot_controller_operation_total_seconds` has been added for the volumegroupsnapshot operations:
1. CreateGroupSnapshot
2. DeleteGroupSnapshot
3. CreateSnapshotAndReady

[k8s-ci-robot]

Hi @yati1998. Thanks for your PR.

I'm waiting for a kubernetes-csi member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

[yati1998]

# HELP process_start_time_seconds [ALPHA] Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.71827805549e+09
# HELP snapshot_controller_operation_total_seconds [ALPHA] Total number of seconds spent by the controller on an operation
# TYPE snapshot_controller_operation_total_seconds histogram
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="dynamic",le="0.1"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="dynamic",le="0.25"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="dynamic",le="0.5"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="dynamic",le="1"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="dynamic",le="2.5"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="dynamic",le="5"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="dynamic",le="10"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="dynamic",le="15"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="dynamic",le="30"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="dynamic",le="60"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="dynamic",le="120"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="dynamic",le="300"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="dynamic",le="600"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="dynamic",le="+Inf"} 1
snapshot_controller_operation_total_seconds_sum{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="dynamic"} 800.652157918
snapshot_controller_operation_total_seconds_count{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="dynamic"} 1
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="dynamic",le="0.1"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="dynamic",le="0.25"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="dynamic",le="0.5"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="dynamic",le="1"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="dynamic",le="2.5"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="dynamic",le="5"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="dynamic",le="10"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="dynamic",le="15"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="dynamic",le="30"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="dynamic",le="60"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="dynamic",le="120"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="dynamic",le="300"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="dynamic",le="600"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="dynamic",le="+Inf"} 1
snapshot_controller_operation_total_seconds_sum{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="dynamic"} 800.652183274
snapshot_controller_operation_total_seconds_count{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="dynamic"} 1
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="0.1"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="0.25"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="0.5"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="1"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="2.5"} 1
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="5"} 1
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="10"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="15"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="30"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="60"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="120"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="300"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="600"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="+Inf"} 2
snapshot_controller_operation_total_seconds_sum{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshot",operation_status="success",snapshot_type="pre-provisioned"} 6.751690228
snapshot_controller_operation_total_seconds_count{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshot",operation_status="success",snapshot_type="pre-provisioned"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="0.1"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="0.25"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="0.5"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="1"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="2.5"} 1
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="5"} 1
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="10"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="15"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="30"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="60"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="120"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="300"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="600"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="+Inf"} 2
snapshot_controller_operation_total_seconds_sum{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned"} 6.752143597
snapshot_controller_operation_total_seconds_count{driver_name="hostpath.csi.k8s.io",operation_name="CreateSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned"} 2
# HELP snapshot_controller_operations_in_flight [ALPHA] Total number of operations in flight
# TYPE snapshot_controller_operations_in_flight gauge
snapshot_controller_operations_in_flight 3

[nixpanic]

@yati1998 please write something in the PR descriptions release-note field.

/ok-to-test

[xing-yang]

Can you also add test for statically provisioned group snapshots?

[xing-yang]

Can you also add test results for delete group snapshot?

[xing-yang]

In the release note, please also add exactly what metric is added.

[yati1998]

Can you also add test results for delete group snapshot?

unfortunately that's not working, cause deletion is getting suck as mentioned earlier. Will try to resolve that issue first.

[yati1998]

In the release note, please also add exactly what metric is added.

no new metric is added, we can use same metric to get the groupvolumesnapshot as well.

[xing-yang]

In the release note, please also add exactly what metric is added.

no new metric is added, we can use same metric to get the groupvolumesnapshot as well.

Can you add the name of the metric in the release note?

[yati1998]

In the release note, please also add exactly what metric is added.

no new metric is added, we can use same metric to get the groupvolumesnapshot as well.

Can you add the name of the metric in the release note?

done

[xing-yang]

Please update the testing results.

[xing-yang]

Please update test results with pre-provisioned group snapshots and add a "TODO" in the PR description for delete group snapshot.

[yati1998]

# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.71845164359e+09
# HELP snapshot_controller_operation_total_seconds [ALPHA] Total number of seconds spent by the controller on an operation
# TYPE snapshot_controller_operation_total_seconds histogram
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="0.1"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="0.25"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="0.5"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="1"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="2.5"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="5"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="10"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="15"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="30"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="60"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="120"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="300"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="600"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="pre-provisioned",le="+Inf"} 2
snapshot_controller_operation_total_seconds_sum{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="pre-provisioned"} 4.770233558
snapshot_controller_operation_total_seconds_count{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshot",operation_status="success",snapshot_type="pre-provisioned"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="0.1"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="0.25"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="0.5"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="1"} 0
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="2.5"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="5"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="10"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="15"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="30"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="60"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="120"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="300"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="600"} 2
snapshot_controller_operation_total_seconds_bucket{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned",le="+Inf"} 2
snapshot_controller_operation_total_seconds_sum{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned"} 4.780226055
snapshot_controller_operation_total_seconds_count{driver_name="hostpath.csi.k8s.io",operation_name="CreateGroupSnapshotAndReady",operation_status="success",snapshot_type="pre-provisioned"} 2
# HELP snapshot_controller_operations_in_flight [ALPHA] Total number of operations in flight
# TYPE snapshot_controller_operations_in_flight gauge
snapshot_controller_operations_in_flight 2

[xing-yang]

/lgtm /approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: xing-yang, yati1998

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/kubernetes-csi/external-snapshotter/blob/master/OWNERS)~~ [xing-yang] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment