Use client-go package for leaderelection to add cleanup logic after leader change

[adikul30]

What this PR does / why we need it:

https://github.com/kubernetes-csi/csi-lib-utils/blob/master/leaderelection/leader_election.go#L187 doesn't allow overriding OnStoppedLeading callback. This means we can't add any code to perform session cleanup which leads to session leak.

client-go leaderelection package provides similar functionality while allowing us to define logic for OnStoppedLeading and other callbacks.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Testing done:

I0709 23:20:59.062686       1 leaderelection.go:245] attempting to acquire leader lease vmware-system-csi/vsphere-syncer...
2024-07-09T23:20:59.109Z    INFO    syncer/main.go:218  new leader detected, current leader: vsphere-csi-controller-845c4d5bf6-smrk9    {"TraceId": "517bb231-f548-4b2a-b675-443bde166d86"}
2024-07-09T23:21:33.693Z    INFO    syncer/main.go:218  new leader detected, current leader: vsphere-csi-controller-845c4d5bf6-2dc9n    {"TraceId": "517bb231-f548-4b2a-b675-443bde166d86"}
I0709 23:21:33.697716       1 leaderelection.go:255] successfully acquired lease vmware-system-csi/vsphere-syncer
2024-07-09T23:21:33.699Z    INFO    syncer/main.go:209  became leader, starting {"TraceId": "517bb231-f548-4b2a-b675-443bde166d86"}

---

root@k8s-control-574-1720547892:~# k create -f scpodpvc.yaml
storageclass.storage.k8s.io/example-vanilla-rwo-filesystem-sc created
persistentvolumeclaim/example-vanilla-rwo-pvc created
pod/example-vanilla-block-pod created

root@k8s-control-574-1720547892:~# k get pvc
NAME                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                        VOLUMEATTRIBUTESCLASS   AGE
example-vanilla-rwo-pvc   Bound    pvc-a3665c91-0d6d-419d-89f6-3c394812d6e8   5Gi        RWO            example-vanilla-rwo-filesystem-sc   <unset>                 4m4s
root@k8s-control-574-1720547892:~# k get po
NAME                        READY   STATUS    RESTARTS   AGE
example-vanilla-block-pod   1/1     Running   0          4m6s

15:41:55  full-sync-test [csi-block-vanilla-destructive] Scale down driver deployment to zero replica and verify PV metadata is created in CNS [p1, block, vanilla, disruptive, core]
15:41:55  /home/worker/workspace/Block-Vanilla/Results/2632/vsphere-csi-driver/tests/e2e/fullsync_test_for_block_volume.go:696

15:48:25  Ran 1 of 877 Specs in 395.796 seconds
15:48:25  SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 876 Skipped
15:48:25  PASS
15:48:25  You're using deprecated Ginkgo functionality:
15:48:25  =============================================
15:48:25    --always-emit-ginkgo-writer is deprecated  - use -v instead, or one of Ginkgo's machine-readable report formats to get GinkgoWriter output for passing specs.
15:48:25  
15:48:25  To silence deprecations that can be silenced set the following environment variable:
15:48:25    ACK_GINKGO_DEPRECATIONS=2.11.0

Special notes for your reviewer:

Release note:

Use client-go package for syncer leaderelection

[adikul30]

cc: @divyenpatel

[adikul30]

make check

(vsphere-csi-driver) $ make check
hack/check-format.sh
hack/check-mdlint.sh
hack/check-shell.sh
hack/check-staticcheck.sh
+ go version
go version go1.21.3 darwin/amd64
++ dirname hack/check-staticcheck.sh
+ cd hack/..
+ go install honnef.co/go/tools/cmd/staticcheck@2023.1
++ go env GOPATH
+ GOOS=linux
+ /Users/adkulkarni/go/bin/staticcheck --version
staticcheck 2023.1 (v0.4.0)
++ go env GOPATH
++ go list ./...
++ grep -v /vendor/
+ GOOS=linux
+ /Users/adkulkarni/go/bin/staticcheck sigs.k8s.io/vsphere-csi-driver/v3/cmd/syncer sigs.k8s.io/vsphere-csi-driver/v3/cmd/vsphere-csi sigs.k8s.io/vsphere-csi-driver/v3/pkg/apis/cnsoperator sigs.k8s.io/vsphere-csi-driver/v3/pkg/apis/cnsoperator/cnsfileaccessconfig/v1alpha1 sigs.k8s.io/vsphere-csi-driver/v3/pkg/apis/cnsoperator/cnsnodevmattachment/v1alpha1 sigs.k8s.io/vsphere-csi-driver/v3/pkg/apis/cnsoperator/cnsregistervolume/v1alpha1 sigs.k8s.io/vsphere-csi-driver/v3/pkg/apis/cnsoperator/cnsvolumemetadata/v1alpha1 sigs.k8s.io/vsphere-csi-driver/v3/pkg/apis/cnsoperator/config sigs.k8s.io/vsphere-csi-driver/v3/pkg/apis/cnsoperator/storagepolicy/v1alpha1 sigs.k8s.io/vsphere-csi-driver/v3/pkg/apis/migration sigs.k8s.io/vsphere-csi-driver/v3/pkg/apis/migration/config sigs.k8s.io/vsphere-csi-driver/v3/pkg/apis/migration/v1alpha1 sigs.k8s.io/vsphere-csi-driver/v3/pkg/apis/storagepool sigs.k8s.io/vsphere-csi-driver/v3/pkg/apis/storagepool/cns sigs.k8s.io/vsphere-csi-driver/v3/pkg/apis/storagepool/cns/v1alpha1 sigs.k8s.io/vsphere-csi-driver/v3/pkg/apis/storagepool/config sigs.k8s.io/vsphere-csi-driver/v3/pkg/common/cns-lib/node sigs.k8s.io/vsphere-csi-driver/v3/pkg/common/cns-lib/volume sigs.k8s.io/vsphere-csi-driver/v3/pkg/common/cns-lib/vsphere sigs.k8s.io/vsphere-csi-driver/v3/pkg/common/config sigs.k8s.io/vsphere-csi-driver/v3/pkg/common/fault sigs.k8s.io/vsphere-csi-driver/v3/pkg/common/prometheus sigs.k8s.io/vsphere-csi-driver/v3/pkg/common/unittestcommon sigs.k8s.io/vsphere-csi-driver/v3/pkg/common/utils sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/common sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/common/commonco sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/common/commonco/k8sorchestrator sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/common/commonco/types sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/common/placementengine sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/logger sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/mounter sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/osutils sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/wcp sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/wcpguest sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/types sigs.k8s.io/vsphere-csi-driver/v3/pkg/internalapis sigs.k8s.io/vsphere-csi-driver/v3/pkg/internalapis/cnsoperator/cnsfilevolumeclient sigs.k8s.io/vsphere-csi-driver/v3/pkg/internalapis/cnsoperator/cnsfilevolumeclient/v1alpha1 sigs.k8s.io/vsphere-csi-driver/v3/pkg/internalapis/cnsoperator/config sigs.k8s.io/vsphere-csi-driver/v3/pkg/internalapis/cnsoperator/triggercsifullsync/v1alpha1 sigs.k8s.io/vsphere-csi-driver/v3/pkg/internalapis/cnsvolumeinfo sigs.k8s.io/vsphere-csi-driver/v3/pkg/internalapis/cnsvolumeinfo/config sigs.k8s.io/vsphere-csi-driver/v3/pkg/internalapis/cnsvolumeinfo/v1alpha1 sigs.k8s.io/vsphere-csi-driver/v3/pkg/internalapis/cnsvolumeoperationrequest sigs.k8s.io/vsphere-csi-driver/v3/pkg/internalapis/cnsvolumeoperationrequest/config sigs.k8s.io/vsphere-csi-driver/v3/pkg/internalapis/cnsvolumeoperationrequest/v1alpha1 sigs.k8s.io/vsphere-csi-driver/v3/pkg/internalapis/csinodetopology sigs.k8s.io/vsphere-csi-driver/v3/pkg/internalapis/csinodetopology/config sigs.k8s.io/vsphere-csi-driver/v3/pkg/internalapis/csinodetopology/v1alpha1 sigs.k8s.io/vsphere-csi-driver/v3/pkg/internalapis/featurestates sigs.k8s.io/vsphere-csi-driver/v3/pkg/internalapis/featurestates/config sigs.k8s.io/vsphere-csi-driver/v3/pkg/internalapis/featurestates/v1alpha1 sigs.k8s.io/vsphere-csi-driver/v3/pkg/kubernetes sigs.k8s.io/vsphere-csi-driver/v3/pkg/syncer sigs.k8s.io/vsphere-csi-driver/v3/pkg/syncer/admissionhandler sigs.k8s.io/vsphere-csi-driver/v3/pkg/syncer/cnsoperator/controller sigs.k8s.io/vsphere-csi-driver/v3/pkg/syncer/cnsoperator/controller/cnsfileaccessconfig sigs.k8s.io/vsphere-csi-driver/v3/pkg/syncer/cnsoperator/controller/cnsnodevmattachment sigs.k8s.io/vsphere-csi-driver/v3/pkg/syncer/cnsoperator/controller/cnsregistervolume sigs.k8s.io/vsphere-csi-driver/v3/pkg/syncer/cnsoperator/controller/cnsvolumemetadata sigs.k8s.io/vsphere-csi-driver/v3/pkg/syncer/cnsoperator/controller/csinodetopology sigs.k8s.io/vsphere-csi-driver/v3/pkg/syncer/cnsoperator/controller/triggercsifullsync sigs.k8s.io/vsphere-csi-driver/v3/pkg/syncer/cnsoperator/manager sigs.k8s.io/vsphere-csi-driver/v3/pkg/syncer/cnsoperator/types sigs.k8s.io/vsphere-csi-driver/v3/pkg/syncer/cnsoperator/util sigs.k8s.io/vsphere-csi-driver/v3/pkg/syncer/k8scloudoperator sigs.k8s.io/vsphere-csi-driver/v3/pkg/syncer/storagepool sigs.k8s.io/vsphere-csi-driver/v3/tests/e2e
hack/check-vet.sh
hack/check-golangci-lint.sh
golangci/golangci-lint info checking GitHub for tag 'v1.59.1'
golangci/golangci-lint info found version: 1.59.1 for v1.59.1/darwin/amd64
golangci/golangci-lint info installed /Users/adkulkarni/go/bin/golangci-lint
INFO golangci-lint has version 1.59.1 built with go1.22.3 from 1a55854a on 2024-06-09T18:08:33Z
INFO [config_reader] Config search paths: [./ /Users/adkulkarni/GolandProjects/vsphere-csi-driver /Users/adkulkarni/GolandProjects /Users/adkulkarni /Users /]
INFO [config_reader] Used config file .golangci.yml
INFO [lintersdb] Active 8 linters: [errcheck gosimple govet ineffassign lll misspell staticcheck unused]
INFO [loader] Go packages loading at mode 575 (deps|imports|name|compiled_files|files|types_sizes|exports_file) took 4.820532095s
INFO [runner/filename_unadjuster] Pre-built 0 adjustments in 618.724006ms
INFO [linters_context/goanalysis] analyzers took 1m31.386805025s with top 10 stages: buildir: 32.062847353s, S1038: 5.177632209s, misspell: 4.878828332s, unused: 2.424426197s, S1039: 2.259500042s, printf: 2.105441711s, SA1012: 1.889129492s, SA4030: 1.873529505s, S1028: 1.751927547s, S1024: 1.720117662s
INFO [runner] Issues before processing: 142, after processing: 0
INFO [runner] Processors filtering stat (out/in): identifier_marker: 52/52, exclude-rules: 1/52, filename_unadjuster: 142/142, autogenerated_exclude: 52/142, skip_files: 142/142, skip_dirs: 142/142, nolint: 0/1, invalid_issue: 142/142, path_prettifier: 142/142, cgo: 142/142, exclude: 52/52
INFO [runner] processing took 39.002958ms with stages: nolint: 29.206267ms, autogenerated_exclude: 6.445116ms, identifier_marker: 1.439883ms, path_prettifier: 1.225551ms, skip_dirs: 344.501µs, exclude-rules: 247.845µs, invalid_issue: 53.649µs, cgo: 22.428µs, filename_unadjuster: 9.885µs, max_same_issues: 1.587µs, fixer: 923ns, uniq_by_line: 757ns, skip_files: 752ns, max_from_linter: 615ns, diff: 577ns, source_code: 455ns, path_shortener: 429ns, exclude: 417ns, sort_results: 411ns, severity-rules: 353ns, max_per_file_from_linter: 308ns, path_prefixer: 249ns
INFO [runner] linters took 19.447913031s with stages: goanalysis_metalinter: 19.408710641s
INFO File cache stats: 276 entries of total size 7.2MiB
INFO Memory: 249 samples, avg is 982.4MB, max is 2846.1MB
INFO Execution took 24.932490487s

[adikul30]

/test pull-vsphere-csi-driver-build

[divyenpatel]

/ok-to-test

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: adikul30, divyenpatel

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/kubernetes-sigs/vsphere-csi-driver/blob/master/OWNERS)~~ [adikul30,divyenpatel] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

[divyenpatel]

/lgtm