kubernetes-csi / external-snapshotter

Sidecar container that watches Kubernetes Snapshot CRD objects and triggers CreateSnapshot/DeleteSnapshot against a CSI endpoint.
Apache License 2.0
484 stars 369 forks source link

[snapshot-controller] Retry PVC finalizer removal on conflict #1133

Closed Fricounet closed 1 month ago

Fricounet commented 2 months ago

What type of PR is this?

/kind bug

What this PR does / why we need it:

The previous removePVCFinalizer function was using the PVC stored in the informer which, in cases where the PVC had been modified since, lead to conflict errors when trying to remove the PVC finalizer through an update. Example error:

E0807 13:09:20.573657       1 snapshot_controller.go:183] error check and remove PVC finalizer for snapshot [snapshot]: snapshot controller failed to update pvc-sts-0 on API server: Operation cannot be fulfilled on persistentvolumeclaims "pvc-sts-0": the object has been modified; please apply your changes to the latest version and try again

Unfortunately, the error was not retried so the PVC ends up stuck.

Now the removePVCFinalizer function uses the RetryOnConflict helper to make sure the update goes through.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Fixes a race condition where the PVC finalizer could end up not being removed by the snapshot-controller if the update had a conflict.
k8s-ci-robot commented 2 months ago

Hi @Fricounet. Thanks for your PR.

I'm waiting for a kubernetes-csi member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
carlory commented 2 months ago

/ok-to-test

xing-yang commented 1 month ago

/lgtm /approve

k8s-ci-robot commented 1 month ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Fricounet, xing-yang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/kubernetes-csi/external-snapshotter/blob/master/OWNERS)~~ [xing-yang] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment