argocd not syncing application PVC following PV/PVC migration

[Shamsun1]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug After upgrading a kubernetes cluster from v1.20.11 to v1.21.10 the persistent volume/persistent volume claims for an application were migrated to use the new azure disk csi provider. In argocd the applications persistent volume claim does not sync due to the volumeName being a difference (this remains the same after migration). The application then syncs if the source yaml for the PVC is updated with the volumeName (PV name) from the live PVC manifest. A replace sync with force will create a new PV using the PVC where the volumeName is in the live PVC manifest but not in the source but this will sync.

The migration steps followed are detailed here (under the "Migrate Persistent Storage to the Azure Disk CSI driver" section: https://github.com/Azure/aks-engine/blob/master/docs/topics/azure-stack.md#migrate-persistent-storage-to-the-azure-disk-csi-driver

To Reproduce A kubernetes cluster with a grafana application deployed was upgraded from kubernetes v1.20.11 to v1.21.10. The PV and PVC for the application were migrated to the azure disk csi driver in kubernetes v1.21.10. This involved deleting the deployment, PV and PVC. Then recreating the deployment, recreating the PVC after removing the following fields: (metadata.annotations, .metadata.creationTimestamp, .metadata.resourceVersion, .metadata.uid, .status) and recreating the PV after removing the following fields: (.metadata.annotations, .metadata.creationTimestamp, .metadata.managedFields, .metadata.resourceVersion, .metadata.uid, .status, .spec.claimRef.resourceVersion, .spec.claimRef.uid, .spec.azureDisk) and updating the spec with: "csi": { "driver": "disk.csi.azure.com", "volumeHandle": "${PV_DISK_URI}", "fsType": "${PV_FS_TYPE}" }

The applications PVC would then fail to sync in argocd due to the volumeName being a difference.

Expected behavior The applications PVC should sync sucessfully following the migration of the PV/PVC

Screenshots

Version v2.2.5 (v2.2.5+8f981cc) but also tested on v2.3.5

Logs Error: one or more objects failed to apply, reason: PersistentVolumeClaim "grafana" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims core.PersistentVolumeClaimSpec{ AccessModes: {"ReadWriteOnce"}, Selector: nil, Resources: {Requests: {s"storage": {i: {...}, s: "1Gi", Format: "BinarySI"}}}, - VolumeName: "", + VolumeName: "pvc-c033ac15-e36e-4d14-8794-8a1a88e13a76", StorageClassName: &"managed-premium", VolumeMode: &"Filesystem", DataSource: nil, }

[lknite]

I'm seeing this same issue (even with a new clean deployment)

[lknite]

Tried using ignoreDifferences in applicationset as a workaround, no luck, but maybe I have something wrong:

      syncPolicy:
        ...

      ignoreDifferences:
      - group: io.k8s.api.core.v1
        kind: PersistentVolumeClaim
        jqPathExpressions:
        - .spec.volumeName

[chris-ng-scmp]

I have the same issue as well for disk migration from Flexvolume to CSI with the same volumeName

[TomMaSS]

Have same issue with thanos stack with compactor as kubernetes cronjob. Pv bounded, jobs run successful. Argo trying to patch PersistentVolumeClaim, what's impossible. Is it possible to deny pvc patching?

[mmerbes]

Also commenting here to say I have this same issue with multiple applications. Despite deleting pods and SVCs the issue persists.

[davidfrickert]

same issue here. would appreciate a fix/workaround

[davidfrickert]

Okay, i think i've found a workaround:

In application spec, tell argo to ignore volumeName diffs (and maybe other problematic fields?) Also ensure argo sync respects this by adding RespectIgnoreDifferences, otherwise your sync's will keep failing: https://argo-cd.readthedocs.io/en/latest/user-guide/sync-options/#respect-ignore-difference-configs

spec:
  ignoreDifferences:
  - jsonPointers:
    - /spec/volumeName
    kind: PersistentVolumeClaim
  syncPolicy:
    syncOptions:
      - RespectIgnoreDifferences=true