[flux v0.26.0-2] Kustomization tries to modify immutable fields

[Legion2]

Describe the bug

I updated to flux 0.26.1 and then observed an reconciliation error in a deployment. I deleted the deployment and now there is a problem with the pvc of that deployment.

PersistentVolumeClaim/monitoring/influxdb-volume dry-run failed, reason: Invalid, error: PersistentVolumeClaim "influxdb-volume" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims
  core.PersistentVolumeClaimSpec{
        AccessModes:      {"ReadWriteOnce"},
        Selector:         nil,
        Resources:        {Requests: {s"storage": {i: {...}, s: "10Gi", Format: "BinarySI"}}},
-       VolumeName:       "",
+       VolumeName:       "pvc-c7f9929e-2741-43ce-b690-ed00816092ad",
        StorageClassName: &"aws-gp2-dynamic",
        VolumeMode:       &"Filesystem",
        DataSource:       nil,
  }

I tried to downgrade the kustomization controller, but that did not resolve the issue.

Steps to reproduce

1. Install flux 0.25.3
2. create a PVC

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
   name: influxdb-volume
   labels:
   kustomize.toolkit.fluxcd.io/prune: disabled
   spec:
   storageClassName: aws-gp2-dynamic
   accessModes:
   - ReadWriteOnce
   volumeMode: Filesystem
   resources:
   requests:
     storage: 10Gi

3. Update flux to 0.26.1

Expected behavior

Should work after the update

Screenshots and recordings

No response

OS / Distro

Ubuntu 21

Flux version

flux: v0.26.1

Flux check

► checking prerequisites ✔ Kubernetes 1.20.11-eks-f17b81 >=1.20.6-0 ► checking controllers ✔ helm-controller: deployment ready ► ghcr.io/fluxcd/helm-controller:v0.16.0 ✔ image-automation-controller: deployment ready ► ghcr.io/fluxcd/image-automation-controller:v0.20.0 ✔ image-reflector-controller: deployment ready ► ghcr.io/fluxcd/image-reflector-controller:v0.16.0 ✔ kustomize-controller: deployment ready ► ghcr.io/fluxcd/kustomize-controller:v0.19.1 ✔ notification-controller: deployment ready ► ghcr.io/fluxcd/notification-controller:v0.21.0 ✔ source-controller: deployment ready ► ghcr.io/fluxcd/source-controller:v0.21.1 ✔ all checks passed

Git provider

Gitlab

Container Registry provider

Gitlab

Additional context

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[stefanprodan]

Using kubectl works because there is no SA, but when you apply things from inside the cluster Knative whats the flux SA. I see no way around this but to delete the service and then it will be created by the kustomize-controller SA.

[tshak]

There was an error on my end. The managedFields patch did in fact work. The error I was getting was pointing to a similarly named emojivoto service. This means that we'll need to patch all knative services prior to upgrading to Flux v2. Thank you for taking the time to help me debug this issue. 🙏

[mrtndwrd]

I'm also running into this issue. The system we're running doesn't really enable us to manually run patches. Is there any way we can apply these patches through flux to roll it out to all the servers tracking our repository?

[mdnfiras]

i'm running into this problem after updating flux from 0.24.1 to 0.28.5:

PersistentVolumeClaim/my-db/my-pvc dry-run failed, reason: Invalid, error: PersistentVolumeClaim "my-pvc" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests for bound claims  
                                                                                                              core.PersistentVolumeClaimSpec{                                                                                                                                                                                                               
                                                                                                                AccessModes:      {"ReadWriteOnce"},                                                                                                                                                                                                        
                                                                                                                Selector:         nil,                                                                                                                                                                                                                      
                                                                                                                Resources:        {Requests: {s"storage": {i: {...}, s: "20Gi", Format: "BinarySI"}}},                                                                                                                                                      
                                                                                                            -   VolumeName:       "",                                                                                                                                                                                                                       
                                                                                                            +   VolumeName:       "pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxx",                                                                                                                                                                                  
                                                                                                            -   StorageClassName: nil,                                                                                                                                                                                                                      
                                                                                                            +   StorageClassName: &"standard",                                                                                                                                                                                                              
                                                                                                                VolumeMode:       &"Filesystem",                                                                                                                                                                                                            
                                                                                                                DataSource:       nil,                                                                                                                                                                                                                      
                                                                                                              }

i'm running on gke 1.20

[snahelou]

Hello

Patch did the job 👍

kubectl -n demo patch pvc test --type=json -p='[{"op": "remove", "path": "/metadata/managedFields/0/fieldsV1/f:spec/f:volumeMode"}]'
kubectl -n demo patch pvc test --type=json -p='[{"op": "remove", "path": "/metadata/managedFields/0/fieldsV1/f:spec/f:volumeName"}]'

Flux : 0.28.5 / K8S: 1.22

Thanks

[uclaeaslam]

I've run into this issue too. I upgraded from Flux v1 to v2 (latest 0.29.3). Everything appeared to be working fine (after modifying all the yamls to use default namespace since v2 requires namespace). A few days later, I noticed that I'm getting reconciliation errors about the immutable fields

[stefanprodan]

@uclaeaslam it seems that Flux v1 took ownership of fields it didn’t manage (I suspect this is a kubectl bug). To fix this you need to remove those fields with a patch, please see the examples in this thread. If after patching, the problem persists, please open an issue in kustomize-controller repo.

[uclaeaslam]

Patching did the trick

The reason I thought it didn't work was because flux wasn't rerunning the reconciliation. I had to do flux suspend kustomization flux-system flux resume kustomization flux-system After that, it retried.

[stefanprodan]

@uclaeaslam the reconciliation interval is set to 10 minutes. You can trigger it with a git commit or with the CLI:

flux reconcile ks flux-system --with-source

[zzvara]

Unfortunately, this is still an issue with Flux and knative. We can not use knative Services in HelmChart, therefore we are looking into Argo CD and whether it could work there.