search

kubernetes-csi / external-provisioner

Sidecar container that watches Kubernetes PersistentVolumeClaim objects and triggers CreateVolume/DeleteVolume against a CSI endpoint
Apache License 2.0
337 stars 323 forks source link

Patch verb for persistentvolumes resources in the external-provisioner-runner clusterrole is not effective #1255

Open sameshai opened 1 month ago

sameshai commented 1 month ago

What happened: As per PR below PR If the feature-gate HonorPVReclaimPolicy is enabled, create a pvc with a delete relaim policy, then delete the pvc, the pv stuck in deleting status, the error message is: https://github.com/kubernetes-csi/external-provisioner/pull/1155

csi-mockplugin-0/csi-provisioner@kind..lane: I0206 11:08:41.536819       1 controller.go:1523] delete "pvc-e4b6e20e-4d77-4bf0-8d96-9f47212e4b38": failed to remove finalizer for persistentvolume: persistentvolumes "pvc-e4b6e20e-4d77-4bf0-8d96-9f47212e4b38" is forbidden: User "system:serviceaccount:csi-mock-honor-pv-reclaim-policy-99-996:csi-mock" cannot update resource "persistentvolumes" in API group "" at the cluster scope: RBAC: [clusterrole.rbac.authorization.k8s.io "cluster-driver-registrar-runner-csi-mock-honor-pv-reclaim-policy-99" not found, clusterrole.rbac.authorization.k8s.io "e2e-test-privileged-psp" not found]
csi-mockplugin-0/csi-provisioner@kind..lane: W0206 11:08:41.536850       1 controller.go:989] Retrying syncing volume "pvc-e4b6e20e-4d77-4bf0-8d96-9f47212e4b38", failure 0
csi-mockplugin-0/csi-provisioner@kind..lane: E0206 11:08:41.536880       1 controller.go:1007] error syncing volume "pvc-e4b6e20e-4d77-4bf0-8d96-9f47212e4b38": persistentvolumes "pvc-e4b6e20e-4d77-4bf0-8d96-9f47212e4b38" is forbidden: User "system:serviceaccount:csi-mock-honor-pv-reclaim-policy-99-996:csi-mock" cannot update resource "persistentvolumes" in API group "" at the cluster scope: RBAC: [clusterrole.rbac.authorization.k8s.io "cluster-driver-registrar-runner-csi-mock-honor-pv-reclaim-policy-99" not found, clusterrole.rbac.authorization.k8s.io "e2e-test-privileged-psp" not found]
I0206 19:08:

I did try this with IBM VPC Block CSI driver and 5.0.2 provisioner but seems without added the patch permission I am still able to delete PVC/PV and no error.

sameershaikh@Sameers-MBP Downloads % k describe pvc
Name:     pvc-block
Namespace:   default
StorageClass: ibmc-vpc-block-5iops-tier
Status:    Bound
Volume:    pvc-351e6980-ac05-4441-a482-6ea544b3aad5
Labels:    <none>
Annotations:  pv.kubernetes.io/bind-completed: yes
        pv.kubernetes.io/bound-by-controller: yes
        volume.beta.kubernetes.io/storage-provisioner: vpc.block.csi.ibm.io
        volume.kubernetes.io/storage-provisioner: vpc.block.csi.ibm.io
Finalizers:  [kubernetes.io/pvc-protection]
Capacity:   10Gi
Access Modes: RWO
VolumeMode:  Filesystem
Used By:    <none>
Events:    <none>
sameershaikh@Sameers-MBP Downloads % k describe pv 
Name:       pvc-351e6980-ac05-4441-a482-6ea544b3aad5
Labels:      <none>
Annotations:    pv.kubernetes.io/provisioned-by: vpc.block.csi.ibm.io
          volume.kubernetes.io/provisioner-deletion-secret-name: 
          volume.kubernetes.io/provisioner-deletion-secret-namespace: 
Finalizers:    [kubernetes.io/pv-protection external-provisioner.volume.kubernetes.io/finalizer]
StorageClass:   ibmc-vpc-block-5iops-tier
Status:      Bound
Claim:       default/pvc-block
Reclaim Policy:  Delete
Access Modes:   RWO
VolumeMode:    Filesystem
Capacity:     10Gi
Node Affinity:   
 Required Terms:  
  Term 0:    failure-domain.beta.kubernetes.io/region in [us-south]
          failure-domain.beta.kubernetes.io/zone in [us-south-3]
Message:      
Source:
  Type:       CSI (a Container Storage Interface (CSI) volume source)
  Driver:      vpc.block.csi.ibm.io
  FSType:      ext4
  VolumeHandle:   r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4
  ReadOnly:     false
  VolumeAttributes:   clusterID=cqsr5km10bghfqv414rg
              failure-domain.beta.kubernetes.io/region=us-south
              failure-domain.beta.kubernetes.io/zone=us-south-3
              iops=3000
              storage.kubernetes.io/csiProvisionerIdentity=1723447099267-9715-vpc.block.csi.ibm.io
              tags=
              volumeCRN=crn:v1:staging:public:is:us-south-3:a/77f2bceddaeb577dcaddb4073fe82c1c::volume:r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4
              volumeId=r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4
Events:
 Type  Reason        Age  From                      Message
 ----  ------        ----  ----                      -------
 Normal VolumeMetaDataSaved 2m20s ibm-vpc-block-csi-controller-588c75b988-xpkbt Success
sameershaikh@Sameers-MBP file-csi-driver % k describe clusterRole vpc-block-provisioner-role
Name:     vpc-block-provisioner-role
Labels:    addonmanager.kubernetes.io/mode=Reconcile
       app=ibm-vpc-block-csi-driver
Annotations: razee.io/build-url: https://travis.ibm.com/alchemy-containers/addon-vpc-block-csi-driver/builds/22121223
       razee.io/source-url: https://github.ibm.com/alchemy-containers/addon-vpc-block-csi-driver/commit/d7c3406c7f992ef727f3e9e3ce60f4e33c4f20c2
       version: 5.2.20_579
PolicyRule:
 Resources                    Non-Resource URLs Resource Names Verbs
 ---------                    ----------------- -------------- -----
 persistentvolumes                []         []       [get list watch create delete]
 persistentvolumeclaims             []         []       [get list watch update]
 nodes                      []         []       [get list watch]
 secrets                     []         []       [get list watch]
 csinodes.storage.k8s.io             []         []       [get list watch]
 storageclasses.storage.k8s.io          []         []       [get list watch]
 volumeattachments.storage.k8s.io        []         []       [get list watch]
 configmaps                   []         []       [get list]
 volumesnapshotcontents.snapshot.storage.k8s.io []         []       [get list]
 volumesnapshots.snapshot.storage.k8s.io     []         []       [get list]
 events                     []         []       [list watch create update patch]
sameershaikh@Sameers-MBP file-csi-driver %


},\"f:failure-domain.beta.kubernetes.io/region\":{},\"f:failure-domain.beta.kubernetes.io/zone\":{},\"f:iops\":{},\"f:storage.kubernetes.io/csiProvisionerIdentity\":{},\"f:tags\":{},\"f:volumeCRN\":{},\"f:volumeId\":{}},\"f:volumeHandle\":{}},\"f:nodeAffinity\":{\".\":{},\"f:required\":{}},\"f:persistentVolumeReclaimPolicy\":{},\"f:storageClassName\":{},\"f:volumeMode\":{}}} } {csi-attacher Update v1 2024-08-12 13:05:53 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:finalizers\":{\"v:\\\"external-attacher/vpc-block-csi-ibm-io\\\"\":{}}}} } {csi-resizer Update v1 2024-08-12 13:07:05 +0000 UTC FieldsV1 {\"f:spec\":{\"f:capacity\":{\"f:storage\":{}}}} } {kube-controller-manager Update v1 2024-08-12 13:08:58 +0000 UTC FieldsV1 {\"f:status\":{\"f:phase\":{}}} status}]},Spec:PersistentVolumeSpec{Capacity:ResourceList{storage: {{21474836480 0} {<nil>} 20Gi BinarySI},},PersistentVolumeSource:PersistentVolumeSource{GCEPersistentDisk:nil,AWSElasticBlockStore:nil,HostPath:nil,Glusterfs:nil,NFS:nil,RBD:nil,ISCSI:nil,Cinder:nil,CephFS:nil,FC:nil,Flocker:nil,FlexVolume:nil,AzureFile:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Local:nil,StorageOS:nil,CSI:&CSIPersistentVolumeSource{Driver:vpc.block.csi.ibm.io,VolumeHandle:r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4,ReadOnly:false,FSType:ext4,VolumeAttributes:map[string]string{clusterID: cqsr5km10bghfqv414rg,failure-domain.beta.kubernetes.io/region: us-south,failure-domain.beta.kubernetes.io/zone: us-south-3,iops: 3000,storage.kubernetes.io/csiProvisionerIdentity: 1723447099267-9715-vpc.block.csi.ibm.io,tags: ,volumeCRN: crn:v1:staging:public:is:us-south-3:a/77f2bceddaeb577dcaddb4073fe82c1c::volume:r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4,volumeId: r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4,},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,ControllerExpandSecretRef:nil,NodeExpandSecretRef:nil,},},AccessModes:[ReadWriteOnce],ClaimRef:&ObjectReference{Kind:PersistentVolumeClaim,Namespace:default,Name:pvc-block,UID:351e6980-ac05-4441-a482-6ea544b3aad5,APIVersion:v1,ResourceVersion:2679,FieldPath:,},PersistentVolumeReclaimPolicy:Delete,StorageClassName:ibmc-vpc-block-5iops-tier,MountOptions:[],VolumeMode:*Filesystem,NodeAffinity:&VolumeNodeAffinity{Required:&NodeSelector{NodeSelectorTerms:[]NodeSelectorTerm{NodeSelectorTerm{MatchExpressions:[]NodeSelectorRequirement{NodeSelectorRequirement{Key:failure-domain.beta.kubernetes.io/region,Operator:In,Values:[us-south],},NodeSelectorRequirement{Key:failure-domain.beta.kubernetes.io/zone,Operator:In,Values:[us-south-3],},},MatchFields:[]NodeSelectorRequirement{},},},},},VolumeAttributesClassName:nil,},Status:PersistentVolumeStatus{Phase:Released,Message:,Reason:,LastPhaseTransitionTime:<nil>,},}"
I0812 13:09:13.994031    1 controller.go:1258] "shouldDelete" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:13.994053    1 controller.go:1288] "shouldDelete is true" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:13.994064    1 controller.go:1132] "shouldDelete" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:13.994078    1 controller.go:1548] "Started" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:13.994098    1 controller.go:1312] volume pvc-351e6980-ac05-4441-a482-6ea544b3aad5 does not need any deletion secrets
I0812 13:09:13.994456    1 connection.go:264] "GRPC call" method="/csi.v1.Controller/DeleteVolume" request="{\"volume_id\":\"r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4\"}"
I0812 13:09:19.569875    1 reflector.go:808] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:858: Watch close - *v1.PersistentVolume total 13 items received
I0812 13:09:24.904270    1 connection.go:270] "GRPC response" response="{}" err=null
I0812 13:09:24.904325    1 controller.go:1563] "Volume deleted" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
W0812 13:09:24.966351    1 warnings.go:70] spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].key: failure-domain.beta.kubernetes.io/region is deprecated since v1.17; use "topology.kubernetes.io/region" instead
W0812 13:09:24.966364    1 warnings.go:70] spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[1].key: failure-domain.beta.kubernetes.io/zone is deprecated since v1.17; use "topology.kubernetes.io/zone" instead
I0812 13:09:24.966561    1 controller.go:1606] "PersistentVolume deleted succeeded" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:24.966783    1 controller.go:1171] "handleProtectionFinalizer" PV="&PersistentVolume{ObjectMeta:{pvc-351e6980-ac05-4441-a482-6ea544b3aad5  3f0a9a59-3442-4171-b67a-efdd232c3664 19236 0 2024-08-12 07:31:18 +0000 UTC 2024-08-12 13:09:24 +0000 UTC 0xc0005b1210 map[] map[pv.kubernetes.io/provisioned-by:vpc.block.csi.ibm.io volume.kubernetes.io/provisioner-deletion-secret-name: volume.kubernetes.io/provisioner-deletion-secret-namespace:] [] [kubernetes.io/pv-protection external-provisioner.volume.kubernetes.io/finalizer] [{csi-provisioner Update v1 2024-08-12 13:01:59 +0000 UTC FieldsV1 {\"f:metadata\":{\"f:annotations\":{\".\":{},\"f:pv.kubernetes.io/provisioned-by\":{},\"f:volume.kubernetes.io/provisioner-deletion-secret-name\":{},\"f:volume.kubernetes.io/provisioner-deletion-secret-namespace\":{}},\"f:finalizers\":{\"v:\\\"external-provisioner.volume.kubernetes.io/finalizer\\\"\":{}}},\"f:spec\":{\"f:accessModes\":{},\"f:capacity\":{},\"f:claimRef\":{\".\":{},\"f:apiVersion\":{},\"f:kind\":{},\"f:name\":{},\"f:namespace\":{},\"f:resourceVersion\":{},\"f:uid\":{}},\"f:csi\":{\".\":{},\"f:driver\":{},\"f:fsType\":{},\"f:volumeAttributes\":{\".\":{},\"f:clusterID\":{},\"f:failure-domain.beta.kubernetes.io/region\":{},\"f:failure-domain.beta.kubernetes.io/zone\":{},\"f:iops\":{},\"f:storage.kubernetes.io/csiProvisionerIdentity\":{},\"f:tags\":{},\"f:volumeCRN\":{},\"f:volumeId\":{}},\"f:volumeHandle\":{}},\"f:nodeAffinity\":{\".\":{},\"f:required\":{}},\"f:persistentVolumeReclaimPolicy\":{},\"f:storageClassName\":{},\"f:volumeMode\":{}}} } {csi-resizer Update v1 2024-08-12 13:07:05 +0000 UTC FieldsV1 {\"f:spec\":{\"f:capacity\":{\"f:storage\":{}}}} } {kube-controller-manager Update v1 2024-08-12 13:08:58 +0000 UTC FieldsV1 {\"f:status\":{\"f:phase\":{}}} status}]},Spec:PersistentVolumeSpec{Capacity:ResourceList{storage: {{21474836480 0} {<nil>} 20Gi BinarySI},},PersistentVolumeSource:PersistentVolumeSource{GCEPersistentDisk:nil,AWSElasticBlockStore:nil,HostPath:nil,Glusterfs:nil,NFS:nil,RBD:nil,ISCSI:nil,Cinder:nil,CephFS:nil,FC:nil,Flocker:nil,FlexVolume:nil,AzureFile:nil,VsphereVolume:nil,Quobyte:nil,AzureDisk:nil,PhotonPersistentDisk:nil,PortworxVolume:nil,ScaleIO:nil,Local:nil,StorageOS:nil,CSI:&CSIPersistentVolumeSource{Driver:vpc.block.csi.ibm.io,VolumeHandle:r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4,ReadOnly:false,FSType:ext4,VolumeAttributes:map[string]string{clusterID: cqsr5km10bghfqv414rg,failure-domain.beta.kubernetes.io/region: us-south,failure-domain.beta.kubernetes.io/zone: us-south-3,iops: 3000,storage.kubernetes.io/csiProvisionerIdentity: 1723447099267-9715-vpc.block.csi.ibm.io,tags: ,volumeCRN: crn:v1:staging:public:is:us-south-3:a/77f2bceddaeb577dcaddb4073fe82c1c::volume:r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4,volumeId: r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4,},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,ControllerExpandSecretRef:nil,NodeExpandSecretRef:nil,},},AccessModes:[ReadWriteOnce],ClaimRef:&ObjectReference{Kind:PersistentVolumeClaim,Namespace:default,Name:pvc-block,UID:351e6980-ac05-4441-a482-6ea544b3aad5,APIVersion:v1,ResourceVersion:2679,FieldPath:,},PersistentVolumeReclaimPolicy:Delete,StorageClassName:ibmc-vpc-block-5iops-tier,MountOptions:[],VolumeMode:*Filesystem,NodeAffinity:&VolumeNodeAffinity{Required:&NodeSelector{NodeSelectorTerms:[]NodeSelectorTerm{NodeSelectorTerm{MatchExpressions:[]NodeSelectorRequirement{NodeSelectorRequirement{Key:failure-domain.beta.kubernetes.io/region,Operator:In,Values:[us-south],},NodeSelectorRequirement{Key:failure-domain.beta.kubernetes.io/zone,Operator:In,Values:[us-south-3],},},MatchFields:[]NodeSelectorRequirement{},},},},},VolumeAttributesClassName:nil,},Status:PersistentVolumeStatus{Phase:Released,Message:,Reason:,LastPhaseTransitionTime:<nil>,},}"
I0812 13:09:24.966803    1 controller.go:1258] "shouldDelete" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:24.966813    1 controller.go:1288] "shouldDelete is true" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:24.966820    1 controller.go:1132] "shouldDelete" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:24.966830    1 controller.go:1548] "Started" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:24.966844    1 controller.go:1312] volume pvc-351e6980-ac05-4441-a482-6ea544b3aad5 does not need any deletion secrets
I0812 13:09:24.966899    1 connection.go:264] "GRPC call" method="/csi.v1.Controller/DeleteVolume" request="{\"volume_id\":\"r134-221cb6f6-a150-40d1-99b1-5f3f1d92e5e4\"}"
I0812 13:09:25.194175    1 connection.go:270] "GRPC response" response="{}" err=null
I0812 13:09:25.194222    1 controller.go:1563] "Volume deleted" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5"
I0812 13:09:25.207594    1 controller.go:1569] "Failed to delete persistentvolume" PV="pvc-351e6980-ac05-4441-a482-6ea544b3aad5" err="persistentvolumes \"pvc-351e6980-ac05-4441-a482-6ea544b3aad5\" not found"
I0812 13:09:25.207641    1 controller.go:1007] "Retrying syncing volume" key="pvc-351e6980-ac05-4441-a482-6ea544b3aad5" failures=0
E0812 13:09:25.207795    1 controller.go:1025] error syncing volume "pvc-351e6980-ac05-4441-a482-6ea544b3aad5": persistentvolumes "pvc-351e6980-ac05-4441-a482-6ea544b3aad5" not found
I0812 13:09:48.412100    1 reflector.go:808] k8s.io/client-go/informers/factory.go:160: Watch close - *v1.CSINode total 9 items received
I0812 13:10:11.430774    1 reflector.go:808] k8s.io/client-go/informers/factory.go:160: Watch close - *v1.PersistentVolumeClaim total 15 items received
I0812 13:11:05.631537    1 reflector.go:808] sigs.k8s.io/sig-storage-lib-external-provisioner/v10/controller/controller.go:861: Watch close - *v1.StorageClass total 11 items received
I0812 13:11:58.431219    1 reflector.go:808] k8s.io/client-go/informers/factory.go:160: Watch close - *v1.Node total 21 items received

What you expected to happen: I was expecting RBAC error

How to reproduce it:

  1. Create PVC with external provisioner
  2. Delete PVC which will trigger the PV deletion
  3. There is no error even if we dont have patch permission.

Anything else we need to know?:

Environment:

Madhu-1 commented 3 days ago

@xing-yang @carlory In the above code we are still doing the update operation and in the RBAC we don't have update access rather as part of https://github.com/kubernetes-csi/external-provisioner/pull/1155 to address the same issue, can you please check this one and let me know if we need to add RBAC for update or change the code in provisioner to use patch as well

https://github.com/kubernetes-csi/external-provisioner/blob/master/pkg/controller/controller.go#L984

https://github.com/kubernetes-csi/external-provisioner/blob/master/pkg/controller/controller.go#L1676

we still need an update for adding the annotation and for updating finalizers we can use the patch

Madhu-1 commented 3 days ago

/assign

carlory commented 3 days ago

I did try this with IBM VPC Block CSI driver and 5.0.2 provisioner but seems without added the patch permission I am still able to delete PVC/PV and no error.

I will try to reproduce it with hostpath driver.

we still need an update for adding the annotation and for updating finalizers we can use the patch

Agreed.

According to xing-yang's comment, to add a new RBAC rule, we need to bump the major version.

carlory commented 3 days ago

we need to add some e2e tests to detect a similar issue. should the new tests be added to the external-provisioner repo? cc @xing-yang @jsafrane

Madhu-1 commented 2 days ago

/unassign

sorry i was wrong i like to the PVC update code not the PV update, i don't see code in main that require PV update access.

carlory commented 2 days ago

@sameshai Can you execute the following command with your service account name? A service account may be bound with more than one cluster role.

kubectl auth can-i patch pv --as=system:serviceaccount:default:csi-hostpathplugin-sa
carlory commented 2 days ago

If the answer is "yes", it means that the external provisioner can patch the persistent volume resource even if the external-provisioner cluster role does not have permission. It is allowed via other cluster roles.