PVs stuck in terminating state because AP's don't exist

[joshcoburn]

/kind bug

What happened? Multiple PVCs were created using dynamic provisioning method via the efs-csi-driver. The associated APs were deleted from EFS within the AWS console.

What you expected to happen? efs-csi-driver can reconcile this if the ap is removed.

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?: If efs-csi-driver cannot reconcile this, Is there a way to force termination of the PV from the cluster side to clear out the errors?

Environment

• Kubernetes version (use kubectl version): 1.30
• Driver version: 2.0.7

Please also attach debug logs to help us better diagnose

I1025 14:10:53.648675       1 controller.go:396] DeleteVolume: called with args {VolumeId:fs-XXXXXXXXXXXX::fsap-0a86234991af48a4d Secrets:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I1025 14:10:53.696643       1 controller.go:396] DeleteVolume: called with args {VolumeId:fs-XXXXXXXXXXXX::fsap-0a86234991af48a4d Secrets:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I1025 14:13:50.265018       1 controller.go:396] DeleteVolume: called with args {VolumeId:fs-XXXXXXXXXXXX::fsap-0446bd8ce3a756f90 Secrets:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I1025 14:13:50.266642       1 controller.go:396] DeleteVolume: called with args {VolumeId:fs-XXXXXXXXXXXX::fsap-0dd0e0eb8b371d841 Secrets:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I1025 14:13:50.268219       1 controller.go:396] DeleteVolume: called with args {VolumeId:fs-XXXXXXXXXXXX::fsap-045996bc39b66cb9f Secrets:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I1025 14:13:50.268591       1 controller.go:396] DeleteVolume: called with args {VolumeId:fs-XXXXXXXXXXXX::fsap-0cb395c77c457b133 Secrets:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I1025 14:13:50.274038       1 controller.go:396] DeleteVolume: called with args {VolumeId:fs-XXXXXXXXXXXX::fsap-0c6b9df3b192c0823 Secrets:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I1025 14:13:50.277634       1 controller.go:396] DeleteVolume: called with args {VolumeId:fs-XXXXXXXXXXXX::fsap-0e2d2606ba74833b3 Secrets:map[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
E1025 14:13:50.293874       1 driver.go:107] GRPC error: rpc error: code = Unauthenticated desc = Access Denied. Please ensure you have the right AWS permissions: Access denied
E1025 14:13:50.295683       1 driver.go:107] GRPC error: rpc error: code = Unauthenticated desc = Access Denied. Please ensure you have the right AWS permissions: Access denied
E1025 14:13:50.296597       1 driver.go:107] GRPC error: rpc error: code = Unauthenticated desc = Access Denied. Please ensure you have the right AWS permissions: Access denied
E1025 14:13:50.297677       1 driver.go:107] GRPC error: rpc error: code = Unauthenticated desc = Access Denied. Please ensure you have the right AWS permissions: Access denied
E1025 14:13:50.298322       1 driver.go:107] GRPC error: rpc error: code = Unauthenticated desc = Access Denied. Please ensure you have the right AWS permissions: Access denied
E1025 14:13:50.299372       1 driver.go:107] GRPC error: rpc error: code = Unauthenticated desc = Access Denied. Please ensure you have the right AWS permissions: Access denied ```

[mskanth972]

Seems external provisioner side car is the issue here, EFS CSI Driver is using v5.0.1-eks-1-30-8 which has a bug or regression and which is mitigated in the latest version v5.1.0-eks-1-31-5. We are working on releasing the latest version with this fix.

[joshcoburn]

@mskanth972 thank you for the info! Look forward to the fix version release.