search

kubernetes-sigs / aws-efs-csi-driver

CSI Driver for Amazon EFS https://aws.amazon.com/efs/
Apache License 2.0
693 stars 524 forks source link

Unable to provision volumes in iso-us-east-1 AWS region #1349

Closed hazmat345 closed 1 month ago

hazmat345 commented 1 month ago

/kind bug

What happened? Attempting to provision a volume fails because the provisioner does not trust the AWS TLS certificate. The following event is generated:

failed to provision volume with StorageClass "efs": rpc error: code = Internal desc = Failed to fetch Access Points or Describe File System: List Access Points failed: RequestError: send request failed
caused by: Get "https://elasticfilesystem.us-iso-east-1.<rest of url>: tls: failed to verify certificate: x509: certificate signed by unknown authority

What you expected to happen? The volume to be provisioned successfully.

How to reproduce it (as minimally and precisely as possible)? Attempt to provision a volume in an AWS region that does not use a certificate issued by an authority in the standard CA certificate bundle.

Anything else we need to know?: The certificate used by the AWS endpoint in my environment is not part of the standard CA bundle, so I need to mount my own CA trust bundle into the pods.

I suspect this would not longer be an issue if this PR is accepted: https://github.com/kubernetes-sigs/aws-efs-csi-driver/pull/1165

Environment

Please also attach debug logs to help us better diagnose

slambrose commented 1 month ago

+1

seanzatzdev-amazon commented 1 month ago

Thank you for bringing this to our attention. We are investigating this issue. In the meantime, can you follow the log-collection steps linked here to provide us with debug logs?

hazmat345 commented 1 month ago

Getting logs from my actual environment is going to be tricky... let me see if I can figure out a way to reproduce things.

whoix commented 1 month ago

@hazmat345 @slambrose

Is there a reason why you can't use the managed add-on for EFS with your EKS cluster? The add-on is supported in ISO regions and fully compatible. The managed add-on already auto injects the custom CA bundle used in ISO regions.

slambrose commented 1 month ago

@whoix For unfortunate "reasons", we are unable to use EKS at the moment, so we are building our RKE2 clusters on EC2 instances w/ terraform and ansible. I believe the PR was merged yesterday, so this issue can probably be closed now.