Environment Variable AWS_USE_FIPS_ENDPOINT does not use AWS Fips Endpoint as expected

[repl-richard-puth]

/kind bug

What happened? First off, I'm not sure if this would qualify as a bug or not, but nonetheless the following does not appear to function as expected. To give a bit of context, our EKS cluster runs the ebs-csi-driver as a daemonset. It coordinates three ebs-csi-driver pods. We've added the AWS_USE_FIPS_ENDPOINT environment variable to the Daemonset's configuration. After testing the new pods with the added environment variable, we were unable to see the AWS Fips endpoint in the logs. Instead, it was the regular AWS Endpoint that got used.

What you expected to happen? We expected Cloudtrail logs to display the AWS FIPS endpoint instead of the regular endpoint. We made sure that the version aws-ebs-csi-driver is using uses the correct go SDK version supporting the AWS_USE_FIPS_ENDPOINT env variable.

How to reproduce it (as minimally and precisely as possible)?

1. Deploy aws-ebs-driver with the following environment variable.

"env": [
          {
            .....
          },
          {
            "name": "AWS_USE_FIPS_ENDPOINT",
            "value": "true"
          }
        ],

2. Then start a test container that uses the ebs-csi-driver. We tested with https://github.com/kubernetes-sigs/aws-ebs-csi-driver/tree/master/examples/kubernetes/block-volume.
3. Once that is done, Cloudtrail logs should've logged an event called AttachVolume. This event shows what was endpoint is used.

   "tlsDetails": {
       "tlsVersion": "TLSv1.3",
       "cipherSuite": "TLS_AES_128_GCM_SHA256",
       "clientProvidedHostHeader": "ec2.us-west-2.amazonaws.com"
   }

Anything else we need to know?:

Environment

• Kubernetes version (use kubectl version):

  version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.3", GitCommit:"aef86a93758dc3cb2c658dd9657ab4ad4afc21cb", GitTreeState:"clean", BuildDate:"2022-07-13T14:30:46Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}
  Kustomize Version: v4.5.4
  Server Version: version.Info{Major:"1", Minor:"28+", GitVersion:"v1.28.13-eks-a737599", GitCommit:"9183cd02caedacf6a14583843262d53d6244fc4a", GitTreeState:"clean", BuildDate:"2024-08-26T21:27:49Z", GoVersion:"go1.22.5", Compiler:"gc", Platform:"linux/amd64"}

• Driver version: 1.36.0

[torredil]

Hi @repl-richard-puth, the AWS_USE_FIPS_ENDPOINT env var needs to be set in the controller's configuration, not in the DaemonSet configuration (the controller pod exercises the API calls).