Connecting to cloud sql instance using workload identity federation credential config file

[v4runMurthy]

Question

👋🏼 Hey folks, I could not find a clear explanation for this issue, hence thought this might be the best place to get some clarity on the Cloud SQL proxy setup for Workload Identity Federation(WIF) with AWS.

I have some Kubernetes pods in my AWS project that would like to access my cloud sql instance in GCP. I am using cloud-sql-proxy: 2.3.0 as a sidecar container on my AWS pods. I followed the WIF docs to set up a pool and provider, associated it with a service account that has Cloud SQL Client role assigned to it. The WIF docs mention that the path to the WIF credential configuration file can be passed to the GOOGLE_APPLICATION_CREDENTIALS environment variable. I have attached the WIF configuration file to the cloud-sql-proxy container and passed its path in the GOOGLE_APPLICATION_CREDENTIALS environment variable.

However, when I try to connect to a database, I run into the following error where the request to get the access token seems to be failing.

2023/06/21 12:14:25 Authorizing with Application Default Credentials
2023/06/21 12:14:25 [dev:us-east1:test] Listening on 127.0.0.1:3306
2023/06/21 12:14:25 The proxy has started successfully and is ready for new connections!
2023/06/21 12:17:51 [dev:us-east1:test] accepted connection from 127.0.0.1:39086
2023/06/21 12:17:51 [dev:us-east1:test] failed to connect to instance: failed to get instance: Refresh error: failed to get instance metadata (connection name = "dev:us-east1:test"): Get "https://sqladmin.googleapis.com/sql/v1beta4/projects/dev/instances/test/connectSettings?alt=json&prettyPrint=false": oauth2/google: unable to generate access token: Post "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/aws-test@dev.iam.gserviceaccount.com:generateAccessToken": oauth2/google: unable to retrieve AWS security credentials - error fetching credentials: forbidden by policy

Initially thought this might be due to a wrong WIF configuration or have to do with restriction on AWS, but I was able to spin up an alpine container in the same pod and generate the access token using the same WIF configuration following the REST API authentication steps in the WIF doc. When I started the cloud-sql-proxy with the generated access token, I was able to connect to the cloud-sql instance, this confirmed that the WIF configuration is correct.

So now I'm trying to understand if the cloud-sql-proxy supports WIF, and if this works with the configuration file path is passed to the GOOGLE_APPLICATION_CREDENTIALS environment variable?

Thanks

Code

No response

Additional Details

No response

[hessjcg]

Hi @v4runMurthy,

Yes, Proxy v2.3.0 does support workload identity federation. We have tested this successfully using all-GCP credentials and services. However, it should work with any supported workload identity configuration, including AWS.

Yes, the proxy does load the the application credentials configuration from the file specified in the GOOGLE_APPLICATION_CREDENTIALS environment variable.

Based on the error message, I suspect that the pod lacks permission to access an AWS metadata service due to some combination of kubernetes roles and AWS IAM policy. I found a similar error and solution from another person trying to set up workload identity on AWS EKS. I suspect that this part of the error: error fetching credentials: forbidden by policy occurs under these circumstances:

1. The pod starts.
2. The proxy attempts to use the google oauth2 library to authenticate.
3. The google oauth2 library loads the configuration from the file path in GOOGLE_APPLICATION_CREDENTIALS env var.
4. The google oauth2 library recognizes that it is using AWS credentials and attempts to get a token from the AWS metadata service.
5. The AWS metadata service denies the request responding with the error response: 403: forbidden by policy

This is an interesting case. Let us know how you solve it.

[v4runMurthy]

@hessjcg thanks for confirming that the cloud-sql-proxy supports WIF with credentials from the path provided to GOOGLE_APPLICATION_CREDENTIALS environment variable. Your explanation for AWS metadata service not being accessible makes sense as I tried to change the IAM role on the pods and got a different AWS error.

I will check this out and post back what I find, after updating access to the AWS metadata service from the EKS pods

[enocom]

I'm going to close this but if you're still hitting Proxy-related problems, feel free to add more info here and we'll help out.