Closed ColeSiegelTR closed 2 years ago
@lsirac
Hey @ColeSiegelTR can you clarify what the inconsistency is here?
Hi @bojeil-google, the metadata endpoint returns the underlying EC2 node role instead of the current active Task role, which forces us to provide a larger scope of permissions than required when configuring workload identity federation with GCP. I would expect that when binding an AWS role with GCP workload identity, we can specify the AWS role of the task rather than the AWS role of the node.
I am not sure the resolution, maybe AWS has a different metadata endpoint which return a more specific role, rather than this underlying node role which your library is using?
This is what is made available by AWS. We need the ability to generate signed requests for the desired AWS role. We can either do it via AWS metadata server directly or via AWS environment variables. Basically we use the AWS security credentials to create the signed requests and verify them on our end. You can't just specify a role explicitly as we cannot trust that. We can only trust signed requests.
Maybe you should reach out to AWS support about this. Perhaps they have some other endpoint or ability to generate signed requests using the current active task role.
Hi @bojeil-google
I did some research and they do have an endpoint for receiving ECS task metadata as opposed to the fetching the underlying EC2 node's metadata.
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint.html
I think this library should be using the task metadata endpoint 169.254.170.2, but it is instead using the underlying EC2 metadata endpoint 169.254.169.254. This means we need to expand permissions more broadly than we would've liked.
Let me know if it makes sense or if you need more clarification.
I think the library is doing as expected. We are not going to change the existing behavior. It will break all of our customers. However, we will do more research on supporting this use case (and what the ECS task metadata provides) and treat it as a feature request. Customers may end up choosing which one to use.
Also thanks @ColeSiegelTR for bringing this to our attention. We haven't thought of this use case until now.
Hey @bojeil-google
Thanks for your response, that makes sense. I am quite new to this library myself so I need to dig in a bit more on our use case and also understand how/where these metadata endpoints are retrieved by the library. We appreciate you taking a look and I will provide more information if I have any.
I'm going to close this, feel free to reopen or ask us to if you have additional questions.
Hey @lesv and @bojeil-google
Is there any ticket we can track for the feature request? I took a closer look at the code and it doesn't look there is anything we can do on our end to better support this use case. Thanks a lot.
All containers running in an AWS ECS Cluster have permissions to make Google Cloud API calls as opposed to just the one required task.
To configure the workload identity federation for making Google Cloud API calls from AWS, we bind an AWS role ARN to a Google Service account. And the Google cloud SDK uses the AWS metadata endpoint to retrieve the current active role. When we run this inside an AWS ECS task, the metadata endpoint returns the underlying EC2 node role instead of the Task role. This forces the user to bind the underlying node role, thereby providing expanding the permission boundary more than desired. This also hinders least privileged access - meaning provide Tasks with their own level of permissions.
There is some inconsistency in the AWS metadata endpoint (when running inside EC2 or Lambda vs ECS) and the way the Google Cloud SDK retrieves the active AWS role.
https://github.com/googleapis/google-auth-library-java/blob/1149581e63267e3553c74ba2114d849c5b24f27b/oauth2_http/java/com/google/auth/oauth2/AwsCredentials.java#L107