fails to get aws metadata when IMDSv2 is required on ECS running on EC2 launch type

[krissell]

Short description

Fails to get aws metadata when IMDSv2 is required on the ECS cluster running on EC2 infrastructure

This check does not work for EC2 based EC2 clusters : https://github.com/open-policy-agent/opa/blob/main/plugins/rest/aws.go#L273

It's causing OPA to skip the token call. However the token is required.

The check seems incorrect in that 'yes, this is ECS' but 'yes, it's also EC2'.

Is this check really trying to do a EC2 vs Fargate check?

Steps To Reproduce

1. Create an aws cluster on ECS using EC2 infrastructure with IMDSv2 as required
2. Start OPA with the a reference to the metadata. ie. --set=services.s3.credentials.s3_signing.metadata_credentials.iam_role=abc
3. On startup OPA will fail to get the metadata with

{"headers":{"Content-Length":["0"],"Content-Type":["text/plain"],"Date":["Thu, 27 Jul 2023 19:13:49 GMT"],"Server":["EC2ws"]},"level":"debug","msg":"Received response from metadata service.","status":"401 Unauthorized","time":"2023-07-27T19:13:49Z","url":"http://169.254.169.254/latest/meta-data/iam/security-credentials/abc"}

IF IMDSv2 is not required, the call works since the token is not required.

[krissell]

Browsing the environment variables on my EC2 launch type of ECS, I have AWS_EXECUTION_ENV=AWS_ECS_EC2 Set. Which looks promising, however after browsing the web I can't seem to find official documentation for such a variable

[ashutosh-narkar]

We'll have to add a check which looks at the environment variable AWS_EXECUTION_ENV=AWS_ECS_EC2 and then does the token request. This should support the ECS cluster with EC2 launch type scenario.

AWS_EXECUTION_ENV seems to be defined in a Lambda context per this so not sure if we can rely on this.

[ashutosh-narkar]

Maybe there is something else that identifies the Amazon ECS launch type.

[krissell]

That's all I found for doc too @ashutosh-narkar . In my case though, I'm not using a lambda. So it looks to be there as we need but not officially documented. However, it maybe the best we have, since browsing the env vars in my EC2 instance didn't yield anything that looked specific to EC2.

[ashutosh-narkar]

I think we need something that's an official reference if we have to make this change. This would be required from a maintenance standpoint as well. I would recommend you follow-up with AWS on that. If ECS cluster with EC2 launch type scenario is officially supported and/or widely used, I would imagine there are some docs or references we can link to.

[krissell]

Hi @ashutosh-narkar The EC2 launch type is supported and documented. Sorry, I was talking about the env var not being documented outside of the lambda reference, even though it's being set.

Supported launch types: Fargate, EC2, & External: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/launch_types.html

[ashutosh-narkar]

To clarify I was talking about the env var or any other way to identify the launch type. For ex, do we know if AWS_EXECUTION_ENV=AWS_ECS_EC2 is a reliable indicator and can we used to determine the launch type? We could add a check that looks for this env var and does something but do we know if this is always set? OTOH this could be the right way to check but not documented anywhere.

[ashutosh-narkar]

@krissell closing this for now. Once we have more information on this feel free to re-open. If you'd like to submit a fix that would be great too. Thanks!

[krissell]

Hi, I opened a support case with aws.

1. They confirmed the environment variable mentioned above is reliable
2. The var is not documented, as we know
3. They opened an internal case to get the ECS doc updated with this variable. However that would take a while. I'll post again when it's updated. Thanks

[krissell]

Hi, The AWS support team recommends way this way to tell the launch type: Use the environment variable discussed here:
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint-v4.html#task-metadata-endpoint-v4-response

Namely: Does env var ECS_CONTAINER_METADATA_URI_V4 exist?

If so GET it. The response includes: "LaunchType":"EC2", // Or could be Fargate



This LaunchType key & value is documented in the above link.

[ashutosh-narkar]

Great, thanks for following up on this! I'll re-open the issue. If you'd like to submit a fix please feel free to do so.

[stale[bot]]

This issue has been automatically marked as inactive because it has not had any activity in the last 30 days. Although currently inactive, the issue could still be considered and actively worked on in the future. More details about the use-case this issue attempts to address, the value provided by completing it or possible solutions to resolve it would help to prioritize the issue.