Add host.docker.internal to list of approved hosts for ECS credentials

[benkehoe]

Originally opened as https://github.com/boto/botocore/issues/2515

When running some code expecting AWS credentials in a docker container, a good way to provide those credentials is through an ECS-compatible endpoint running on the host, providing some credentials the host has available. The ECS endpoint is better than the IMDSv2 endpoint, as it requires a shared secret (IMDSv2 is protected against SSRF but isn't truly authenticated). I have a tool which will do just that, aws-export-credentials.

The ECS credentialing mechanism in the SDKs allows an endpoint to be set by the environment variable AWS_CONTAINER_CREDENTIALS_FULL_URI, but it checks the hostname against a list of approved endpoints, which includes localhost and the ECS metadata server at 169.254.170.2 (for the most part, see below for the messy details).

However, for docker, the host (when access to it is provided) is reachable as host.docker.internal (docs), and this is not on the list of approved hosts for ECS endpoints in any of the SDKs.

On Linux, the workaround is to use --network host, which makes it accessible as localhost from inside the container, but on Mac and Windows this doesn't work (the docker network is always isolated from the host).

host.docker.internal should be added to the list of approved hosts, so the locally-provided ECS credentials endpoint can work even if the code is running in a docker container.

This would also provide the opportunity to synchronize the list of approved hosts, which currently varies across SDKs.

For example, in botocore, ContainerMetadataFetcher currently allows three hosts for container metadata endpoints: 169.254.170.2, localhost, and 127.0.0.1.

botocore allows 169.254.170.2 in AWS_CONTAINER_CREDENTIALS_FULL_URI, where at least Go, Java v1, and JavaScript v2 only allow localhost/127.0.0.1 in AWS_CONTAINER_CREDENTIALS_FULL_URI and use 169.254.170.2 only for AWS_CONTAINER_CREDENTIALS_RELATIVE_URI 🤷

I should note that the amazon-ecs-local-container-endpoints project doesn't solve this problem; it provides the credential endpoint at 169.254.170.2 for other containers on the docker network, but to provide it with credentials, it says to mount ~/.aws, just shifting the problem to that container.

[tim-finnigan]

Thanks for the feature request — we encourage others to 👍 the issue if also interested in this.

[nathanpeck]

I like this feature suggestion. However I would note that this is very Docker specific and more and more people are running containers in other tools that aren't "Docker". I wonder if its better to have people be able to pass in an arbitrary approved IP or hostname of their own choice when instantiating the SDK.

I don't see any downsides to giving people ability to programmatically append their own arbitrary IP addresses or hostnames to the allow list. This hardcoded list is always going to be a pain point that needs to be updated to keep up with the latest ways that people are running containers, compared to just giving users the ability to extend the list as they see fit.

For reference:

• podman uses host.containers.internal - https://docs.podman.io/en/v3.3.1/markdown/podman-pod-create.1.html
• nerdctl / lima (basis for Finch) uses host.lima.internal - https://github.com/containerd/nerdctl/issues/747
• minikube uses host.minikube.internal - https://minikube.sigs.k8s.io/docs/handbook/host-access/

[benkehoe]

I agree about docker not being the only container runtime that matters, though it does feel like a 90/10 situation at the moment in terms of including it in the built-in allowlist that doesn't require the user to get involved.

[benkehoe]

Given that ICANN is planning to reserve .internal for private use, perhaps the answer here is to allow *.internal?

[benkehoe]

I'd also like to add that the utility of this functionality has increased, since the Roles Anywhere credential helper has added PKCS#11 support. This permits credentials used in applications to be sourced from hardware-secured keys, but that chain is only as secure as the intermediary steps. With the process mechanism, anyone with execute permissions can get credentials. With the IMDSv2 mechanism, anybody who can reach the HTTP endpoint can get credentials. But with the ECS mechanism, a shared credential needs to be provided to the application to be able to retrieve credentials. So enabling this containerized use case would help us achieve the dream of AWS credentials for non-AWS machines tied securely to hardware-secured identity.