IMDS not accessible from docker container

jesseduffield commented 3 weeks ago

I'm trying to run AWS commands from within a docker container on my CI runner and it appears that it's failing to get credentials from IMDS:

    * No valid credential sources found: Please see https://registry.terraform.io/providers/hashicorp/aws
for more information about providing credentials.
Error: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, access disabled to EC2 IMDS via client option, or "AWS_EC2_METADATA_DISABLED" environment

Previously I directly stored IAM user credentials in github but I've switched to instead using the EC2InstanceCustomPolicy cloudformation param. Other steps in the CI workflow that are outside of docker are still able to successfully do things (e.g. pushing docker images to ECR).

I wonder if this is related to what this comment says about permitting more than one network hop to access IMDS: https://github.com/hashicorp/terraform-provider-aws/issues/23110#issuecomment-1035712587

If so, could we up the number of allowed network hops so that we can access IMDS from within a docker container?

Thanks

jesseduffield commented 3 weeks ago

Also for the record I'm using this workaround currently to fetch the credentials directly from IMDS:

- name: Export temp AWS credentials from IMDS
  shell: bash
  run: |
    TOKEN=$(curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
    ROLE_NAME=$(curl -H "X-aws-ec2-metadata-token: $TOKEN" -s http://169.254.169.254/latest/meta-data/iam/security-credentials/)
    CREDENTIALS=$(curl -H "X-aws-ec2-metadata-token: $TOKEN" -s http://169.254.169.254/latest/meta-data/iam/security-credentials/$ROLE_NAME)
    echo "AWS_ACCESS_KEY_ID=$(echo $CREDENTIALS | jq -r .AccessKeyId)" >> $GITHUB_ENV
    echo "AWS_SECRET_ACCESS_KEY=$(echo $CREDENTIALS | jq -r .SecretAccessKey)" >> $GITHUB_ENV
    echo "AWS_SESSION_TOKEN=$(echo $CREDENTIALS | jq -r .Token)" >> $GITHUB_ENV

crohr commented 3 weeks ago

@jesseduffield This is surprising because the number of allowed hops is already set to 2, to allow for just this use case: https://github.com/runs-on/runs-on/blob/cd56b4ad3d5a2ba7ce2f11a7b31a48f2d3096ea7/cloudformation/template-v2.5.8.yaml#L756

I'll try to reproduce.

runs-on / runs-on

IMDS not accessible from docker container #158