Does AWS SDK Go V2 take into account Fargate's different IMDS credential endpoint?

HJTP commented 7 months ago

Acknowledgements

[X] I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
[X] I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

We run Go in an ECS container in Fargate. I attached a policy to the ECS task execution role that should get it access to SES (we want to send an e-mail) but it fails to access the IMDS endpoint:

Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/": dial tcp 169.254.169.254:80: connect: invalid argument

Therefore, it cannot access credentials and thus the call to SES fails.

Expected Behavior

I would expect the SDK to load the credentials from the role/policy. Perhaps this requires accessing a different IP address as listed here: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html

Credentials can be retrieved from curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

Current Behavior

It tries to connect to http://169.254.169.254/latest/meta-data/iam/security-credentials/ but it fails

The IP address is hardcoded here: https://github.com/aws/aws-sdk-go-v2/blob/49b368e9d7a38a2373c833be135270e5390c2b41/feature/ec2/imds/api_client.go#L59

Reproduction Steps

The example from the docs will do:

package main

import (
    "context"
    "log"
    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/s3"
)

func main() {
    // Load the Shared AWS Configuration (~/.aws/config)
    cfg, err := config.LoadDefaultConfig(context.TODO())
    if err != nil {
        log.Fatal(err)
    }

    // Create an Amazon S3 service client
    client := s3.NewFromConfig(cfg)

    // Get the first page of results for ListObjectsV2 for a bucket
    output, err := client.ListObjectsV2(context.TODO(), &s3.ListObjectsV2Input{
        Bucket: aws.String("my-bucket"),
    })
    if err != nil {
        log.Fatal(err)
    }

    log.Println("first page results:")
    for _, object := range output.Contents {
        log.Printf("key=%s size=%d", aws.ToString(object.Key), object.Size)
    }
}

But make sure to run it inside a Docker container on Fargate, for example with this task-definition.json


{
  "family": "my-application",
  "executionRoleArn": "my-ecs-task-execution-role-that-has-AmazonSESFullAccess",
  "networkMode": "awsvpc",
  "cpu": "4096",
  "memory": "8192",
  "requiresCompatibilities": ["FARGATE"],
  "containerDefinitions": [
    {
      "name": "backend",
      "image": "myaccount.dkr.ecr.eu-west-1.amazonaws.com/image:latest",
      "cpu": 512,
      "memory": 1024,
      "essential": true,
      "portMappings": [
        {
          "containerPort": 8080,
          "protocol": "tcp"
        },
        {
          "hostPort": 22,
          "containerPort": 22,
          "protocol": "tcp"
        }
      ],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-create-group": "true",
          "awslogs-group": "/ecs/service-service",
          "awslogs-region": "eu-west-1",
          "awslogs-stream-prefix": "backend"
        }
      }
    }

### Possible Solution

Should the SDK detect that the container is running in Fargate and query the `169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI` endpoint instead of the `169.254.169.254` one?

Or perhaps this should be configurable

### Additional Information/Context

_No response_

### AWS Go SDK V2 Module Versions Used

github.com/aws/aws-sdk-go-v2 v1.24.1
github.com/aws/aws-sdk-go-v2/config v1.26.6
github.com/aws/aws-sdk-go-v2/service/ses v1.19.6

### Compiler and Version used

1.19

### Operating System and version

docker.io/library/golang:1.19

HJTP commented 7 months ago

Closing this bug report now, this was my mistake. For any future reader: I attached the policy to the ECS task execution role, but processes running inside ECS containers use credentials from the task role. A subtle, but crucial difference. Adding a new role as task role and attaching the policy to that fixed my problem.

github-actions[bot] commented 7 months ago

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.

aws / aws-sdk-go-v2