ECS: Unable to get IAM security credentials from EC2 Instance Metadata Service

[Trevortni]

I'm trying to use this in an ECS Task, and I'm getting the error "Unable to get IAM security credentials from EC2 Instance Metadata Service."

I have set up policies on my ECS Task role to provide access to the specific Secrets I am trying to access, as well as KMS and Session Manager, according to https://aws.amazon.com/premiumsupport/knowledge-center/ecs-data-security-container-task/ and a few other links.

Is there anything else that I'm missing that needs to be done to get this to work?

[Kralizek]

Could you share your setup code?

[Trevortni]

Do you mean this?

    public class Program
    {
        public static void Main(string[] args)
        {
            CreateHostBuilder(args).Build().Run();
        }

        public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .ConfigureAppConfiguration((hostingContext, config) =>
                {
#if !DEBUG
                    config.AddSecretsManager();
#endif
                })
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.UseStartup<Startup>();
                });
    }

[divekarvinit]

Hey @Trevortni , are you able to access those secrets through CLI? Did you have to switch roles to access the secrets?

[vminkovski]

Hi, I am running into the same error in ECS. Locally, from Visual Studio all is working fine and I think the IAM roles are set up correctly in AWS. There is nothing specific in the setup - I just specify the region and add a filter for the secrets, no credentials are passed. Any help will be highly appreciated. Thanks in advance

[Trevortni]

Hi, I am running into the same error in ECS. Locally, from Visual Studio all is working fine and I think the IAM roles are set up correctly in AWS. There is nothing specific in the setup - I just specify the region and add a filter for the secrets, no credentials are passed. Any help will be highly appreciated. Thanks in advance

Did you set up the secrets in the container definition? I remember that being something I didn't have a grasp on when I originally set this up, though I can't remember if that was before or after asking this question. I did eventually get it working, though I can't remember all the details.

One other thing I remember is being unclear between the task role and the task execution role; I think I currently have both of them set up with the IAM policy, since my recollection of which one originally worked was at odds with which one seemed to be working after it mysteriously stopped working after working for a while.

[werebear73]

This is how I got credentials

               _logger.LogInformation("Attempting to get credentials from AWS Fargate");

                // Get HTTP client to retrieve the AWS credentials from the AWS Fargate metadata service
                var client = new HttpClient()
                {
                    BaseAddress = new Uri($"http://169.254.170.2{builder.Configuration.GetValue(typeof(string), "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI")}")
                };

                // Get the AWS credentials from the AWS Fargate metadata service
                var response = client.GetAsync("");
                var result = response.Result;

                // Process the response and add the AWS Secrets Manager to the configuration builder
                if (result.IsSuccessStatusCode)
                {
                    var json = result.Content.ReadAsStringAsync().Result;
                    _logger.LogInformation($"Got credentials from AWS Fargate: {json}");  // PROBABLY SHOULD NOT LOG THE CREDENTIALS -- REMOVE LATER
                    var data = JsonSerializer.Deserialize<Dictionary<string, string>>(json);
                    var tempCredentials = new SessionAWSCredentials(data["AccessKeyId"], data["SecretAccessKey"], data["Token"]);
                    tempCredentials.GetCredentials();
                    builder.Services.AddDefaultAWSOptions(new AWSOptions
                    {
                        Credentials = tempCredentials
                    });

However, once I run the AddSecretsManager I don't get any Secrets to appear in my Configuration. I have both these methods.

                    configurationBuilder.AddSecretsManager(tempCredentials, operatingEnvironment.Region ?? RegionEndpoint.APSoutheast1, options =>
                    {
                        options.SecretFilter = (list) => list.Name.StartsWith($"{builder.Environment.EnvironmentName}_IdVerification__RdsConnectionInformation");
                        options.KeyGenerator = (secret, name) => name.Replace("__", ":");
                        options.PollingInterval = TimeSpan.FromMinutes(15);
                    });

and

                    appBuilder.AddSecretsManager(tempCredentials, operatingEnvironment.Region ?? RegionEndpoint.APSoutheast1, options =>
                    {
                        options.SecretFilter = (list) => list.Name.StartsWith($"{builder.Environment.EnvironmentName}_IdVerification__RdsConnectionInformation");
                        options.KeyGenerator = (secret, name) => name.Replace("__", ":");
                        options.PollingInterval = TimeSpan.FromMinutes(15);
                    });