Docker-compose: Unable to retrieve credentials. Message = "Unable to reach credentials server"

[kron0s19]

I have a legacy app running on a EC2 instance (NOT using ECS or EKS) , which is running on docker-compose with an nginx proxy.

If I run the app locally, It runs properly without issues.

Im pulling the secrets using that library with this code:

configurationManager.AddSecretsManager(configurator: options =>
        {
            options.SecretFilter = entry => entry.Name.StartsWith($"{environmentName}_{applicationName}");
            options.KeyGenerator = (_, s) => s
                .Replace($"{environmentName}_{applicationName}_", string.Empty)
                .Replace("__", ":");
            options.PollingInterval = TimeSpan.FromSeconds(10);
        });

I added an IAM Role to the EC2 instance, and that role has SecretsManagerReadWrite permissions.

Using the cli on the instance, I can do:

curl http://169.254.169.254/latest/meta-data/iam/security-credentials/MyRoleName And get a proper response

When we first ran the app, we WERE getting the following exception:

**Unhandled exception. Amazon.Runtime.AmazonServiceException: Unable to get IAM security credentials from EC2 Instance Metadata Service**.
   at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.FetchCredentials()
   at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.GetCredentials()
   at Amazon.Runtime.DefaultInstanceProfileAWSCredentials.GetCredentialsAsync()
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Kralizek.Extensions.Configuration.Internal.SecretsManagerConfigurationProvider.FetchAllSecretsAsync(CancellationToken cancellationToken)
   at Kralizek.Extensions.Configuration.Internal.SecretsManagerConfigurationProvider.FetchConfigurationAsync(CancellationToken cancellationToken)
   at Kralizek.Extensions.Configuration.Internal.SecretsManagerConfigurationProvider.LoadAsync()
   at Kralizek.Extensions.Configuration.Internal.SecretsManagerConfigurationProvider.Load()
   at Microsoft.Extensions.Configuration.ConfigurationManager.AddSource(IConfigurationSource source)
   at Microsoft.Extensions.Configuration.ConfigurationManager.Microsoft.Extensions.Configuration.IConfigurationBuilder.Add(IConfigurationSource source)
   at Microsoft.Extensions.Configuration.SecretsManagerExtensions.AddSecretsManager(IConfigurationBuilder configurationBuilder, AWSCredentials credentials, RegionEndpoint region, Action1 configurator)

I updated the docker-compose adding 2 env variables (AWS_REGION and AWS_CONTAINER_CREDENTIALS_RELATIVE_URI). And it started working (no idea why).

This is the docker-compose

services:
  nginx-proxy:
    image: custom-nginx:latest
    restart: always
    ports:
      - 80:80
      - 443:443
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro

  production:
    image: production:latest
    restart: always
    environment:
      - AWS_REGION=us-east-2
      - AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/latest/meta-data/iam/security-credentials/MyRoleName

  staging:
    image: staging:latest
    restart: always
    environment:
      - AWS_REGION=us-east-2
      - AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/latest/meta-data/iam/security-credentials/MyRoleName

However, after updating the docker image and 'docker-compose up -d', it started failing again, but now with a different exception:

From the call stack, I can see that it is calling ECSTaskCredentials.GenerateNewCredentials(), but this is not running on ECS

2023-05-09 14:20:33.668 +00:00 [ERR] AWS ERROR: **Unable to retrieve credentials. Message = "Unable to reach credentials server"**.
Amazon.Runtime.AmazonServiceException: Unable to retrieve credentials. Message = "Unable to reach credentials server".
   at Amazon.Runtime.ECSTaskCredentials.GenerateNewCredentials()
   at Amazon.Runtime.RefreshingAWSCredentials.<GenerateNewCredentialsAsync>b__16_0()
   at System.Threading.Tasks.Task1.InnerInvoke()
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Amazon.Runtime.RefreshingAWSCredentials.GetCredentialsAsync()
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Kralizek.Extensions.Configuration.Internal.SecretsManagerConfigurationProvider.FetchAllSecretsAsync(CancellationToken cancellationToken)
   at Kralizek.Extensions.Configuration.Internal.SecretsManagerConfigurationProvider.FetchConfigurationAsync(CancellationToken cancellationToken)
   at Kralizek.Extensions.Configuration.Internal.SecretsManagerConfigurationProvider.LoadAsync()
   at Kralizek.Extensions.Configuration.Internal.SecretsManagerConfigurationProvider.Load()
   at Microsoft.Extensions.Configuration.ConfigurationManager.AddSource(IConfigurationSource source)
   at Microsoft.Extensions.Configuration.ConfigurationManager.Microsoft.Extensions.Configuration.IConfigurationBuilder.Add(IConfigurationSource source)

Any help provided would be greatly appreciated beyond measure.

[Kralizek]

Hi, I think that you are enabling some ECS specific path in the AWS SDK by adding the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI variable (see: https://docs.aws.amazon.com/sdkref/latest/guide/feature-container-credentials.html).

Did you add a proper instance profile to your EC2 instance?

[kron0s19]

As it turns out, yes, the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI was creating an issue.

I fixed it by actually settings the credentials by instantiating InstanceProfileAWSCredentials

AWSCredentials? credentials = IsAWS() ? new InstanceProfileAWSCredentials() : null;

        configurationManager.AddSecretsManager(credentials, Region, options =>
        {
            var prefixes = new[] { $"{environmentName}_{applicationName}", applicationName };
            options.SecretFilter = entry => prefixes.Any(prefix => entry.Name.StartsWith(prefix));
            options.KeyGenerator = (_, s) => FormatKey(s, environmentName, applicationName);
            options.PollingInterval = PollingInterval;
        });