GetCallerIdentityAsync throws an exception when calling it from a docker container

[tectac]

Hi,

In AWS CodeBuild, you have the ability to build your code in a Docker container. To propagate the credentials to your container, you need to pass few extra parameters:

docker build --build-arg AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION --build-arg AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI .

To validate this config, I ran aws sts get-caller-identity and got this response:

Step 9/31 : RUN aws sts get-caller-identity 
 ---> Running in 080c749553b9 
{ 
    "Account": "123456789",  
    "UserId": "AROXXXXXXX:AWSCodeBuild-(...)",  
    "Arn": "arn:aws:sts::123456789:assumed-role/codebuild-core-srv-pricing-service-role/AWSCodeBuild-(...)" 
}

When running sleet in this environment, I get the following exception:

[System.ArgumentException] Failed to determine AWS identity - ensure you have an IAM role set, have set up default credentials or have specified a profile/key pair.

Unfortunately we do no have much details regarding the exception raised by the AWS client. Would be great to add a exception.ToString() so we can troubleshoot.

I'm not sure why we are getting this error but wan we try adding a parameter called --inherit-permissions bypassing this check?

[emgarten]

Set the env var SLEET_DEBUG=1 and it should give you the full exception.

[seroche]

(used the wrong GitHub account to create my issue...)

Added this to my dockerfile:

Seems it is not generating more detailed logs:

I'm not sure we can retrieve more details about this error since you are not re-throwing the initial exception.

[emgarten]

Thanks for the logs @seroche. I've made a change to include the inner exception from AWS in the argument exception, and made a fix for SLEET_DEBUG which was getting ignored.

https://github.com/emgarten/Sleet/pull/93

[seroche]

Great. Sounds good. Let me know when you push a new version :)

[emgarten]

The latest logging changes are in https://www.nuget.org/packages/Sleet/3.0.7

[seroche]

It seems to be caused by the async method ran synchronously? Or an exception is thrown during the process and it causes the async method call to be cancelled.

Should try Task.Run(() => _client.GetCallerIdentityAsync(new GetCallerIdentityRequest { })).GetAwaiter().GetResult()

We could also bypass this check and let the SDK deal with the credentials?

System.ArgumentException: Failed to determine AWS identity - ensure you have an IAM role set, have set up default credentials or have specified a profile/key pair. 
 ---> System.AggregateException: One or more errors occurred. (A task was canceled.) 
 ---> System.Threading.Tasks.TaskCanceledException: A task was canceled. 
   --- End of inner exception stack trace --- 
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification) 
   at Sleet.FileSystemFactory.CreateFileSystem(LocalSettings settings, LocalCache cache, String source) in D:\a\1\s\src\SleetLib\FileSystem\FileSystemFactory.cs:line 25 
   --- End of inner exception stack trace --- 
   at Sleet.FileSystemFactory.CreateFileSystem(LocalSettings settings, LocalCache cache, String source) in D:\a\1\s\src\SleetLib\FileSystem\FileSystemFactory.cs:line 138 
   at Sleet.Util.CreateFileSystemOrThrow(LocalSettings settings, String sourceName, LocalCache cache) in D:\a\1\s\src\Sleet\Util.cs:line 27 
   at Sleet.PushAppCommand.<>c__DisplayClass1_0.<<Run>b__0>d.MoveNext() in D:\a\1\s\src\Sleet\PushAppCommand.cs:line 60 
--- End of stack trace from previous location where exception was thrown --- 
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.<>c__DisplayClass90_0.<OnExecute>b__0() 
   at McMaster.Extensions.CommandLineUtils.CommandLineApplication.Execute(String[] args) 
   at Sleet.Program.MainCore(String[] args, ILogger log) in D:\a\1\s\src\Sleet\Program.cs:line 62 
System.Threading.Tasks.TaskCanceledException: A task was canceled. 
[System.ArgumentException] Failed to determine AWS identity - ensure you have an IAM role set, have set up default credentials or have specified a profile/key pair. 

[emgarten]

FileSystemFactory.CreateFileSystem should be changed to an async method in my opinion. Running this as a task may work around the issue also, but the reason this method wasn't async before is that before the aws check there were async operations going on.

Any ideas why this behaves differently in a docker build?

[seroche]

I assume that's related to the fact that we are building code in docker container hosted in another docker container. To make it work (sts get-caller-identity), we need to pass extra params to the "docker build" command.

To me authentication should be handled by the AWS SDK client itself. If I remember correctly it checks credentials OR check profile OR check IAM role (...). In other words, if the environment is not correct it should throw an exception.

What do you think?

[emgarten]

Makes sense. I'm just wondering why a console app would have trouble with threads like this.

I'm happy to accept a PR to make FileSystemFactory.CreateFileSystem async to try and resolve this, or I can take a look at updating this next week.

[seroche]

Let me have a look on this.

[seroche]

I'm going to open a PR soon so you can have a look asap :). I have changed the "S3 part" quite a bit so I will need some more times to go through the tests.

[tectac]

Fixed by PR

[seroche]

Can confirm it's working well when executing sleet in a docker container.

Don't forget to include the following lines in your dockerfile:

# Build and deploy Domain NuGet packages
FROM mcr.microsoft.com/dotnet/core/sdk:3.0-alpine3.9 AS base
ARG AWS_DEFAULT_REGION
ARG AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

ENV AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI
ENV AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION

Then call docker build: docker build --build-arg AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION --build-arg AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI .