Support assuming IAM roles for ECR

[smithhannahm]

I have created an AWS credentials file that defines the AWS user and role to assume. However, it gives me an error that says that there are missing modules for role assumption.

[cd.go.artifact.docker.registry] Failed to publish Artifact[id=go-contacts-version, storeId=pe_nonprod_ecr, artifactPlanConfig={"Image":"go-contacts","Tag":"gocd-v${GO_PIPELINE_LABEL}"}]: com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)), SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey), com.amazonaws.auth.profile.ProfileCredentialsProvider@48ae0f01: To use assume role profiles the aws-java-sdk-sts module must be on the class path., com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@5d5cbf17: Unable to load credentials from service endpoint]

The ability to assume a role should be included in this plugin, as roles are the main way that we control resource access in our AWS accounts.

[varshavaradarajan]

@smithhannahm - where did you create the aws credential file? The error that you see comes from the gocd agent. So, the agent must have the aws keys.

if the aws keys are not configured in the artifact store, the default aws credential chain is used - https://github.com/gocd/docker-registry-artifact-plugin/blob/master/src/main/java/cd/go/artifact/docker/registry/AWSTokenRequestGenerator.java#L66 and https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/DefaultAWSCredentialsProviderChain.html

[smithhannahm]

I create it at build time on the agent using a secret I'm pulling from a secret store.

            - exec:
                command: sh
                arguments:
                - -c
                - mkdir $HOME/.aws; echo "$AWS_CREDS" > $HOME/.aws/credentials

The credential file is as follows:

[default]
role_arn = arn:aws:iam::12345:role/@MyRoleName
source_profile = pe_nonprod

[pe_nonprod]
aws_access_key_id=xxxxx
aws_secret_access_key=xxxxxxx

The AWS credentials are not specified in the artifact store definition, and the error message I'm receiving shows that it's at least trying to use the assumed role, but it needs a different class.

... To use assume role profiles the aws-java-sdk-sts module must be on the class path., com.amazonaws.auth.EC2ContainerCredentialsProviderWrapper@5d5cbf17: Unable to load credentials from service endpoint]

[varshavaradarajan]

@smithhannahm - I'd expect the ProfileCredentialsProvider to be used and not EC2ContainerCredentialsProviderWrapper. But from your errors, the ProfileCredentialsProvider needs the sts dependency. Can you please build the plugin with #146 and check or wait for someone to confirm it?

Edit: Looks like ProfileCredentialsProvider _is_ used and shows the sts error

[arvindsv]

@smithhannahm I thought about setting up the required accounts to try our @varshavaradarajan's change and decided that it's going to be easier to let you try it yourself, since you have it set up already. :)

I've built and uploaded a plugin with @varshavaradarajan's changes here: https://www.dropbox.com/sh/82lij8l87zkzrfg/AADrY8Z82iZmdoONpRMbhHPna?dl=0

Do you mind trying it and letting us know? If it works, then I can merge that PR and we can make an official release. Thank you!

[arvindsv]

Of course, if you prefer, please build it yourself and try it. All you need to do is to check out that PR and run ./gradle/assemble and you'll find the plugin in the build/libs directory.

[smithhannahm]

So, with the updated plugin from @arvindsv 's dropbox, I'm now getting the following error:

[cd.go.artifact.docker.registry] unauthorized: incorrect username or password
[cd.go.artifact.docker.registry] Failed to publish Artifact[id=go-contacts-version, storeId=pe_nonprod_ecr, artifactPlanConfig={"Image":"go-contacts","Tag":"gocd-v${GO_PIPELINE_LABEL}"}]: com.spotify.docker.client.exceptions.DockerException: java.lang.RuntimeException: unauthorized: incorrect username or password

This is without an access key/secret access key specified in the artifact registry config, and pulling an AWS credentials file onto the agent at the default location

[arvindsv]

@smithhannahm Ok. I'll set this up and take a look. Can you make sure that the access key and secret key are correct? Maybe by getting on to the agent node or cat-ing the file and trying it yourself, maybe?

[smithhannahm]

They are correct. I am able to manually push the images through the CLI with the same setup.

[arvindsv]

@smithhannahm I spent a while on this. It doesn't seem related to STS. Can you try this please?

1. Tag your docker image with the name: 123456789.dkr.ecr.us-east-1.amazonaws.com/go-contacts:v1 (manually for this test, if needed).

2. Configure the image in GoCD's artifact config as: 123456789.dkr.ecr.us-east-1.amazonaws.com/go-contacts.

... where 123456789 is the AWS account ID and there is a region in that as well. This works, right?

Which leads me to believe it is not a problem with STS, but is some kind of a configuration quirk. We should better document a working AWS ECR config.

cc: @varshavaradarajan: Is that prefix really needed?

[arvindsv]

Actually, let me try that once more. I had done an aws ecr get-login before and my credentials would have been stored.

[arvindsv]

Hmm. No. They don't seem to be.

[arvindsv]

The docker client code is logging in properly: https://github.com/gocd/docker-registry-artifact-plugin/blob/0ebe7f07670df5c321f01179d959a73729d95606/src/main/java/cd/go/artifact/docker/registry/DockerClientFactory.java#L40

So, please try what I mentioned above and if that works, we can see what a proper AWS ECR config looks like.

[smithhannahm]

@arvindsv Following your steps worked. The final configuration is as follows:

        jobs:
          Docker:
            timeout: 0
            elastic_profile_id: dind
            artifacts:
              - external:
                  id: go-contacts-version
                  store_id: pe_nonprod_ecr
                  configuration:
                    options:
                      Image: 1234567890.dkr.ecr.us-east-1.amazonaws.com/go-contacts
                      Tag: gocd-v${GO_PIPELINE_LABEL}
            tasks:
            - exec:
                working_directory: src
                command: sh
                arguments:
                - -c
                - docker build -t 1234567890.dkr.ecr.us-east-1.amazonaws.com/go-contacts:gocd-v${GO_PIPELINE_LABEL} .

I had assumed that since the artifact store configuration asked for both the region and account number, it was doing some sort of configuration on the backend to create the standard ecr repository name.

[arvindsv]

@smithhannahm Thank you for checking. Yes, that was my assumption too, that we don't need those details in the image.

We should see if we can do some sort of configuration (as you mentioned) to make that transparent. However, docker itself seems to use that to decide which registry to use.

[smithhannahm]

@arvindsv There is a tool called trebuchet that does something similar as a command line tool (which was what I was using to manually push my image to check that the credentials I was importing worked)

[arvindsv]

@smithhannahm Thanks. Yes, I think the difference is that trebuchet uses AWS APIs which allow it to pass in the right information. But, this plugin uses a docker client library which seems to (nearly) wrap around the docker command-line client and it seems to need th docker images tagged with the name of the ECR registry.

@varshavaradarajan has submitted a PR to update the docs. I'll take a look and update it.