Doesn't play nice with `docker-compose`

[jjmilburn]

I'm likely missing something, but this plugin doesn't appear to play nice with the docker-compose plugin. I'm basically trying to replace a previous use of the Docker Login plugin with ECR (moved a docker image from Docker Hub to a private AWS ECR).

        - ecr#v2.0.0:
            login: true
            account_ids: "<my_aws_account_id>"
            no-include-email: true
            region: us-east-1
        - docker-compose#v2.5.1:
            # Intentionally don't include image-repository, as we will
            # manually update the Docker image to use on the ECR as needed.     
            config: buildkite/docker-compose.yml
            build: <some_name>
            cache-from: <my_aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/<image-name>:latest

From the logs, I can see that the ECR plugin successfully logs in (so awscli and the required AWS keys are OK). This is indicated by "Login Succeeded" in the "Authenticating with AWS ECR to " log item.

However, we then hit ""pull access denied for latest, repository does not exist or may require 'docker login'"

I would have thought that this plugin replaced the need for docker login, but perhaps that isn't the case?

[lox]

That is very odd @jjmilburn! The ecr plugin basically just calls docker login under the hood and should work exactly as you describe 🤔 My first thought is that for some reason the account id's or regions aren't matching up between the two, but some troubleshooting is needed.

Mind dropping us a link to the failing build to support@buildkite.com and we will have a look?

[jjmilburn]

Will do, thank you!

@lox I dropped an email to support@buildkite.com and can continue to engage there or here - either way.

[orien]

I've seen this behavior when the AWS role used by the Buildkite agent doesn't have access to the AWS ECR: the login succeeds, but the cannot access the repository.

Worth a look to see if it's the cause of your problem.

[TSMMark]

EDIT: PEBCAK. I was missing the leading ci: in front of cache-from: <ACCOUNT>.dkr.ecr.us-west-2.amazonaws.com/d2-ci:latest. It should have been cache-from: ci:<ACCOUNT>.dkr.ecr.us-west-2.amazonaws.com/d2-ci:latest

---

Original post below

I'm having this same/similar issue. For the record, the IAM role has been granted full ECR access, as a sanity check, and still does not pull from ECR, although it pushes to ECR totally fine...

Here's the relevant part of my pipeline.yml:

steps:
  - label: ":docker: Build an image"
    plugins:
      - ecr#v2.1.1:
          login: true
          region: us-west-2
          no-include-email: true
      - docker-compose#v3.7.0:
          build: ci
          image-repository: <ACCOUNT>.dkr.ecr.us-west-2.amazonaws.com/d2-ci
          cache-from: <ACCOUNT>.dkr.ecr.us-west-2.amazonaws.com/d2-ci:latest
          push:
            - ci:<ACCOUNT>.dkr.ecr.us-west-2.amazonaws.com/d2-ci
            - ci:<ACCOUNT>.dkr.ecr.us-west-2.amazonaws.com/d2-ci:latest

AFAICT it should be working but it fails to pull from the repo, even tho the images are there and tagged latest. Here's the log of successful ECR login:

Logs successful ECR login:
Running plugin ecr environment hook
[2020-10-16T14:35:18Z] $ var/lib/buildkite-agent/plugins/github-com-buildkite-plugins-ecr-buildkite-plugin-v2-1-1/hooks/environment
[2020-10-16T14:35:19Z] WARNING! Your password will be stored unencrypted in /var/lib/buildkite-agent/.docker/config.json.
[2020-10-16T14:35:19Z] Configure a credential helper to remove this warning. See
[2020-10-16T14:35:19Z] https://docs.docker.com/engine/reference/commandline/login/#credentials-store
[2020-10-16T14:35:19Z]
[2020-10-16T14:35:19Z] Login Succeeded

But then the ECR pull fails, saying I may need docker login:

:docker: Pulling cache image for <ACCOUNT>.dkr.ecr.us-west-2.amazonaws.com/d2-ci
[2020-10-16T14:35:24Z] $ docker pull latest
[2020-10-16T14:35:25Z] Using default tag: latest
[2020-10-16T14:35:26Z] Error response from daemon: pull access denied for latest, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
[2020-10-16T14:35:26Z] Exited with 1
[2020-10-16T14:35:26Z] !!! :docker: Pull failed. latest will not be used as a cache for <ACCOUNT>.dkr.ecr.us-west-2.amazonaws.com/d2-ci

The repo + image definitely exist in ECR, and are in the us-west-2 region: <ACCOUNT>.dkr.ecr.us-west-2.amazonaws.com/d2-ci:latest

As @lox said, I should not need a to define a separate docker login plugin, since ecr should be using it under the hood anyway. Also I'm able to push to ECR fine, confirming that ecr login is working, at least in that case.

Any advice would be a huge help, thank you in advance.