support for docker credentials helpers

[gadams00]

I'm trying to use watchtower with an image hosted on ECR in AWS. ECR auth tokens expire every 12 hours, but there's a credentials helper that uses cli credentials or AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables to automatically refresh authentication when docker pull is used. I tried creating my own docker image from watchtower, adding the ecr credential helper, with Dockerfile:

FROM v2tec/watchtower
ADD docker-credential-ecr-login /bin/
ADD config.json /

config.json:

{
    "credHelpers": {
        "{aws_account_id}.dkr.ecr.us-west-2.amazonaws.com": "ecr-login"
    }
}

docker-credential-ecr-login was built based on https://github.com/awslabs/amazon-ecr-credential-helper.

I then run watchtower via docker-compose:

version: "3"
services:
  my-image:
    image: {aws_account_id}.dkr.ecr.us-west-2.amazonaws.com/my-image:latest
  watchtower:
    image: ecr-watchtower
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      AWS_ACCESS_KEY_ID: "{access_key}"
      AWS_SECRET_ACCESS_KEY: "{secret_key}"
    command: --interval 30 --debug

watchtower doesn't seem to respect the credential helper. logs:

watchtower_1  | time="2017-09-14T06:31:06Z" level=debug msg="Pulling {aws_account_id}.dkr.ecr.us-west-2.amazonaws.com/my-image:latest for /watchtower_my-image_1"
watchtower_1  | time="2017-09-14T06:31:06Z" level=debug msg="No credentials for {aws_account_id}.dkr.ecr.us-west-2.amazonaws.com/my-image:latest in /config.json"
watchtower_1  | time="2017-09-14T06:31:06Z" level=debug msg="No authentication credentials found for {aws_account_id}.dkr.ecr.us-west-2.amazonaws.com/my-image:latest"
watchtower_1  | time="2017-09-14T06:31:06Z" level=debug msg="Error pulling image {aws_account_id}.dkr.ecr.us-west-2.amazonaws.com/my-image:latest, Error response from daemon: Get https://{aws_account_id}.dkr.ecr.us-west-2.amazonaws.com/v2/my-image/manifests/latest: no basic auth credentials"
watchtower_1  | time="2017-09-14T06:31:06Z" level=info msg="Unable to update container /watchtower_my-image_1. Proceeding to next."
watchtower_1  | time="2017-09-14T06:31:06Z" level=debug msg="Error response from daemon: Get https://{aws_account_id}.dkr.ecr.us-west-2.amazonaws.com/v2/my-image/manifests/latest: no basic auth credentials"
watchtower_1  | time="2017-09-14T06:31:06Z" level=debug msg="Scheduled next run: 2017-09-14 06:31:35 +0000 UTC"

[ingwinlu]

same goes for googles equivalent helper gcr.

btw you shouldn't need to build a new image to get this functionality just mounting the credential helper into $PATH as well as setting $HOME it knows where to lookup the config should be enough. See the following snipped on how I made it work for compose:

docker run --name egm-compose -v /var/run/docker.sock:/var/run/docker.sock -v "/home/egm/easyguestmanagement:/rootfs/easyguestmanagement" -w="/rootfs/easyguestmanagement" -e HOME=/rootfs/easyguestmanagement -v /usr/bin/docker-credential-gcr:/bin/docker-credential-gcr docker/compose:1.16.1 up

[gabormay]

FYI For a workaround/alternative setup for getting watchtower work with ECR see my setup at https://github.com/v2tec/watchtower/issues/38#issuecomment-331419620

[shadycuz]

@gabormay But if you are using cron and get login you don't need the credential helper or json? right? The whole point of the cred helper is to never have to run get login

[gabormay]

@shadycuz Correct, in my setup I did not use the credential helper.

Btw, I would assume that the credential helper has to do a 'get login' or equivalent API call anyway, but indeed it would run it 'on-demand' as opposed to you (or cron) having to run it regularly.

[andyault]

I was able to get watchtower to play nice with ECR by following a similar approach (full explanation here), but I wasn't able to get my watchtower fork to see my AWS credentials until I changed it to start from alpine instead of from scratch. Once I changed it, I was able to use both environment variables and the ~/.aws/credentials file to authenticate.

[simskij]

adding a credential helper for aws ecr would, in my opinion, mean we'd need to add credential helpers for other popular cloud providers as well. we've yet to decide whether that is actually within the scope of watchtower or if some version of @gabormay's approach should be the recommended way of doing it.

care to weigh in anyone? how do you suggest we solve this in a vendor generic manner?

[koreno]

adding a credential helper for aws ecr would, in my opinion, mean we'd need to add credential helpers for other popular cloud providers as well. we've yet to decide whether that is actually within the scope of watchtower or if some version of @gabormay's approach should be the recommended way of doing it.

care to weigh in anyone? how do you suggest we solve this in a vendor generic manner?

you don't need to add specific support for each helper. from what I understand a .docker/config.json tells you which helper to use for each repository, and the helper's API is supposed to be consistent. ultimately, if you're using a docker client all of that should be handled for you...

[simskij]

Awesome! Would you be interested in having a stab at implementing this?

[koreno]

It seems like someone already gave it a shot - https://github.com/archetypesc/watchtower-ecr/blob/master/armhf/Dockerfile - however I couldn't get it to work either...

I've finally found pyouroboros/ouroboros which seems to cover this issue if you provide it with the creds helper and proper configuration:

  ouroboros:
    container_name: ouroboros
    image: pyouroboros/ouroboros
    restart: unless-stopped
    environment:
      - LOG_LEVEL=debug
      - SELF_UPDATE=true
      - INTERVAL=300
      - AWS_REGION=eu-west-1  # required for docker-credential-ecr-login
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /bin/docker-credential-ecr-login:/bin/docker-credential-ecr-login
      - $HOME/.aws:/root/.aws 
      - $HOME/.docker/config.json:/root/.docker/config.json  # contains '{"credsStore": "ecr-login"}'

[simskij]

I've been able to get it to work in a dev environment. As soon as I get around to it I'll try to integrate it into the image. However, that will likely not be until august.

[simskij]

I'm very happy to announce that this has been solved - and with no modifications to watchtower on top of that!

Documentation on how to do this is available at https://github.com/containrrr/watchtower/blob/master/docs/credential-helpers.md

Thanks for your patience!

[simskij]

It seems like someone already gave it a shot - https://github.com/archetypesc/watchtower-ecr/blob/master/armhf/Dockerfile - however I couldn't get it to work either...

I've finally found pyouroboros/ouroboros which seems to cover this issue if you provide it with the creds helper and proper configuration: [...]

If you found a solution that works for you, then that's great! For me, and many others I've talked with, the size of the ouroboros image is way to large to make it usable in any real world scenarios.

Compared to watchtower, ouroboros weighs in at 7 times the image size, which quickly adds up to a large network utilization considering that image diffing at the moment's only possible by downloading the image and comparing the checksum.

[chander]

If it makes a difference, I had the exact same issue as the OP here. I was able to fix it by adding 👍

"credStore": "ecr-login"

That one line in config.json fixed it all for me. Now it works like a champ.

To be more complete, this is a shell function I use to get the helper setup:

function build_credential_helper {
  # Build the credential helper on a volume to use
  cat -<< EOF > Dockerfile
  FROM golang:latest

  ENV CGO_ENABLED 0
  ENV REPO github.com/awslabs/amazon-ecr-credential-helper/ecr-login/cli/docker-credential-ecr-login

  RUN go get -u \$REPO

  RUN rm /go/bin/docker-credential-ecr-login

  RUN go build \
    -o /go/bin/docker-credential-ecr-login \
    /go/src/\$REPO

  WORKDIR /go/bin/

EOF

  docker volume create helper
  docker build -t aws-ecr-dock-cred-helper .

  docker run  -d --rm --name aws-cred-helper --volume helper:/go/bin aws-ecr-dock-cred-helper

  mkdir -p ~/.docker
  cat -<< EOF > ~/.docker/config.json
  {
    "credsStore" : "ecr-login",
    "HttpHeaders" : {
      "User-Agent" : "Docker-Client/19.03.1 (XXXXXX)"
    },
    "auths" : {
      "<AWS_ACCOUNT_ID>.dkr.ecr.us-west-1.amazonaws.com" : {},
      "<AWS_ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com": {}
    },
    "credHelpers": {
      "<AWS_ACCOUNT_ID>.dkr.ecr.us-west-1.amazonaws.com" : "ecr-login",
      "<AWS_ACCOUNT_ID>.dkr.ecr.us-east-1.amazonaws.com" : "ecr-login"
    }
  }
EOF

  rm Dockerfile
}

Once that is complete, here is the snippet I use in my docker-compose.yaml file to start it:

version: "3.4"
services:
  # Check for new images and restart things if a new image exists
  # for any of our containers.
  watchtower:
    image: containrrr/watchtower:latest
    command: --interval 30
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - .docker/config.json:/config.json
      - helper:/go/bin
    environment:
      - HOME=/
      - PATH=$PATH:/go/bin
      - AWS_REGION=us-west-1
volumes:
  helper: 
    external: true

A few things to note here:

1. With docker-compose the volume (helper, in this case) MUST be set to external: true, otherwise docker-compose will preface it with the directory name.
2. Note that "credsStore" : "ecr-login" is needed - and in theory if you have that you can remove the credHelpers section (in my case, credHelpers didn't really do anything.)
3. I have this running on an EC2 instance that has credentials assigned to it - so no keys are needed; however, you may need to include the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables as well.
4. An alternative to adding the various variables is to create a ~/.aws/config and ~/.aws/credentials files and place the settings there, then mount the ~/.aws directory to / in the container.

[simskij]

@chander Thank you for that explanation! I'll go ahead and lock this issue to prevent further discussions that might get lost in the ether, but if you don't mind, it'd be really great if you were to provide a PR that adds this to the docs.