Facing issue in configuring AWS ECR as a registry using credential helper for Nebula worker.

[Sharvin26]

Hello

I have configured a nebula worker on the Raspberry. I am using AWS ECR as a registry to store the Images. The AWS ECR dynamically updates the auth password every 12 hours. I can't update this password every time at the worker. So I have configured AWS credential helper which automatically updates the auth password every 12 hours on the edge device.

Expected/Wanted Behavior

Whenever I push the update, the worker will pull new image from AWS ECR.

Actual Behavior

It is working perfectly when I add REGISTRY_AUTH_USER and REGISTRY_AUTH_PASSWORD manually every 12 hours. The worker is able to pull the update from the AWS ECR registry.

But now when I have configured the AWS ECR credential helper the nebula work is unable to pull the Image. To test if my AWS ECR credential helper is working properly I tried the command docker pull <my_registry_url>/<image_name> and it worked. Note: I also tried this command after 12 hours when my auth_password became invalid and it still worked.

I have added worker docker-compose.yml and the worker logs for the reference purpose =>

docker-compose.yml =>

version: '3'
services:
  worker:
    container_name: worker
    build:
      context: .
      dockerfile: Dockerfile
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    restart: unless-stopped
    hostname: worker
    environment:
      REGISTRY_HOST: < my_regisrty_url >
      MAX_RESTART_WAIT_IN_SECONDS: 0
      NEBULA_MANAGER_AUTH_USER: nebula
      NEBULA_MANAGER_AUTH_PASSWORD: nebula
      NEBULA_MANAGER_HOST: < my_manager_url >
      NEBULA_MANAGER_PORT: 80
      NEBULA_MANAGER_PROTOCOL: http
      NEBULA_MANAGER_CHECK_IN_TIME: 30
      DEVICE_GROUP: test
      KAFKA_BOOTSTRAP_SERVERS: < my_manager_url >:9092
      KAFKA_TOPIC: nebula-reports

worker logs =>

Creating network "nebula_worker_default" with the default driver
Creating worker ... done
Attaching to worker
worker    | reading config variables
worker    | /usr/local/lib/python3.7/site-packages/parse_it/file/file_reader.py:55: UserWarning: config_folder_location does not exist, only envvars & cli args will be used
worker    |   warnings.warn("config_folder_location does not exist, only envvars & cli args will be used")
worker    | reading config variables
worker    | created a bridge type network named nebula
worker    | no registry user pass combo defined, skipping registry login
worker    | checking nebula manager connection
worker    | nebula manager connection ok
worker    | stopping all preexisting nebula managed app containers in order to ensure a clean slate on boot
worker    | initial start of <my_image> app
worker    | pulling image <my_registry_url>/<my_image>:latest
worker    | <my_registry_url>/<my_image>
worker    | 500 Server Error: Internal Server Error ("Get https://<my_registry_url>/v2/<my_image>/manifests/latest: no basic auth credentials")
worker    | problem pulling image <my_registry_url>/<my_image>:latest

Steps to Reproduce the Problem

1. I have configured a worker from this repository
2. I have configured the AWS ECR credential helper from this repository https://github.com/awslabs/amazon-ecr-credential-helper Note: I have used make docker command with the flag TARGET_GOARCH=arm
3. I have configured the ~/.docker/config.json as follows =>

{
    "credHelpers": { "<my_registry_url>": "ecr-login" }
}

[issue-label-bot[bot]]

Issue-Label Bot is automatically applying the label question to this issue, with a confidence of 0.61. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

[naorlivne]

The reason your seeing issues is likely because that while you configured ~/.docker/config.json you did so on the host, not on the worker container - you'll need to mount the ~/.docker/config.json onto the same path inside the worker container for it to be able to see your configured config.json (read only will likely be enough but not 100% sure about it).

There's a bit more info about it on https://nebula.readthedocs.io/en/latest/config/worker/ on the registry_auth_user & registry_auth_password description fields.

[Sharvin26]

Hello @naorlivne

Thanks for the response. I mounted ~/.docker/config.json onto the same path inside the worker container. But I am getting this error =>

worker    | reading config variables
worker    | /usr/local/lib/python3.7/site-packages/parse_it/file/file_reader.py:55: UserWarning: config_folder_location does not exist, only envvars & cli args will be used
worker    |   warnings.warn("config_folder_location does not exist, only envvars & cli args will be used")
worker    | reading config variables
worker    | no registry user pass combo defined, skipping registry login
worker    | checking nebula manager connection
worker    | nebula manager connection ok
worker    | stopping all preexisting nebula managed app containers in order to ensure a clean slate on boot
worker    | initial start of <my-image> app
worker    | pulling image <my_registry_url>/<my-image>:latest
worker    | <my_registry_url>/<my-image>
worker    | problem pulling image <my_registry_url>/<my-image>:latest
worker    | docker-credential-ecr-login not installed or not available in PATH

This is my docker-compose.yml =>

version: '3'
services:
  worker:
    container_name: worker
    build:
      context: .
      dockerfile: Dockerfile
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /root/.docker/config.json:/root/.docker/config.json
    restart: unless-stopped
    hostname: worker
    environment:
      REGISTRY_HOST: < my_regisrty_url >
      MAX_RESTART_WAIT_IN_SECONDS: 0
      NEBULA_MANAGER_AUTH_USER: nebula
      NEBULA_MANAGER_AUTH_PASSWORD: nebula
      NEBULA_MANAGER_HOST: < my_manager_url >
      NEBULA_MANAGER_PORT: 80
      NEBULA_MANAGER_PROTOCOL: http
      NEBULA_MANAGER_CHECK_IN_TIME: 30
      DEVICE_GROUP: test
      KAFKA_BOOTSTRAP_SERVERS: < my_manager_url >:9092
      KAFKA_TOPIC: nebula-reports

what am I doing wrong here?

[naorlivne]

worker | docker-credential-ecr-login not installed or not available in PATH seems like the important line to me, I admit to not being a big fan of ECR exectly due to headaches like this so I may be wrong but it seems like it requires access to the ECR codebase rather then just the config.json (which is not part of the container but rather the host again) making this related to https://github.com/awslabs/amazon-ecr-credential-helper/issues/56

Seems like you'll have to install the Amazon ECR Docker Credential Helper & configure it (including the AWS access keys) inside the worker container to get it to work.

Possibly helpful link - https://serverfault.com/questions/897636/how-to-add-amazon-ecr-credential-helper-to-path

[Sharvin26]

Hello, @naorlivne Thanks for the response. Is there a better alternative for ECR that I can integrate with the Nebula worker.

Note: I have already tried docker registry open source. But I am looking for a solution which has a GUI which makes easy in the management of the releases (i.e. Images ). As ECR was suitable for that use case so I went with the ECR.

[naorlivne]

Personally I use docker hub as it's managed by Docker Inc but any registry that supports the standard docker login process will work

[Sharvin26]

Thanks, I'll check it.