Nomad is not able to pull from DockerHub

[ThomasObenaus]

When trying to deploy a docker image from docker-hub nomad responds with the following error message:

failed to initialize task "ping_service_task" for alloc "8f46a473-90de-3e96-71bb-149ad2916453": Failed to find docker auth for repo "thobe/ping_service": docker-credential-ecr-login with input "thobe/ping_service" failed with stderr: credentials not found in native keychain

Example job file:

# job>group>task>service
# container for tasks or task-groups that nomad should run
job "ping_service" {
  datacenters = ["public-services"]
  #,"private-services","content-connector","backoffice"]
  type = "service"

  meta {
    my-key = "example"
  }

  # The group stanza defines a series of tasks that should be co-located on the same Nomad client.
  # Any task within a group will be placed on the same client.
  group "ping_service_group" {
    count = 1

    # restart-policy
    restart {
      attempts = 10
      interval = "5m"
      delay = "25s"
      mode = "delay"
    }

     ephemeral_disk {
      migrate = false
      size    = "50"
      sticky  = false
    }

    # The task stanza creates an individual unit of work, such as a Docker container, web application, or batch processing.
    task "ping_service_task" {
      driver = "docker"
      config {
        # Docker Hub:
        image = "thobe/ping_service:0.0.9"
      }

      logs {
        max_files     = 2
        max_file_size = 10
      }

      config {
        port_map = {
          http = 8080
        }
      }

      resources {
        cpu    = 100 # MHz
        memory = 20 # MB
        network {
          mbits = 10
          port "http" {
          }
        }
      }

      # The service stanza instructs Nomad to register the task as a service using the service discovery integration
      service {
        name = "ping-service"
        tags = ["urlprefix-/ping"] # fabio
        port = "http"
        check {
          name     = "Ping-Service Alive State"
          port     = "http"
          type     = "http"
          method   = "GET"
          path     = "/ping"
          interval = "10s"
          timeout  = "2s"
        }
       }

      env {
        SERVICE_NAME        = "${NOMAD_DC}",
        PROVIDER            = "ping-service",
        # uncomment to enable sd over consul
        CONSUL_SERVER_ADDR  = "172.17.0.1:8500"
        #PROVIDER_ADDR = "ping-service:25000"
      }
    }
  }
}

[MatthiasScholz]

This should be kind of expected since the AIM configures the credentials to use the AWS ECR.

[ThomasObenaus]

The goal is to support also to pull from public repos like docker hub. This can be done if docker is configured to use credHelper: ecr-login instead of credStore: ecr-login. Doing this only for the docker pull calls against the concrete ecr url the ecr-login credential helper will be used. For the calls against a public repo from docker hub no credential helper will be used and the pull call succeeds.

I tested it with:

{
    "credHelpers": {
    "<my_aws_accound_id>.dkr.ecr.us-east-1.amazonaws.com": "ecr-login"
    }
}

It worked for: docker pull <my_aws_accound_id>.dkr.ecr.us-east-1.amazonaws.com/ping-service and docker pull thobe/ping_service:0.0.9

[ThomasObenaus]

The only open question is: How to inject the correct aws account id into the /etc/docker/config.json in an elegant way.

Mby using cloud-init/ user-data

[MatthiasScholz]

Did you check if wildcards are supported? It might be possible to do something like this: "*.awazonaws.com": "ecr-login

[MatthiasScholz]

If wildcards are not supported, I would rather recommend to migrated docker hub images to ECR than to hand in the AWS account id. Furthermore doing a migration might reduce the versatility, but will increase the control about which images are used in the cluster.

Hint: There is a script which could easily be adapted to support docker hub as well.

[ThomasObenaus]

Solved with #52