Closed rokka-n closed 7 years ago
You should either specify auth using the auth block https://www.nomadproject.io/docs/drivers/docker.html#authentication or put it in a file and configure Nomad to read from that file: https://www.nomadproject.io/docs/drivers/docker.html#docker_auth_config
Hi Alex,
We went with option 2, and that's where it fails.
First option works just fine, but we're trying to avoid it since ecr credentials are ephemeral and will cause confusion for developers when expired.
/etc/nomad.d/client.hcl
client {
enabled = true
node_class = "Linux"
client_max_port = 15000
options {
"docker.auth.config" = "/root/.docker/config.json"
"docker.cleanup.image" = "0"
"driver.raw_exec.enable" = "1"
}
meta {
region = "us"
machine_type = "m3.medium"
machine_function = "nomad-client"
}
}
logs of nomad client, I don't see the line "Failed to find docker auth with key" in docker.go
2017/01/24 04:13:48.576442 [INFO] client: Restarting task "hello-task" for alloc "f64724b3-b493-e180-fb94-d01ca7c5d2a3" in 18.534507544s
2017/01/24 04:14:00.978182 [INFO] Failed to find docker auth with key https://xxxxxxxx.dkr.ecr.us-west-2.amazonaws.com
Oooh, it is a bug: if repository named as xxxxxx.dkr.ecr.us-west-2.amazonaws.com/blah-project/blah-image (compare to simply xxxxx.dkr.ecr.us-west-2.amazonaws.com/blah-image) - then everything works as expected.
Splitting strings fails somewhere in that docker.go :)
Actually, let me take the last statement back. It is some sort of problem between docker, ecr and nomad.
Nomad somehow can't authenticate with ecr when permissions stores in .docker/config.json as following:
{
"auths": {
"xxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com": {
"auth": "xxx
But docker pull
works fine with this formatting.
However, nomad works fine too if the address includes https, e.g
{
"auths": {
"https://xxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com": {
"auth": "xxx
How could such a simple thing as http base auth got out of control and become such a mess?!
Btw, aws folks got frustrated with this and wrote helper: https://github.com/awslabs/amazon-ecr-credential-helper
@rokka-n Can you try on Nomad 0.5.3 and report back? We have updated the way we parse auth blocks in the file to be inline with how docker itself does it.
@dadgar Confirming, 0.5.3 eliminated it. Thank you!
Sweet!
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.
Nomad version
0.5.2
Operating system and Environment details
linux ubuntu
Issue
auth fails for docker images stored in ecr
Nomad Server logs (if appropriate)
Nomad Client logs (if appropriate)
/var/log/upstart/docker.log
Job file (if appropriate)