search

aws / amazon-ecs-agent

Amazon Elastic Container Service Agent
http://aws.amazon.com/ecs/
Apache License 2.0
2.08k stars 616 forks source link

Unable to pull private images from dockerhub #273

Closed wwallace closed 8 years ago

wwallace commented 8 years ago

Unable to pull private dockerhub images. Getting this error: Status reason CannotPullContainerError: Error: image myorg/platform:latest not found...

I can pull just fine locally and i can pull from within the ec2 instance if i run "docker login" and "docker pull myorg/platform:latest".

I have verified that the correct credentials are both in the host ec2 instance and in the ecs-agent. Here is the ecs.confg: cat /etc/ecs/ecs.config ECS_CLUSTER=awseb-clustername-test ECS_ENGINE_AUTH_TYPE=dockercfg ECS_ENGINE_AUTH_DATA={"https://index.docker.io/v1/":{"auth":"THEAUTHHASH","email":"william@removedmyemail.com"}}

This is on the agent: docker inspect ecs-agent "ECS_ENGINE_AUTH_DATA={\"https://index.docker.io/v1/\":{\"auth\":\"THEAUTHHASH\",\"email\":\"william@removedmyemail.net\"}}"

EDIT: Just to note, this is a multicontainer environment that was provisioned by elastic beanstalk

jsh2134 commented 8 years ago

did you restart the ecs agent after adding the ecs.config?

mbaumbach commented 8 years ago

I've been hitting the same problem (1.7.0 agent and Docker 1.9.1). The suggestion was to move to v2 API, but I couldn't get that going either due to a different error. This is the forum post I created on the topic: https://forums.aws.amazon.com/thread.jspa?threadID=221753

It doesn't start off this way either, it works for a little while and then starts throwing this error. Eventually Docker itself was completely unresponsive, but I think for us we ran out of disk space on the container instance and things in general were broken. We run a lot of fast running tasks and I noticed the number of containers in /var/lib/docker/containers was huge. We re-use the same image for every task and just modify the command in the run task call.

I don't know if they are related, but thought I'd throw in my two cents.

ivan-lorenz commented 8 years ago

Same error here with identical configuration (1.7.0 agent and Docker 1.9.1). Can pull with docker pull but not with ecs agent. Double check docker auth and is ok. Using dockercfg. I've just copy and paste docker config.json into ECS_ENGINE_AUTH_DATA, so I'm sure this part is ok.

Deshke commented 8 years ago

i found the problem... the ecs agent uses the remote api 1.17 to pull docker images, what ends up in a GET repository v1 request on the registry.

If you pull the image with docker pull it uses the docker remote api 1.21

time="2015-12-22T11:53:25.359272514Z" level=info msg="POST /v1.17/containers/create?name=ecs-image-21-image-8888f5f4e6c5aae13a00" 
time="2015-12-22T11:53:25.359949023Z" level=error msg="Handler for POST /v1.17/containers/create returned error: No such image: private-registry.tld:5043/user/image:e0994da-1512221135" 
time="2015-12-22T11:53:25.359982174Z" level=error msg="HTTP Error" err="No such image: private-registry.tld:5043/user/image:e0994da-1512221135" statusCode=404 
time="2015-12-22T11:53:32.141393198Z" level=info msg="POST /v1.21/images/create?fromImage=private-registry.tld%3A5043%2Fuser%2Fimage%3Ae0994da-1512221135"

problem is the docker client 1.9.1 can't speak docker-registry v1 only v2

docker-registry (v2):

nginx_1    | 54.229.57.28 - - [22/Dec/2015:12:22:21 +0000] "GET /v2/ HTTP/1.1" 401 194 "-" "docker/1.9.1 go/go1.4.2 kernel/4.1.13-19.30.amzn1.x86_64 os/linux arch/amd64" "-"
nginx_1    | 54.229.57.28 - - [22/Dec/2015:12:22:21 +0000] "GET /v2/ HTTP/1.1" 401 194 "-" "docker/1.9.1 go/go1.4.2 kernel/4.1.13-19.30.amzn1.x86_64 os/linux arch/amd64" "-"
nginx_1    | 54.229.57.28 - - [22/Dec/2015:12:22:22 +0000] "GET /v1/repositories/user/docker-image/images HTTP/1.1" 404 168 "-" "docker/1.9.1 go/go1.4.2 kernel/4.1.13-19.30.amzn1.x86_64 os/linux arch/amd64" "-"
nginx_1    | 2015/12/22 12:22:22 [error] 5#5: *2098 open() "/etc/nginx/html/v1/repositories/user/docker-image/images" failed (2: No such file or directory), client: 54.229.57.28, server: docker-reg.user-services.com, request: "GET /v1/repositories/user/docker-image/images HTTP/1.1", host: "docker-reg.user-services.com:5043"

docker pull from console:

nginx_1    | 54.229.57.28 - - [22/Dec/2015:12:23:52 +0000] "GET /v2/ HTTP/1.1" 401 194 "-" "docker/1.9.1 go/go1.4.2 kernel/4.1.13-19.30.amzn1.x86_64 os/linux arch/amd64" "-"
registry_1 | time="2015-12-22T12:23:54Z" level=info msg="response completed" go.version=go1.5.2 http.request.host="private-docker-registry.tld:5043" http.request.id=a5439534-5c73-4000-b34e-bdfd402c4d1c http.request.method=GET http.request.remoteaddr=54.229.57.28 http.request.uri="/v2/user/docker-image/manifests/e0994da-1512221135" http.request.useragent="docker/1.9.1 go/go1.4.2 kernel/4.1.13-19.30.amzn1.x86_64 os/linux arch/amd64" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.109745962s http.response.status=200 http.response.written=9715 instance.id=0766c8ee-91ac-42c5-941c-44a55c50de29 version=v2.2.1 
registry_1 | 172.17.0.3 - - [22/Dec/2015:12:23:52 +0000] "GET /v2/user/docker-image/manifests/e0994da-1512221135 HTTP/1.0" 200 9715 "" "docker/1.9.1 go/go1.4.2 kernel/4.1.13-19.30.amzn1.x86_64 os/linux arch/amd64"
nginx_1    | 54.229.57.28 - user-user [22/Dec/2015:12:23:54 +0000] "GET /v2/user/docker-image/manifests/e0994da-1512221135 HTTP/1.1" 200 9715 "-" "docker/1.9.1 go/go1.4.2 kernel/4.1.13-19.30.amzn1.x86_64 os/linux arch/amd64" "-"
registry_1 | time="2015-12-22T12:23:55Z" level=info msg="response completed" go.version=go1.5.2 http.request.host="private-docker-registry.tld:5043" http.request.id=df506620-bae8-478a-8b81-5a29fd445b4d http.request.method=HEAD http.request.remoteaddr=54.229.57.28 http.request.uri="/v2/user/docker-image/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" http.request.useragent="docker/1.9.1 go/go1.4.2 kernel/4.1.13-19.30.amzn1.x86_64 os/linux arch/amd64" http.response.contenttype="application/octet-stream" http.response.duration=1.403983748s http.response.status=307 http.response.written=0 instance.id=0766c8ee-91ac-42c5-941c-44a55c50de29 version=v2.2.1 
registry_1 | 172.17.0.3 - - [22/Dec/2015:12:23:54 +0000] "HEAD /v2/user/docker-image/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.0" 307 0 "" "docker/1.9.1 go/go1.4.2 kernel/4.1.13-19.30.amzn1.x86_64 os/linux arch/amd64"
nginx_1    | 54.229.57.28 - user-user [22/Dec/2015:12:23:55 +0000] "HEAD /v2/user/docker-image/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.1" 307 0 "-" "docker/1.9.1 go/go1.4.2 kernel/4.1.13-19.30.amzn1.x86_64 os/linux arch/amd64" "-"
registry_1 | time="2015-12-22T12:23:55Z" level=info msg="response completed" go.version=go1.5.2 http.request.host="private-docker-registry.tld:5043" http.request.id=5f04f066-4d74-49b9-8b54-936617ccbdf6 http.request.method=HEAD http.request.remoteaddr=54.229.57.28 http.request.uri="/v2/user/docker-image/blobs/sha256:a12abf165a6b02797523504dc0d57f0134485657d731ff4411e5da24fb1c9f8b" http.request.useragent="docker/1.9.1 go/go1.4.2 kernel/4.1.13-19.30.amzn1.x86_64 os/linux arch/amd64" http.response.contenttype="application/octet-stream" http.response.duration=1.422177359s http.response.status=307 http.response.written=0 instance.id=0766c8ee-91ac-42c5-941c-44a55c50de29 version=v2.2.1 
registry_1 | 172.17.0.3 - - [22/Dec/2015:12:23:54 +0000] "HEAD /v2/user/docker-image/blobs/sha256:a12abf165a6b02797523504dc0d57f0134485657d731ff4411e5da24fb1c9f8b HTTP/1.0" 307 0 "" "docker/1.9.1 go/go1.4.2 kernel/4.1.13-19.30.amzn1.x86_64 os/linux arch/amd64"
nginx_1    | 54.229.57.28 - user-user [22/Dec/2015:12:23:55 +0000] "HEAD /v2/user/docker-image/blobs/sha256:a12abf165a6b02797523504dc0d57f0134485657d731ff4411e5da24fb1c9f8b HTTP/1.1" 307 0 "-" "docker/1.9.1 go/go1.4.2 kernel/4.1.13-19.30.amzn1.x86_64 os/linux arch/amd64" "-"

docker-log (/var/log/docker)

time="2015-12-22T12:22:21.593045075Z" level=info msg="POST /v1.17/images/create?fromImage=private-docker-registry.tld%3A5043%2Fdocker_user%2Fdocker_image%3Ae0994da-1512221135" 
time="2015-12-22T12:22:22.060766943Z" level=info msg="POST /v1.17/containers/create?name=ecs-recording-preview-21-recording-preview-fedd9388819cd98edb01" 
time="2015-12-22T12:22:22.061311634Z" level=error msg="Handler for POST /v1.17/containers/create returned error: No such image: private-docker-registry.tld:5043/docker_user/docker_image:e0994da-1512221135" 
time="2015-12-22T12:22:22.061336035Z" level=error msg="HTTP Error" err="No such image: private-docker-registry.tld:5043/docker_user/docker_image:e0994da-1512221135" statusCode=404 
time="2015-12-22T12:22:31.112759386Z" level=info msg="POST /v1.17/images/create?fromImage=private-docker-registry.tld%3A5043%2Fdocker_user%2Fdocker_image%3Ae0994da-1512221135" 
time="2015-12-22T12:22:31.563382311Z" level=info msg="POST /v1.17/containers/create?name=ecs-recording-preview-21-recording-preview-f6fda9eaa2d3d9e1f401" 
time="2015-12-22T12:22:31.564112701Z" level=error msg="Handler for POST /v1.17/containers/create returned error: No such image: private-docker-registry.tld:5043/docker_user/docker_image:e0994da-1512221135" 
time="2015-12-22T12:22:31.564142803Z" level=error msg="HTTP Error" err="No such image: private-docker-registry.tld:5043/docker_user/docker_image:e0994da-1512221135" statusCode=404 
time="2015-12-22T12:22:41.694677147Z" level=info msg="POST /v1.17/images/create?fromImage=private-docker-registry.tld%3A5043%2Fdocker_user%2Fdocker_image%3Ae0994da-1512221135" 
time="2015-12-22T12:22:42.143309082Z" level=info msg="POST /v1.17/containers/create?name=ecs-recording-preview-21-recording-preview-febab6d3cb9cece26000" 
time="2015-12-22T12:22:42.143943518Z" level=error msg="Handler for POST /v1.17/containers/create returned error: No such image: private-docker-registry.tld:5043/docker_user/docker_image:e0994da-1512221135" 
time="2015-12-22T12:22:42.143974444Z" level=error msg="HTTP Error" err="No such image: private-docker-registry.tld:5043/docker_user/docker_image:e0994da-1512221135" statusCode=404 
time="2015-12-22T12:22:52.316502213Z" level=info msg="POST /v1.17/images/create?fromImage=private-docker-registry.tld%3A5043%2Fdocker_user%2Fdocker_image%3Ae0994da-1512221135" 
time="2015-12-22T12:22:52.771217950Z" level=info msg="POST /v1.17/containers/create?name=ecs-recording-preview-21-recording-preview-d2d6aeaeaf9afffff901" 
time="2015-12-22T12:22:52.771711429Z" level=error msg="Handler for POST /v1.17/containers/create returned error: No such image: private-docker-registry.tld:5043/docker_user/docker_image:e0994da-1512221135" 
time="2015-12-22T12:22:52.771735038Z" level=error msg="HTTP Error" err="No such image: private-docker-registry.tld:5043/docker_user/docker_image:e0994da-1512221135" statusCode=404 
time="2015-12-22T12:23:01.790782973Z" level=info msg="POST /v1.17/images/create?fromImage=private-docker-registry.tld%3A5043%2Fdocker_user%2Fdocker_image%3Ae0994da-1512221135" 
time="2015-12-22T12:23:02.241550743Z" level=info msg="POST /v1.17/containers/create?name=ecs-recording-preview-21-recording-preview-aacfd1e4b2bea4b3fd01" 
time="2015-12-22T12:23:02.242069999Z" level=error msg="Handler for POST /v1.17/containers/create returned error: No such image: private-docker-registry.tld:5043/docker_user/docker_image:e0994da-1512221135" 
time="2015-12-22T12:23:02.242095067Z" level=error msg="HTTP Error" err="No such image: private-docker-registry.tld:5043/docker_user/docker_image:e0994da-1512221135" statusCode=404 
time="2015-12-22T12:23:11.366802508Z" level=info msg="POST /v1.17/images/create?fromImage=private-docker-registry.tld%3A5043%2Fdocker_user%2Fdocker_image%3Ae0994da-1512221135" 
time="2015-12-22T12:23:11.813533092Z" level=info msg="POST /v1.17/containers/create?name=ecs-recording-preview-21-recording-preview-84fccf92c9cedfef0500" 
time="2015-12-22T12:23:11.814021266Z" level=error msg="Handler for POST /v1.17/containers/create returned error: No such image: private-docker-registry.tld:5043/docker_user/docker_image:e0994da-1512221135" 
time="2015-12-22T12:23:11.814045483Z" level=error msg="HTTP Error" err="No such image: private-docker-registry.tld:5043/docker_user/docker_image:e0994da-1512221135" statusCode=404 
time="2015-12-22T12:23:21.916563369Z" level=info msg="POST /v1.17/images/create?fromImage=private-docker-registry.tld%3A5043%2Fdocker_user%2Fdocker_image%3Ae0994da-1512221135" 
time="2015-12-22T12:23:22.357886383Z" level=info msg="POST /v1.17/containers/create?name=ecs-recording-preview-21-recording-preview-d8cde08fc589fdf9e701" 
time="2015-12-22T12:23:22.358419697Z" level=error msg="Handler for POST /v1.17/containers/create returned error: No such image: private-docker-registry.tld:5043/docker_user/docker_image:e0994da-1512221135" 
time="2015-12-22T12:23:22.358464378Z" level=error msg="HTTP Error" err="No such image: private-docker-registry.tld:5043/docker_user/docker_image:e0994da-1512221135" statusCode=404 
time="2015-12-22T12:23:32.504994492Z" level=info msg="POST /v1.17/images/create?fromImage=private-docker-registry.tld%3A5043%2Fdocker_user%2Fdocker_image%3Ae0994da-1512221135" 
time="2015-12-22T12:23:32.955361509Z" level=info msg="POST /v1.17/containers/create?name=ecs-recording-preview-21-recording-preview-b0a7e6eff7f5e3af8901" 
time="2015-12-22T12:23:32.956062897Z" level=error msg="Handler for POST /v1.17/containers/create returned error: No such image: private-docker-registry.tld:5043/docker_user/docker_image:e0994da-1512221135" 
time="2015-12-22T12:23:32.956088612Z" level=error msg="HTTP Error" err="No such image: private-docker-registry.tld:5043/docker_user/docker_image:e0994da-1512221135" statusCode=404 
time="2015-12-22T12:23:42.018270493Z" level=info msg="POST /v1.17/images/create?fromImage=private-docker-registry.tld%3A5043%2Fdocker_user%2Fdocker_image%3Ae0994da-1512221135" 
time="2015-12-22T12:23:42.476480942Z" level=info msg="POST /v1.17/containers/create?name=ecs-recording-preview-21-recording-preview-fae2868785bf9ce3fe01" 
time="2015-12-22T12:23:42.477133862Z" level=error msg="Handler for POST /v1.17/containers/create returned error: No such image: private-docker-registry.tld:5043/docker_user/docker_image:e0994da-1512221135" 
time="2015-12-22T12:23:42.477158454Z" level=error msg="HTTP Error" err="No such image: private-docker-registry.tld:5043/docker_user/docker_image:e0994da-1512221135" statusCode=404 
time="2015-12-22T12:23:51.786509260Z" level=info msg="POST /v1.21/images/create?fromImage=private-docker-registry.tld%3A5043%2Fdocker_user%2Fdocker_image%3Ae0994da-1512221135" 
time="2015-12-22T12:23:52.536753568Z" level=info msg="POST /v1.17/images/create?fromImage=private-docker-registry.tld%3A5043%2Fdocker_user%2Fdocker_image%3Ae0994da-1512221135" 
time="2015-12-22T12:24:39.029300134Z" level=info msg="POST /v1.17/containers/create?name=ecs-recording-preview-21-recording-preview-f0a4a790ff95f6b9df01" 
time="2015-12-22T12:24:39.199313805Z" level=info msg="GET /v1.17/containers/980c49c604a26517afefea2d6e9539031a0cf53ab957831d8bc4cbdff2b5513f/json" 
time="2015-12-22T12:24:39.259003402Z" level=info msg="GET /v1.17/containers/980c49c604a26517afefea2d6e9539031a0cf53ab957831d8bc4cbdff2b5513f/json" 
time="2015-12-22T12:24:39.260483204Z" level=info msg="POST /v1.17/containers/980c49c604a26517afefea2d6e9539031a0cf53ab957831d8bc4cbdff2b5513f/start" 
time="2015-12-22T12:24:39.300874170Z" level=info msg="GET /v1.17/containers/980c49c604a26517afefea2d6e9539031a0cf53ab957831d8bc4cbdff2b5513f/json" 
time="2015-12-22T12:24:39.394428408Z" level=info msg="GET /v1.17/containers/980c49c604a26517afefea2d6e9539031a0cf53ab957831d8bc4cbdff2b5513f/json"

update: added logs from the registry

Deshke commented 8 years ago

@samuelkarp can you please take a look at this?

rikonor commented 8 years ago

Until this is resolved by the maintainers, here's the way I got ECS to pull images.

Instead of using the ECS optimized image, use the general-purpose Amazon Linux AMI (ami-60b6c60a).

Set the following user-data:

#!/bin/bash
yum install -y aws-cli
mkdir -p /etc/ecs/
aws s3 cp s3://vocativ-darknet-settings/ecs.config /etc/ecs/ecs.config

# install docker
yum install -y docker
service docker start

# install ecs-init
yum install -y ecs-init
start ecs

Assuming you put your authentication data in ecs.config, it should pull private repos as expected.

wwallace commented 8 years ago

everything started working "magically", but I am going to leave this ticket open, as I don't like magic fixes and it seems there are many other people having the same issue.

samuelkarp commented 8 years ago

@wwallace @Deshke I've been unable to reproduce this myself when pulling from Docker Hub, a private registry with basic auth, or from ECR. I did see the problem you reported when I had malformed ECS_ENGINE_AUTH_DATA, invalid ECS_ENGINE_AUTH_DATA, or when I was trying to pull a tag that did not exist.

Can you try running Docker with debug logs and looking at the output from when it tries to pull? I see different output in the daemon debug logs when the image is not there versus when the auth data is wrong.

Bad image/tag:

time="2015-12-31T00:23:45.713915909Z" level=info msg="POST /v1.17/images/create?fromImage=MYUSERNAME%2Fprivate%3ATAG"
time="2015-12-31T00:23:45.713994690Z" level=debug msg="pulling image from host \"docker.io\" with remote name \"MYUSERNAME/private\""
time="2015-12-31T00:23:45.714059170Z" level=debug msg="pinging registry endpoint https://index.docker.io/v1/"
time="2015-12-31T00:23:45.714075179Z" level=debug msg="attempting v1 ping for registry endpoint https://index.docker.io/v1/"
time="2015-12-31T00:23:45.714092232Z" level=debug msg="pulling v2 repository with local name \"MYUSERNAME/private\""
time="2015-12-31T00:23:45.714107935Z" level=debug msg="pinging registry endpoint https://registry-1.docker.io/v2/"
time="2015-12-31T00:23:45.714122414Z" level=debug msg="attempting v2 ping for registry endpoint https://registry-1.docker.io/v2/"
time="2015-12-31T00:23:45.714148482Z" level=debug msg="hostDir: /etc/docker/certs.d/registry-1.docker.io"
time="2015-12-31T00:23:46.083744029Z" level=debug msg="Getting authorization for MYUSERNAME/private [pull]"
time="2015-12-31T00:23:46.083780804Z" level=debug msg="Pulling tag from V2 registry: \"TAG\""
time="2015-12-31T00:23:46.083837831Z" level=debug msg="[registry] Calling \"GET\" https://registry-1.docker.io/v2/MYUSERNAME/private/manifests/TAG"
time="2015-12-31T00:23:46.083858994Z" level=debug msg="Getting bearer token with map[realm:https://auth.docker.io/token service:registry.docker.io] for MYUSERNAME"
time="2015-12-31T00:23:46.083920838Z" level=debug msg="hostDir: /etc/docker/certs.d/auth.docker.io"
time="2015-12-31T00:23:46.469246627Z" level=debug msg="hostDir: /etc/docker/certs.d/registry-1.docker.io"
time="2015-12-31T00:23:46.854081284Z" level=debug msg="image does not exist on v2 registry, falling back to v1"
time="2015-12-31T00:23:46.854150209Z" level=debug msg="pulling v1 repository with local name \"MYUSERNAME/private\""
time="2015-12-31T00:23:46.854417296Z" level=debug msg="[registry] Calling GET https://index.docker.io/v1/repositories/MYUSERNAME/private/images"
time="2015-12-31T00:23:46.854487723Z" level=debug msg="hostDir: /etc/docker/certs.d/index.docker.io"
time="2015-12-31T00:23:47.945755666Z" level=debug msg="Retrieving the tag list"
time="2015-12-31T00:23:47.945812187Z" level=debug msg="hostDir: /etc/docker/certs.d/registry-1.docker.io"
time="2015-12-31T00:23:48.452596847Z" level=debug msg="Got status code 404 from https://registry-1.docker.io/v1/repositories/MYUSERNAME/private/tags"
time="2015-12-31T00:23:48.452661608Z" level=error msg="unable to get remote tags: Repository not found"
time="2015-12-31T00:23:48.454942398Z" level=debug msg="Calling POST /containers/create"
time="2015-12-31T00:23:48.454983240Z" level=info msg="POST /v1.17/containers/create?name=ecs-private-sleep-1-sleeper-b894ffbadbdefc90c901"
time="2015-12-31T00:23:48.455165030Z" level=error msg="Handler for POST /containers/create returned error: No such image: MYUSERNAME/private:TAG (tag: TAG)"
time="2015-12-31T00:23:48.455200769Z" level=error msg="HTTP Error" err="No such image: MYUSERNAME/private:TAG (tag: TAG)" statusCode=404

Bad auth data:

time="2015-12-31T00:13:59.874974470Z" level=error msg="Error from V2 registry: token auth attempt for registry https://registry-1.docker.io/v2/: https://auth.docker.io/token?account=MYUSERNAME&scope=repository%3AMYUSERNAME%2Fprivate%3Apull&service=registry.docker.io request failed with status: 401 Unauthorized"

Note that in both of the above cases, Docker first tries a v2 endpoint and then falls back to v1 as it assumes the problem is that the registry doesn't support v2. The error handling here is not particularly nuanced.

Success:

time="2015-12-31T00:25:50.983796629Z" level=info msg="POST /v1.17/images/create?fromImage=MYUSERNAME%2Fprivate%3ATAG"
time="2015-12-31T00:25:50.983876669Z" level=debug msg="pulling image from host \"docker.io\" with remote name \"MYUSERNAME/private\""
time="2015-12-31T00:25:50.983943873Z" level=debug msg="pinging registry endpoint https://index.docker.io/v1/"
time="2015-12-31T00:25:50.983959304Z" level=debug msg="attempting v1 ping for registry endpoint https://index.docker.io/v1/"
time="2015-12-31T00:25:50.983976126Z" level=debug msg="pulling v2 repository with local name \"MYUSERNAME/private\""
time="2015-12-31T00:25:50.983991624Z" level=debug msg="pinging registry endpoint https://registry-1.docker.io/v2/"
time="2015-12-31T00:25:50.984002878Z" level=debug msg="attempting v2 ping for registry endpoint https://registry-1.docker.io/v2/"
time="2015-12-31T00:25:50.984022342Z" level=debug msg="hostDir: /etc/docker/certs.d/registry-1.docker.io"
time="2015-12-31T00:25:51.375152722Z" level=debug msg="Getting authorization for MYUSERNAME/private [pull]"
time="2015-12-31T00:25:51.375184081Z" level=debug msg="Pulling tag from V2 registry: \"TAG\""
time="2015-12-31T00:25:51.375256257Z" level=debug msg="[registry] Calling \"GET\" https://registry-1.docker.io/v2/MYUSERNAME/private/manifests/TAG"
time="2015-12-31T00:25:51.375281281Z" level=debug msg="Getting bearer token with map[realm:https://auth.docker.io/token service:registry.docker.io] for MYUSERNAME"

All of these logs come from our 2015.09.c AMI with Docker 1.7.1 and ECS Agent 1.7.0.

Deshke commented 8 years ago

crazy, i will debug this further in the new year

drobakowski commented 8 years ago

@samuelkarp I'm running into the same error as @Deshke. I can confirm that my configuration is working with the ECS_ENGINE_AUTH_TYPE=docker:

ECS_ENGINE_AUTH_TYPE=docker
ECS_ENGINE_AUTH_DATA={"https://index.docker.io/v1/": {"username": "...", "password": "...", "email": "..."}}

but when I'm using ECS_ENGINE_AUTH_TYPE=dockercfg I'm getting the following errors:

time="2016-01-08T18:25:33.894727937Z" level=info msg="POST /v1.17/images/create?fromImage=<ACCOUNT>%2<PRIVATE_REPOS>%3A<VERSION>"
time="2016-01-08T18:25:35.021356784Z" level=error msg="Error from V2 registry: Authentication is required."
time="2016-01-08T18:25:35.427962055Z" level=info msg="POST /v1.18/containers/create?name=ecs-..."
time="2016-01-08T18:25:35.428384401Z" level=error msg="Handler for POST /containers/create returned error: No such image: <ACCOUNT>/<PRIVATE_REPOS>:<VERSION> (tag: <VERSION>)"
time="2016-01-08T18:25:35.428467802Z" level=error msg="HTTP Error" err="No such image: <ACCOUNT>/<PRIVATE_REPOS>:<VERSION> (tag: <VERSION>)" statusCode=404

I've copied the docker config.json directly to ecs.config so there shouldn't be any malformed data.

not working ecs.config:

ECS_CLUSTER=...
ECS_ENGINE_AUTH_TYPE=dockercfg
ECS_ENGINE_AUTH_DATA={"auths": {"https://index.docker.io/v1/": {"auth": "...", "email": "..."}}}

I'm also using 2015.09.c AMI with Docker 1.7.1 and ECS Agent 1.7.0 running in the eu-central-1 region and also verified that the ecs-agent is using the updated configuration.

samuelkarp commented 8 years ago

@drobakowski When using dockercfg, config.json cannot be directly copied to ECS_ENGINE_AUTH_DATA. Specifically, the auths outer field is not used; instead you should just have the map from registry (https://index.docker.io/v1/) to auth data. You'll want it to look as follows:

ECS_ENGINE_AUTH_TYPE=dockercfg
ECS_ENGINE_AUTH_DATA={"https://index.docker.io/v1/":{"auth":"zq212MzEXAMPLE7o6T25Dk0i","email":"email@example.com"}

See the documentation for more details.

drobakowski commented 8 years ago

@samuelkarp you are absolutely right! After tweaking my misconfiguration this way, the image can be successfully pulled from docker. Sorry for the unnecessary noise and thanks a lot for the quick help!

UriKatz commented 8 years ago

I am having the same issue. I triple checked all configurations and they are correct. The only plausible explanation (I heared so far) is API versioing.

samuelkarp commented 8 years ago

@UriKatz, so far we've been unsuccessful in reproducing the problem. Since you are experiencing it, can you look in the Docker daemon logs (as I described) and see what they say? Additionally, can you post sanitized versions of your ECS_ENGINE_AUTH_TYPE and ECS_ENGINE_AUTH_DATA like @drobakowski did (please make sure to remove your actual credential data)?

UriKatz commented 8 years ago

@samuelkarp here is the information you requested:

More info: (1) I am using Elastic Beanstalk; (2) I tested setting my images to public, and everything works.

ECS_CLUSTER=awseb-axis-staging-nymhgu9enm ECS_ENGINE_AUTH_TYPE=dockercfg ECS_ENGINE_AUTH_DATA={"auths":{"https://index.docker.io/v1/":{"auth":"xxx","email":"xxx@xxx.com"}}}

time="2016-01-21T20:23:48.049198837Z" level=info msg="DELETE /v1.19/images/6bd8695f794a" time="2016-01-21T20:23:52.578116522Z" level=info msg="DELETE /v1.19/images/8c2b81d41510" time="2016-01-21T20:23:52.578219402Z" level=error msg="Handler for DELETE /images/{name:.} returned error: No such image: 8c2b81d41510" time="2016-01-21T20:23:52.578240856Z" level=error msg="HTTP Error" err="No such image: 8c2b81d41510" statusCode=404 time="2016-01-21T20:23:52.578465569Z" level=info msg="DELETE /v1.19/images/0d31c477e494" time="2016-01-21T20:23:52.578538397Z" level=error msg="Handler for DELETE /images/{name:.} returned error: No such image: 0d31c477e494" time="2016-01-21T20:23:52.578557676Z" level=error msg="HTTP Error" err="No such image: 0d31c477e494" statusCode=404 time="2016-01-21T20:23:52.578740433Z" level=info msg="DELETE /v1.19/images/9d0e8e26845f" time="2016-01-21T20:23:52.578807745Z" level=error msg="Handler for DELETE /images/{name:.} returned error: No such image: 9d0e8e26845f" time="2016-01-21T20:23:52.578825598Z" level=error msg="HTTP Error" err="No such image: 9d0e8e26845f" statusCode=404 time="2016-01-21T20:23:52.579018670Z" level=info msg="DELETE /v1.19/images/ad32405e6b07" time="2016-01-21T20:23:52.579100535Z" level=error msg="Handler for DELETE /images/{name:.} returned error: No such image: ad32405e6b07" time="2016-01-21T20:23:52.579118683Z" level=error msg="HTTP Error" err="No such image: ad32405e6b07" statusCode=404 time="2016-01-21T20:23:52.579302240Z" level=info msg="DELETE /v1.19/images/3a13fe6d067c" time="2016-01-21T20:23:52.579386133Z" level=error msg="Handler for DELETE /images/{name:.} returned error: No such image: 3a13fe6d067c" time="2016-01-21T20:23:52.579404179Z" level=error msg="HTTP Error" err="No such image: 3a13fe6d067c" statusCode=404 time="2016-01-21T20:23:52.579611953Z" level=info msg="DELETE /v1.19/images/18fd13acb241" time="2016-01-21T20:23:52.579683858Z" level=error msg="Handler for DELETE /images/{name:.} returned error: No such image: 18fd13acb241" time="2016-01-21T20:23:52.579701361Z" level=error msg="HTTP Error" err="No such image: 18fd13acb241" statusCode=404 time="2016-01-21T20:23:52.579892102Z" level=info msg="DELETE /v1.19/images/9fd49db1519a" time="2016-01-21T20:23:52.579975379Z" level=error msg="Handler for DELETE /images/{name:.} returned error: No such image: 9fd49db1519a" time="2016-01-21T20:23:52.579995426Z" level=error msg="HTTP Error" err="No such image: 9fd49db1519a" statusCode=404 time="2016-01-21T20:23:52.580191425Z" level=info msg="DELETE /v1.19/images/e5b8ef372059" time="2016-01-21T20:23:52.580285321Z" level=error msg="Handler for DELETE /images/{name:.} returned error: No such image: e5b8ef372059" time="2016-01-21T20:23:52.580303302Z" level=error msg="HTTP Error" err="No such image: e5b8ef372059" statusCode=404 time="2016-01-21T20:23:52.580523871Z" level=info msg="DELETE /v1.19/images/a6e95c2e4006" time="2016-01-21T20:23:52.580589554Z" level=error msg="Handler for DELETE /images/{name:.} returned error: No such image: a6e95c2e4006" time="2016-01-21T20:23:52.580607447Z" level=error msg="HTTP Error" err="No such image: a6e95c2e4006" statusCode=404 time="2016-01-21T20:23:52.580803241Z" level=info msg="DELETE /v1.19/images/d4b2ba78e3b4" time="2016-01-21T20:23:52.580889039Z" level=error msg="Handler for DELETE /images/{name:.} returned error: No such image: d4b2ba78e3b4" time="2016-01-21T20:23:52.580908199Z" level=error msg="HTTP Error" err="No such image: d4b2ba78e3b4" statusCode=404 time="2016-01-21T20:23:52.581092660Z" level=info msg="DELETE /v1.19/images/cb6fb082434e" time="2016-01-21T20:23:52.581169774Z" level=error msg="Handler for DELETE /images/{name:.} returned error: No such image: cb6fb082434e" time="2016-01-21T20:23:52.581188566Z" level=error msg="HTTP Error" err="No such image: cb6fb082434e" statusCode=404 time="2016-01-21T20:23:52.581374174Z" level=info msg="DELETE /v1.19/images/ac6ec1b03a06" time="2016-01-21T20:23:52.582719917Z" level=error msg="Handler for DELETE /images/{name:.} returned error: Conflict, cannot delete ac6ec1b03a06 because the running container 386523c252ff is using it, stop it and use -f to force" time="2016-01-21T20:23:52.582745973Z" level=error msg="HTTP Error" err="Conflict, cannot delete ac6ec1b03a06 because the running container 386523c252ff is using it, stop it and use -f to force" statusCode=409 time="2016-01-21T20:23:52.582988532Z" level=info msg="DELETE /v1.19/images/541a48f5f917" time="2016-01-21T20:23:52.584165633Z" level=error msg="Handler for DELETE /images/{name:.} returned error: Conflict, cannot delete 541a48f5f917 because the running container 386523c252ff is using it, stop it and use -f to force" time="2016-01-21T20:23:52.584192103Z" level=error msg="HTTP Error" err="Conflict, cannot delete 541a48f5f917 because the running container 386523c252ff is using it, stop it and use -f to force" statusCode=409 time="2016-01-21T20:23:52.584433398Z" level=info msg="DELETE /v1.19/images/e0d07bbce5ce" time="2016-01-21T20:23:52.585766940Z" level=error msg="Handler for DELETE /images/{name:.} returned error: Conflict, cannot delete e0d07bbce5ce because the running container 386523c252ff is using it, stop it and use -f to force" time="2016-01-21T20:23:52.585791374Z" level=error msg="HTTP Error" err="Conflict, cannot delete e0d07bbce5ce because the running container 386523c252ff is using it, stop it and use -f to force" statusCode=409 time="2016-01-21T20:23:52.586015963Z" level=info msg="DELETE /v1.19/images/d14931524bce" time="2016-01-21T20:23:52.587428414Z" level=error msg="Handler for DELETE /images/{name:.} returned error: Conflict, cannot delete d14931524bce because the running container 386523c252ff is using it, stop it and use -f to force" time="2016-01-21T20:23:52.587453352Z" level=error msg="HTTP Error" err="Conflict, cannot delete d14931524bce because the running container 386523c252ff is using it, stop it and use -f to force" statusCode=409 time="2016-01-21T20:23:52.587695988Z" level=info msg="DELETE /v1.19/images/dc91cd0a7a3c" time="2016-01-21T20:23:52.589204491Z" level=error msg="Handler for DELETE /images/{name:.} returned error: Conflict, cannot delete dc91cd0a7a3c because the running container 386523c252ff is using it, stop it and use -f to force" time="2016-01-21T20:23:52.589229281Z" level=error msg="HTTP Error" err="Conflict, cannot delete dc91cd0a7a3c because the running container 386523c252ff is using it, stop it and use -f to force" statusCode=409 time="2016-01-21T20:23:52.589454087Z" level=info msg="DELETE /v1.19/images/a0977567daf1" time="2016-01-21T20:23:52.591135875Z" level=error msg="Handler for DELETE /images/{name:._} returned error: Conflict, cannot delete a0977567daf1 because the running container 386523c252ff is using it, stop it and use -f to force" time="2016-01-21T20:23:52.591171794Z" level=error msg="HTTP Error" err="Conflict, cannot delete a0977567daf1 because the running container 386523c252ff is using it, stop it and use -f to force" statusCode=409 time="2016-01-21T20:23:54.083481141Z" level=info msg="GET /v1.15/version" time="2016-01-21T20:23:54.083777697Z" level=info msg="GET /v1.15/_ping" time="2016-01-21T20:23:54.083995918Z" level=info msg="GET /v1.15/containers/json?all=1&filters=%7B%22status%22%3A%5B%5D%7D" time="2016-01-21T20:23:54.084722016Z" level=info msg="POST /v1.15/containers/386523c252fffdcb4b914c6184f1434aebf4f5dd317ac44297f40489de249756/stop?t=10" time="2016-01-21T20:23:54.274475000Z" level=info msg="GET /v1.15/version" time="2016-01-21T20:23:54.274777101Z" level=info msg="GET /v1.15/_ping" time="2016-01-21T20:23:54.274980278Z" level=info msg="GET /v1.15/images/json?all=1" time="2016-01-21T20:23:54.281177302Z" level=info msg="GET /v1.15/version" time="2016-01-21T20:23:54.281437568Z" level=info msg="GET /v1.15/_ping" time="2016-01-21T20:23:54.281658480Z" level=info msg="GET /v1.15/containers/json?all=1&filters=%7B%22status%22%3A%5B%5D%7D" time="2016-01-21T20:23:54.282482323Z" level=info msg="DELETE /v1.15/containers/386523c252fffdcb4b914c6184f1434aebf4f5dd317ac44297f40489de249756?force=1" time="2016-01-21T20:23:55.056948640Z" level=info msg="POST /v1.15/containers/create?name=ecs-agent" time="2016-01-21T20:23:55.309107122Z" level=info msg="POST /v1.15/containers/162a39c362ded11130053efab9b91e146a4feca0ba5655c286f3989cfea05545/start" time="2016-01-21T20:23:55.454506588Z" level=info msg="POST /v1.15/containers/162a39c362ded11130053efab9b91e146a4feca0ba5655c286f3989cfea05545/wait" time="2016-01-21T20:23:55.466501679Z" level=info msg="GET /v1.17/version" time="2016-01-21T20:23:55.467076478Z" level=info msg="GET /v1.17/_ping" time="2016-01-21T20:23:55.467258878Z" level=info msg="GET /v1.17/_ping" time="2016-01-21T20:23:55.467505718Z" level=info msg="GET /v1.18/version" time="2016-01-21T20:23:55.467972899Z" level=info msg="GET /v1.18/_ping" time="2016-01-21T20:23:55.468213072Z" level=info msg="GET /v1.19/version" time="2016-01-21T20:23:55.468464607Z" level=info msg="GET /v1.19/ping" time="2016-01-21T20:23:55.468635518Z" level=info msg="GET /v1.20/version" time="2016-01-21T20:23:55.541022287Z" level=info msg="GET /events" time="2016-01-21T20:23:55.541233588Z" level=info msg="GET /v1.17/containers/e23c571b004f73e52c5de346d110adaf4a49ce2c8ccbe6e38297452f3eaad5f8/json" time="2016-01-21T20:23:55.541395868Z" level=error msg="Handler for GET /containers/{name:.}/json returned error: no such id: e23c571b004f73e52c5de346d110adaf4a49ce2c8ccbe6e38297452f3eaad5f8" time="2016-01-21T20:23:55.541420571Z" level=error msg="HTTP Error" err="no such id: e23c571b004f73e52c5de346d110adaf4a49ce2c8ccbe6e38297452f3eaad5f8" statusCode=404 time="2016-01-21T20:23:55.541737528Z" level=info msg="GET /v1.17/containers/dd61a435354e04c303049105129226b5ac680e15e73675551f6d4aff4027be05/json" time="2016-01-21T20:23:55.541846562Z" level=error msg="Handler for GET /containers/{name:.*}/json returned error: no such id: dd61a435354e04c303049105129226b5ac680e15e73675551f6d4aff4027be05" time="2016-01-21T20:23:55.541868014Z" level=error msg="HTTP Error" err="no such id: dd61a435354e04c303049105129226b5ac680e15e73675551f6d4aff4027be05" statusCode=404 time="2016-01-21T20:23:55.544309622Z" level=info msg="GET /v1.17/version" time="2016-01-21T20:23:55.544930374Z" level=info msg="GET /v1.17/_ping" time="2016-01-21T20:23:55.545098856Z" level=info msg="GET /v1.17/_ping" time="2016-01-21T20:23:55.545358501Z" level=info msg="GET /events" time="2016-01-21T20:23:55.545509158Z" level=info msg="GET /v1.17/containers/json" time="2016-01-21T20:23:55.548942854Z" level=info msg="GET /v1.17/version" time="2016-01-21T20:23:57.296833246Z" level=info msg="POST /v1.17/images/create?fromImage=urikatz%2Fxprt-bot%3Alatest" time="2016-01-21T20:23:57.387087096Z" level=error msg="Error from V2 registry: Authentication is required." time="2016-01-21T20:23:57.438151181Z" level=info msg="POST /v1.17/containers/create?name=ecs-awseb-xprtBot-prod-yb6rmjiupv-5-xprt-bot-dee89bddded2dfeff701" time="2016-01-21T20:23:57.438375982Z" level=error msg="Handler for POST /containers/create returned error: No such image: urikatz/xprt-bot (tag: latest)" time="2016-01-21T20:23:57.438399790Z" level=error msg="HTTP Error" err="No such image: urikatz/xprt-bot (tag: latest)" statusCode=404 time="2016-01-21T20:23:59.995510818Z" level=info msg="GET /v1.19/containers/json" time="2016-01-21T20:24:03.785039404Z" level=info msg="POST /v1.17/images/create?fromImage=urikatz%2Fxprt-bot%3Alatest" time="2016-01-21T20:24:03.842154749Z" level=error msg="Error from V2 registry: Authentication is required." time="2016-01-21T20:24:03.891398101Z" level=info msg="POST /v1.17/containers/create?name=ecs-awseb-xprtBot-prod-yb6rmjiupv-5-xprt-bot-b68c9a9cb2be8bc29b01" time="2016-01-21T20:24:03.891673477Z" level=error msg="Handler for POST /containers/create returned error: No such image: urikatz/xprt-bot (tag: latest)" time="2016-01-21T20:24:03.891704279Z" level=error msg="HTTP Error" err="No such image: urikatz/xprt-bot (tag: latest)" statusCode=404 time="2016-01-21T20:24:07.049014459Z" level=info msg="GET /v1.19/containers/json" time="2016-01-21T20:26:25.003576387Z" level=info msg="GET /v1.19/containers/json"

samuelkarp commented 8 years ago

@UriKatz Your ECS_ENGINE_AUTH_DATA is not well-formed. See https://github.com/aws/amazon-ecs-agent/issues/273#issuecomment-170991588.

UriKatz commented 8 years ago

@samuelkarp this was automatically set by elastic beanstalk from a conifg file on S3. This config file was created by the command docker login. Digging deeper I found this: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker.html#docker-images-private which explains why I was having a problem. Thanks.

TLDR - The config file on S3 should look like this (docker changed the format versions after 1.6.2):

{ "server" : { "auth" : "auth_token", "email" : "email" } }

jflasher commented 8 years ago

I am running into the same issue (can pull on instance, but agent fails to pull) with ECS Agent 1.6.0 and Docker version 1.7.1.

ecs.config contains following

ENGINE_AUTH_TYPE=docker
ECS_ENGINE_AUTH_DATA={"https://index.docker.io/v1/":{"username":"user","password":"pw","email":"my+email@email.com"}}

I do see an error in the log complaining about auth failure, but it seems like my auth data is well formed?

time="2016-01-27T17:39:16.790043385Z" level=info msg="GET /v1.17/containers/a3b057c6f5cd3b5d511cfedc3bcb41d73fa0319fae5d84ba5f3a5c618a726e34/json" 
time="2016-01-27T17:39:16.790177937Z" level=info msg="GET /v1.17/containers/a3b057c6f5cd3b5d511cfedc3bcb41d73fa0319fae5d84ba5f3a5c618a726e34/json" 
time="2016-01-27T17:39:16.816178559Z" level=info msg="GET /v1.18/containers/a3b057c6f5cd3b5d511cfedc3bcb41d73fa0319fae5d84ba5f3a5c618a726e34/json" 
time="2016-01-27T17:39:16.818515469Z" level=info msg="POST /v1.17/images/create?fromImage=developmentseed%2Fwapo-backend%3A5fdbc6e66a2cc7c790707644c36c98b676a8691b" 
time="2016-01-27T17:39:17.487218992Z" level=info msg="GET /v1.19/containers/json" 
time="2016-01-27T17:39:17.801994014Z" level=error msg="Error from V2 registry: Authentication is required." 
time="2016-01-27T17:39:18.119083303Z" level=info msg="POST /v1.17/images/create?fromImage=developmentseed%2Fwapo-backend%3A5fdbc6e66a2cc7c790707644c36c98b676a8691b" 
time="2016-01-27T17:39:18.571859374Z" level=info msg="GET /v1.19/containers/json" 
time="2016-01-27T17:39:19.173509496Z" level=error msg="Error from V2 registry: Authentication is required."

Also, is it odd that it's a v2 auth failure but everything else is tied to v1?

Edit: Have updated to ECS Agent 1.7.1 and having same issue but no longer seeing any error messages.

Edit 2: So I don't think this was my initial issue, but you can see that I have a mistype env var set (at the top), changing this has seemed to fix the issue for me.

lanceblais commented 8 years ago

Hi,

My logs in /var/log/docker are showing issues with being able to pull private images, but no Authentication problems.

My /etc/ecs/ecs.config:

ECS_ENGINE_AUTH_TYPE=dockercfg
ECS_ENGINE_AUTH_DATA={"https://index.docker.io/v1/":{"auth":"HASH","email":"email@example.com"}}
ECS_CLUSTER=default

We did recently update to the latest ecs-agent, but we haven't tried pulling a fresh image (only launch new tasks with images already on the hosts), so this might have something to do with it.

Could someone help me debug this issue and make sure all the settings are correct?

Thanks

samuelkarp commented 8 years ago

@lanceblais Can you post the relevant log lines here?

lanceblais commented 8 years ago

My bad for not checking back in.

We relaunched 2 fresh instances with the pre-updated ecs-agent version and all was good.

We did something wrong with the upgrade process and that's probably what made us unable to pull from our private repos.

This can be closed.

samuelkarp commented 8 years ago

Thanks. I'm going to close this for now, as it looks like it was just configuration issues. Please feel free to add more information here or open a new issue if you continue to experience problems.

MaikuMori commented 8 years ago

I have similar issue where it can't pull images from Docker Hub, but in my case the container instances were running and pulling successfully before and then suddenly stopped doing that.

Sadly I don't have agent logs, but I'll try to get them.

agentfitz commented 8 years ago

I am here because I am also unable to pull private repo images from Docker Hub from Amazon Elastic Beanstalk. I have ssh'd into the EC2 instance, but not sure where to find the relevant logs.

aaithal commented 8 years ago

@agentfitz you can find instructions getting agent logs here. You can also look at docker logs in the /var/log/docker file.

agentfitz commented 8 years ago

Thanks. Not sure if this is helpful, but here I am logged into the EC2 instance: first I try to pull down agentfitz/private-test (which does exist), and it fails locally and then also fails to find on docker hub.

Then I attempt to pull down a public image lucee/lucee4-nginx, and it fails to find locally but succeeds in pulling in image from docker hub.

I have no ECS log as I'm not running this in the Elastic Container Service. In this case, I am running an Elastic Beanstalk EC2 instance setup with the "generic docker" configuration.

[ec2-user@ip-172-31-14-191 log]$ cat docker
time="2016-03-03T03:01:28.926123596Z" level=info msg="POST /v1.21/containers/create?name=agentfitz-private-test"
time="2016-03-03T03:01:28.927761909Z" level=error msg="Handler for POST /v1.21/containers/create returned error: No such image: agentfitz/private-test:latest"
time="2016-03-03T03:01:28.927811591Z" level=error msg="HTTP Error" err="No such image: agentfitz/private-test:latest" statusCode=404
time="2016-03-03T03:01:28.928702959Z" level=info msg="POST /v1.21/images/create?fromImage=agentfitz%2Fprivate-test&tag=latest"
time="2016-03-03T03:06:08.746702200Z" level=info msg="POST /v1.21/containers/create?name=lucee-test"
time="2016-03-03T03:06:08.747811890Z" level=error msg="Handler for POST /v1.21/containers/create returned error: No such image: lucee/lucee4-nginx:latest"
time="2016-03-03T03:06:08.747861129Z" level=error msg="HTTP Error" err="No such image: lucee/lucee4-nginx:latest" statusCode=404
time="2016-03-03T03:06:08.748621234Z" level=info msg="POST /v1.21/images/create?fromImage=lucee%2Flucee4-nginx&tag=latest"
time="2016-03-03T03:07:45.392318607Z" level=info msg="POST /v1.21/containers/create?name=lucee-test"
time="2016-03-03T03:07:47.397228250Z" level=info msg="POST /v1.21/containers/73f54a7cfa2a0a0aa15a61f5f3d83d7df8cf6a7fc2fadd132ea999ff8ea86e64/start"
samuelkarp commented 8 years ago

@agentfitz If you're using Elastic Beanstalk's Single Container Docker Environment, you'll get better help from the Elastic Beanstalk forum.

agentfitz commented 8 years ago

Oh, sorry. "Better help" might be a stretch based on my thread with 5 days and no response :) That said, I have signed up for a developer support account; even still, no one seems to have answers regarding why connecting to a private docker image is so problematic. Well, guess I'm in the wrong place (after Googling "unable to pull private images from docker hub,") thanks anyway.

beregond commented 8 years ago

@samuelkarp

Some info I've gathered:

[root@ip-172-31-7-162 ecs]# docker --version
Docker version 1.9.1, build a34a1d5/1.9.1
[root@ip-172-31-7-162 ecs]# docker ps
CONTAINER ID        IMAGE                            COMMAND             CREATED             STATUS              PORTS                        NAMES
7821d341f50e        amazon/amazon-ecs-agent:latest   "/agent"            9 minutes ago       Up 9 minutes        127.0.0.1:51678->51678/tcp   ecs-agent
[root@ip-172-31-7-162 ecs]# cat ecs.config 
AUTH_TYPE=docker
ECS_ENGINE_AUTH_DATA={"https://index.docker.io/v1/":{"username":"xxx","password":"xxx","email":"szczepan.cieslik@gmail.com"}}
ECS_CLUSTER=staging
ECS_LOGLEVEL=debug

although I've tried also with:

[root@ip-172-31-7-162 ecs]# cat ecs.config 
AUTH_TYPE=dockercfg
ECS_ENGINE_AUTH_DATA={"https://index.docker.io/v1/":{"auth":"xxx","email":"szczepan.cieslik@gmail.com"}}

Still can't fetch private image: http://pastebin.com/UWJcqLnP

beregond commented 8 years ago

Also (dunno if important) I can pull this image with no problem while on host machine and then deploy works (since the image is pulled already).

samuelkarp commented 8 years ago

@beregond What you have as AUTH_TYPE should be ECS_ENGINE_AUTH_TYPE.

beregond commented 8 years ago

@samuelkarp Ouch, you're right. I think I needed fresh look, so thanks for quick answer! :)