search

concourse / docker-image-resource

a resource for docker images
Apache License 2.0
165 stars 259 forks source link

Cannot pull from dockerhub within docker-compose setup #137

Open chris-aeviator opened 7 years ago

chris-aeviator commented 7 years ago

When trying to add a resource from dockerhub (compare hello world tutorial) task will fail when using the the setup through official docker-compose.yml.

When changing the docker registry mirror to e.g. http://quay.io the tasi will perfectly pull the image.

This issue is basically blocking all work I'm planing to do with concourse and I'd be really thankful for a workaround?!

Config dockerhub

- name: rsync-resource
  type: docker-image
  source:
      repository: mrsixw/concourse-rsync-resource
      tag: latest

Output dockerhub:

sha256:6f1e44344622359b89797538916bea5e27566fea5f79e3fbd8bc8e8df50341e6: Pulling from mrsixw/concourse-rsync-resource
b0348695caf2: Pulling fs layer
204ea97fe778: Pulling fs layer
416b5a2739d4: Pulling fs layer
error pulling image configuration: Get https://dseasb33srnrn.cloudfront.net/registry-v2/docker/registry/v2/blobs/sha256/5a/5a551f16b2a80da331e0b7eb4d908892a1424fc1cd83f86709f02d2b02cc5ee9/data?Expires=1503433983&Signature=UFBxOqGoi9QkDYP8omqklN4CbJrdpZZLsYftz93ur245CzIRCSAee5oD6DIVAfD95XgyMP4dpxx0OfJQ2NCuHRaK7pexQdbXDwk7nbumYFQ6XixgrVtMUk1zn9gWLRPpjV-e09dtpVIWTmsZbZ47kOEpq5EmafM9CV0eCZ3AqKU_&Key-Pair-Id=APKAJECH5M7VWIS5YZ6Q: dial tcp: lookup dseasb33srnrn.cloudfront.net on 172.18.0.4:53: read udp 10.254.0.90:48799->172.18.0.4:53: i/o timeout

Pulling mrsixw/concourse-rsync-resource@sha256:6f1e44344622359b89797538916bea5e27566fea5f79e3fbd8bc8e8df50341e6 (attempt 2 of 3)...
sha256:6f1e44344622359b89797538916bea5e27566fea5f79e3fbd8bc8e8df50341e6: Pulling from mrsixw/concourse-rsync-resource
b0348695caf2: Pulling fs layer
204ea97fe778: Pulling fs layer
416b5a2739d4: Pulling fs layer
error pulling image configuration: Get https://dseasb33srnrn.cloudfront.net/registry-v2/docker/registry/v2/blobs/sha256/5a/5a551f16b2a80da331e0b7eb4d908892a1424fc1cd83f86709f02d2b02cc5ee9/data?Expires=1503433995&Signature=GXhG~-SE2oy~cSoL753mLk85WCIfgyFKi47h~s1DXaA4hPoLm7VPaCwSR7UVHXlyodJSYnP4kPDe-QGn~mKMuC2CJNm2dpesLJQKo-VQOrRwq17XaH6hZe98Fz4lRgk4OtOGLbWoHkD3hcLlmx1wB7jrE2xHq2xaRMpBjHZM2ek_&Key-Pair-Id=APKAJECH5M7VWIS5YZ6Q: dial tcp: lookup dseasb33srnrn.cloudfront.net on 172.18.0.4:53: read udp 10.254.0.90:58077->172.18.0.4:53: i/o timeout

Pulling mrsixw/concourse-rsync-resource@sha256:6f1e44344622359b89797538916bea5e27566fea5f79e3fbd8bc8e8df50341e6 (attempt 3 of 3)...
sha256:6f1e44344622359b89797538916bea5e27566fea5f79e3fbd8bc8e8df50341e6: Pulling from mrsixw/concourse-rsync-resource
b0348695caf2: Pulling fs layer
204ea97fe778: Pulling fs layer
416b5a2739d4: Pulling fs layer
error pulling image configuration: Get https://dseasb33srnrn.cloudfront.net/registry-v2/docker/registry/v2/blobs/sha256/5a/5a551f16b2a80da331e0b7eb4d908892a1424fc1cd83f86709f02d2b02cc5ee9/data?Expires=1503434006&Signature=RSbEK-pQ7PLLHy6VlhsSjD0bmm83vaieanITGUWElZT0AW7Hqsp3cXQQFPBRp3fXkosIBDYdMXMkM8wyfreGT1hyQLPrRIyPCB0z-uUW0k1vSwloHq1zxWe~E5mFBWS7yY5zJQNVuz4BJSbMQj271uwtftHiv-WffJXvCVQwO0Y_&Key-Pair-Id=APKAJECH5M7VWIS5YZ6Q: dial tcp: lookup dseasb33srnrn.cloudfront.net on 172.18.0.4:53: read udp 10.254.0.90:53558->172.18.0.4:53: i/o timeout

Config registry mirror

- task: web-app-tests
    config:
      platform: linux
      image_resource:
          type: docker-image
          source:
            registry_mirror: http://quay.io
            repository: jgwmaxwell/debian
            tag: latest

output registry mirror

Pulling jgwmaxwell/debian@sha256:9d87f169eb52534739c17d44194c8ab5d4131ad52f4b1e1f2fd3c53a5c55ac19...
sha256:9d87f169eb52534739c17d44194c8ab5d4131ad52f4b1e1f2fd3c53a5c55ac19: Pulling from jgwmaxwell/debian

downloads fine

chris-aeviator commented 7 years ago

as a quick solution I am now pulling the needed Docker images to the host machine and running my own docker-registry in the docker network where I push the images from the host maschine.

That way concourse is able to find the images when adding like so:

resource_types:
- name: rsync-resource
  type: docker-image
  source:
      insecure_registries: ["172.18.0.5:5000"]
      repository: 172.18.0.5:5000/concourse-rsync
      tag: master
jtarchie commented 7 years ago

@chris-aeviator, it looks like your local DNS is not able to lookup dockerhub.

Can you run dig hub.docker.com on your host machine, in a container, and on a task?

eembsen commented 6 years ago

Exact same problem here, and also switched to the workaround as described by Chris. I can run dig on the host machine and it resolves:

; <<>> DiG 9.10.3-P4-Ubuntu <<>> hub.docker.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41690
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;hub.docker.com.            IN  A

;; ANSWER SECTION:
hub.docker.com.     20  IN  CNAME   elb-default.us-east-1.aws.dckr.io.
elb-default.us-east-1.aws.dckr.io. 139 IN CNAME us-east-1-elbdefau-1nlhaqqbnj2z8-140214243.us-east-1.elb.amazonaws.com.
us-east-1-elbdefau-1nlhaqqbnj2z8-140214243.us-east-1.elb.amazonaws.com. 39 IN A 52.73.244.162
us-east-1-elbdefau-1nlhaqqbnj2z8-140214243.us-east-1.elb.amazonaws.com. 39 IN A 52.55.168.191
us-east-1-elbdefau-1nlhaqqbnj2z8-140214243.us-east-1.elb.amazonaws.com. 39 IN A 34.201.187.190

;; AUTHORITY SECTION:
us-east-1.elb.amazonaws.com. 1364 IN    NS  ns-235.awsdns-29.com.
us-east-1.elb.amazonaws.com. 1364 IN    NS  ns-934.awsdns-52.net.
us-east-1.elb.amazonaws.com. 1364 IN    NS  ns-1119.awsdns-11.org.
us-east-1.elb.amazonaws.com. 1364 IN    NS  ns-1793.awsdns-32.co.uk.

;; Query time: 2 msec
;; SERVER: 213.133.98.98#53(213.133.98.98)
;; WHEN: Sat Sep 09 09:03:58 CEST 2017
;; MSG SIZE  rcvd: 356

In the concourse worker container this also works but I had to install dnsutils first. Did a docker exec to test this, btw.

On a task it fails with the above error message. Task def is:

    config:
      platform: linux
      image_resource:
        type: docker-image
        source:
          repository: ubuntu
          tag: 16.04
      run:
        path: bash
        args:
        - -exc
        - |
          dig hub.docker.com

UFW and fail2ban were stopped/disabled in all test cases.

Cheers, Erwin

daniellavoie commented 6 years ago

I've got exactly the same symptoms described by @eembsen. Any reasonable explanation on this? Why would the DNS problems occur only for the containers spawned by our tasks?

dvianello commented 6 years ago

We managed to get around it by iptables DNAT-ing the DNS packets to a different DNS server than the one hosted by the Concourse Worker process. Any idea why the DNS on the Worker does not work with the resource trying to upload the image, but yet works from the hijacked container?