Error when pulling images from remote hosts with Registry:2.3 & Docker 1.10

[jmlrt]

Hi, Since docker upgrade to 1.10, I can't no more pull images from remote docker hosts.

• When trying to pull an image in my private registry from local docker, it's OK
• When trying to pull an image in my private registry from remote docker hosts, it stays with message "retrying in X seconds" and never succeed to pull my images

Registry configuration:

version: '2'

services:
    registry:
        restart: always
        image: registry:2
        container_name: registry
        environment:
            REGISTRY_STORAGE: s3
            REGISTRY_STORAGE_S3_REGION: eu-west-1
            REGISTRY_STORAGE_S3_BUCKET: XXXXXXXXX
            REGISTRY_STORAGE_S3_ACCESSKEY: XXXXXXXXXX
            REGISTRY_STORAGE_S3_SECRETKEY: XXXXXXXXXXX
            REGISTRY_HTTP_TLS_CERTIFICATE: /conf/domain.crt
            REGISTRY_HTTP_TLS_KEY: /conf/domain.key
            REGISTRY_AUTH: htpasswd
            REGISTRY_AUTH_HTPASSWD_PATH: /conf/auth
            REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
        ports:
            - "5000:5000"
        volumes:
            - /etc/docker-registry:/conf

Docker server running Registry:

docker info
Containers: 15
 Running: 10
 Paused: 0
 Stopped: 5
Images: 448
Server Version: 1.10.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 561
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 14.72 GiB

docker remote host:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 7
Server Version: 1.10.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 130
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: null host bridge
Kernel Version: 3.16.0-60-generic
Operating System: Ubuntu 14.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 11.33 GiB

Error when pulling images from remote host:

docker pull docker-1:5000/jenkins-slave
Using default tag: latest
latest: Pulling from jenkins-slave
d4bce7fd68df: Retrying in 5 seconds 
a3ed95caeb02: Retrying in 5 seconds 
816152842605: Retrying in 5 seconds 

docker logs registry:

192.168.1.1 - - [15/Feb/2016:17:28:33 +0000] "GET /v2/jenkins-slave/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.1" 307 344 "" "docker/1.10.1 go/go1.5.3 git-commit/9e83765 kernel/3.16.0-60-generic os/linux arch/amd64"
time="2016-02-15T17:28:34Z" level=info msg="response completed" go.version=go1.5.3 http.request.host="docker-1:5000" http.request.id=278ef888-7440-490c-ac45-ef1f72ce955b http.request.method=GET http.request.remoteaddr="192.168.1.1:49500" http.request.uri="/v2/jenkins-slave/blobs/sha256:d4bce7fd68df2e8bb04e317e7cb7899e981159a4da89339e38c8bf30e6c318f0" http.request.useragent="docker/1.10.1 go/go1.5.3 git-commit/9e83765 kernel/3.16.0-60-generic os/linux arch/amd64" http.response.contenttype="application/octet-stream" http.response.duration=471.622036ms http.response.status=307 http.response.written=338 instance.id=34fa54fc-9cba-4f0c-93d3-205b1a57de4c version=v2.3.0 
192.168.1.1 - - [15/Feb/2016:17:28:33 +0000] "GET /v2/jenkins-slave/blobs/sha256:d4bce7fd68df2e8bb04e317e7cb7899e981159a4da89339e38c8bf30e6c318f0 HTTP/1.1" 307 338 "" "docker/1.10.1 go/go1.5.3 git-commit/9e83765 kernel/3.16.0-60-generic os/linux arch/amd64"
time="2016-02-15T17:28:34Z" level=info msg="response completed" go.version=go1.5.3 http.request.host="docker-1:5000" http.request.id=65ff470f-718b-4d6f-a826-3c093a69e693 http.request.method=GET http.request.remoteaddr="192.168.1.1:49501" http.request.uri="/v2/jenkins-slave/blobs/sha256:816152842605fe3ede1dc7c47f33e641f74cb4ae0d5c51a6c19cc8d85da934f3" http.request.useragent="docker/1.10.1 go/go1.5.3 git-commit/9e83765 kernel/3.16.0-60-generic os/linux arch/amd64" http.response.contenttype="application/octet-stream" http.response.duration=545.94185ms http.response.status=307 http.response.written=340 instance.id=34fa54fc-9cba-4f0c-93d3-205b1a57de4c version=v2.3.0 

[dmp42]

Can you copy your docker daemon logs?

[jmlrt]

Docker daemon logs:

Feb 15 18:17:04 docker-1 docker[807]: time="2016-02-15T18:17:04.088644060Z" level=warning msg="Your kernel does not support OOM notifications: There is no path for \"memory\" in state"
Feb 15 18:17:04 docker-1 docker[807]: time="2016-02-15T18:17:04.088671730Z" level=warning msg="Your kernel does not support OOM notifications: There is no path for \"memory\" in state"
Feb 15 19:17:02 docker-1 docker[807]: time="2016-02-15T19:17:02.196443384Z" level=warning msg="Your kernel does not support OOM notifications: There is no path for \"memory\" in state"
Feb 15 19:17:02 docker-1 docker[807]: time="2016-02-15T19:17:02.196473207Z" level=warning msg="Your kernel does not support OOM notifications: There is no path for \"memory\" in state"
Feb 15 20:17:02 docker-1 docker[807]: time="2016-02-15T20:17:02.039462683Z" level=warning msg="Your kernel does not support OOM notifications: There is no path for \"memory\" in state"
Feb 15 20:17:02 docker-1 docker[807]: time="2016-02-15T20:17:02.039495163Z" level=warning msg="Your kernel does not support OOM notifications: There is no path for \"memory\" in state"
Feb 15 21:17:02 docker-1 docker[807]: time="2016-02-15T21:17:02.434577006Z" level=warning msg="Your kernel does not support OOM notifications: There is no path for \"memory\" in state"
Feb 15 21:17:02 docker-1 docker[807]: time="2016-02-15T21:17:02.434605710Z" level=warning msg="Your kernel does not support OOM notifications: There is no path for \"memory\" in state"
Feb 15 22:17:01 docker-1 docker[807]: time="2016-02-15T22:17:01.964408939Z" level=warning msg="Your kernel does not support OOM notifications: There is no path for \"memory\" in state"
Feb 15 22:17:01 docker-1 docker[807]: time="2016-02-15T22:17:01.964436258Z" level=warning msg="Your kernel does not support OOM notifications: There is no path for \"memory\" in state"

[dmp42]

@jmlrt none of this is showing any pull attempt.

Suggestion:

• stop the docker daemon
• enable debug mode on the daemon (add -D to the default launch arguments)
• start docker
• tail -f your daemon logs
• try to pull, and reproduce the problem
• copy the resulting logs

[jmlrt]

Hi @dmp42, My Docker server is used for production environment. I can't find a window to restart it with debug mode for now. I'll do it & post logs when I can