search

docker-archive / docker-registry

This is **DEPRECATED**! Please go to https://github.com/docker/distribution
Apache License 2.0
2.88k stars 877 forks source link

Push fails with EOF error #989

Closed Trefex closed 9 years ago

Trefex commented 9 years ago

Dear all,

When pushing to my registry, I get the following error:

~ ❯ docker push docker-r3lab.uni.lu/trefex/ubu                                                                                                                                                                                                 06:27:02 ⏎
The push refers to a repository [docker-r3lab.uni.lu/trefex/ubu] (len: 1)
Sending image list
Pushing repository docker-r3lab.uni.lu/trefex/ubu (1 tags)
511136ea3c5a: Pushing [==================================================>] 1.024 kB/1.024 kB
FATA[0031] Failed to upload layer: Put https://docker-r3lab.uni.lu/v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/layer: EOF

I am behind an Apache Traffic Server, which maps docker-r3lab to docker-r3lab-server.

Here is my nginx config

upstream docker-registry {
 server 0.0.0.0:5000;
}
server {
    listen 443;
    server_name docker-r3lab-server.uni.lu;

    ssl on;
    ssl_certificate /etc/ssl/certs/docker-r3lab-server;

    ssl_certificate_key /etc/ssl/private/docker-r3lab-server;

    proxy_set_header Host       $http_host;   # required for docker client's sake
    proxy_set_header X-Real-IP  $remote_addr; # pass on real client's IP
    proxy_set_header Authorization    "";

    client_max_body_size 0; # disable any limits to avoid HTTP 413 for large image uploads

    # required to avoid HTTP 411: see Issue #1486 (https://github.com/dotcloud/docker/issues/1486)
    chunked_transfer_encoding on;
  location / {
     proxy_pass http://docker-registry;
    proxy_set_header Host $host;
  }

  location /_ping {
     auth_basic off;
     proxy_pass http://docker-registry;
  }

  location /v1/_ping {
     auth_basic off;
     proxy_pass http://docker-registry;
  }

  location /v1/search {
    auth_basic off;
    proxy_pass http://docker-registry;
  }
}

Does anybody have any ideas? Where to start? How to debug?

root@docker-r3lab:~# docker -v Docker version 1.5.0, build a8a31ef

root@boot2docker:~# docker -v Docker version 1.5.0, build a8a31ef

boot2docker docker daemon log excerpt

INFO[0006] POST /v1.17/images/docker-r3lab.uni.lu/centos/push?tag=
INFO[0006] +job push(docker-r3lab.uni.lu/centos)
INFO[0006] +job resolve_repository(docker-r3lab.uni.lu/centos)
INFO[0006] -job resolve_repository(docker-r3lab.uni.lu/centos) = OK (0)
DEBU[0006] pinging registry endpoint https://docker-r3lab.uni.lu/v0/
DEBU[0006] attempting v2 ping for registry endpoint https://docker-r3lab.uni.lu/v2/
DEBU[0006] hostDir: /etc/docker/certs.d/docker-r3lab.uni.lu
DEBU[0006] attempting v1 ping for registry endpoint https://docker-r3lab.uni.lu/v1/
DEBU[0006] hostDir: /etc/docker/certs.d/docker-r3lab.uni.lu
DEBU[0006] RegistryInfo.Version: ""
DEBU[0006] Registry standalone header: 'True'
DEBU[0006] RegistryInfo.Standalone: true
DEBU[0006] attempting v1 ping for registry endpoint https://docker-r3lab.uni.lu/v1/
DEBU[0006] hostDir: /etc/docker/certs.d/docker-r3lab.uni.lu
DEBU[0006] RegistryInfo.Version: ""
DEBU[0006] Registry standalone header: 'True'
DEBU[0006] RegistryInfo.Standalone: true
DEBU[0006] Endpoint https://docker-r3lab.uni.lu/v1/ is eligible for private registry registry. Enabling decorator.
DEBU[0006] Local repo: map[latest:214a4932132a05e9cc97624436a77743f368948c4776b186975d319140e8b45c]
DEBU[0006] Image list: [511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158 b6718650e87e3706c52682c87ecfd7a7e1fc176c9095d73d627ca41b2584839b 41054cbffecca8ca909a0c4c163eb0ddb25f50311a96de136239d3b4287b3618 214a4932132a05e9cc97624436a77743f368948c4776b186975d319140e8b45c]
DEBU[0006] Tags by image: map[214a4932132a05e9cc97624436a77743f368948c4776b186975d319140e8b45c:[latest]]
DEBU[0006] Preparing to push map[latest:214a4932132a05e9cc97624436a77743f368948c4776b186975d319140e8b45c] with the following images and tags
DEBU[0006] Pushing ID: 511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158 with Tag:
DEBU[0006] Pushing ID: b6718650e87e3706c52682c87ecfd7a7e1fc176c9095d73d627ca41b2584839b with Tag:
DEBU[0006] Pushing ID: 41054cbffecca8ca909a0c4c163eb0ddb25f50311a96de136239d3b4287b3618 with Tag:
DEBU[0006] Pushing ID: 214a4932132a05e9cc97624436a77743f368948c4776b186975d319140e8b45c with Tag: latest
DEBU[0006] [registry] PUT https://docker-r3lab.uni.lu/v1/repositories/centos/
DEBU[0006] Image list pushed to index:
[{"id":"511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158"},{"id":"b6718650e87e3706c52682c87ecfd7a7e1fc176c9095d73d627ca41b2584839b"},{"id":"41054cbffecca8ca909a0c4c163eb0ddb25f50311a96de136239d3b4287b3618"},{"id":"214a4932132a05e9cc97624436a77743f368948c4776b186975d319140e8b45c","Tag":"latest"}]
DEBU[0006] https://docker-r3lab.uni.lu/v1/repositories/centos/ -- HEADERS: map[User-Agent:[docker/1.5.0 go/go1.4.1 git-commit/a8a31ef kernel/3.18.5-tinycore64 os/linux arch/amd64] Authorization:[Basic Og==]]
DEBU[0006] hostDir: /etc/docker/certs.d/docker-r3lab.uni.lu
DEBU[0006] Auth token: [Token signature=VZQ7XEVAI81GLY7S,repository="library/centos",access=write]
DEBU[0006] https://docker-r3lab.uni.lu/v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/json -- HEADERS: map[User-Agent:[docker/1.5.0 go/go1.4.1 git-commit/a8a31ef kernel/3.18.5-tinycore64 os/linux arch/amd64] Authorization:[Basic Og==]]
DEBU[0006] hostDir: /etc/docker/certs.d/docker-r3lab.uni.lu
DEBU[0006] https://docker-r3lab.uni.lu/v1/images/b6718650e87e3706c52682c87ecfd7a7e1fc176c9095d73d627ca41b2584839b/json -- HEADERS: map[User-Agent:[docker/1.5.0 go/go1.4.1 git-commit/a8a31ef kernel/3.18.5-tinycore64 os/linux arch/amd64] Authorization:[Basic Og==]]
DEBU[0006] hostDir: /etc/docker/certs.d/docker-r3lab.uni.lu
DEBU[0006] https://docker-r3lab.uni.lu/v1/images/41054cbffecca8ca909a0c4c163eb0ddb25f50311a96de136239d3b4287b3618/json -- HEADERS: map[User-Agent:[docker/1.5.0 go/go1.4.1 git-commit/a8a31ef kernel/3.18.5-tinycore64 os/linux arch/amd64] Authorization:[Basic Og==]]
DEBU[0006] hostDir: /etc/docker/certs.d/docker-r3lab.uni.lu
DEBU[0006] https://docker-r3lab.uni.lu/v1/images/214a4932132a05e9cc97624436a77743f368948c4776b186975d319140e8b45c/json -- HEADERS: map[User-Agent:[docker/1.5.0 go/go1.4.1 git-commit/a8a31ef kernel/3.18.5-tinycore64 os/linux arch/amd64] Authorization:[Basic Og==]]
DEBU[0006] hostDir: /etc/docker/certs.d/docker-r3lab.uni.lu
ERRO[0006] Error in LookupRemoteImage: HTTP code 404
ERRO[0006] Error in LookupRemoteImage: HTTP code 404
ERRO[0006] Error in LookupRemoteImage: HTTP code 400
ERRO[0006] Error in LookupRemoteImage: HTTP code 404
DEBU[0006] [registry] Calling PUT https://docker-r3lab.uni.lu/v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/json
DEBU[0006] https://docker-r3lab.uni.lu/v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/json -- HEADERS: map[User-Agent:[docker/1.5.0 go/go1.4.1 git-commit/a8a31ef kernel/3.18.5-tinycore64 os/linux arch/amd64] Authorization:[Basic Og==]]
DEBU[0006] hostDir: /etc/docker/certs.d/docker-r3lab.uni.lu
DEBU[0006] rendered layer for 511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158 of [1024] size
DEBU[0006] [registry] Calling PUT https://docker-r3lab.uni.lu/v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/layer
DEBU[0006] https://docker-r3lab.uni.lu/v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/layer -- HEADERS: map[User-Agent:[docker/1.5.0 go/go1.4.1 git-commit/a8a31ef kernel/3.18.5-tinycore64 os/linux arch/amd64] Authorization:[Basic Og==]]
DEBU[0006] hostDir: /etc/docker/certs.d/docker-r3lab.uni.lu
Failed to upload layer: Put https://docker-r3lab.uni.lu/v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/layer: EOF
INFO[0036] -job push(docker-r3lab.uni.lu/centos) = ERR (1)

Launch settings of docker registry

sudo docker run -p 5000:5000 -v /root/:/root/ -e DOCKER_REGISTRY_CONFIG=/root/dockerconfig.yml -e STORAGE_PATH=/root/dockerstorage -e SETTINGS_FLAVOR=test registry:0.9.1

Docker registry config

root@docker-r3lab:~/dockerstorage# cat /root/dockerconfig.yml
common: &common
    standalone: true
    loglevel: debug
    storage: local
    storage_path: /root/dockerstorage
#    search_backend: sqlalchemy
#    sqlalchemy_index_database: sqlite:////tmp/docker-registry.db

prod:
    <<: *common
    loglevel: error
    smtp_host: smtp.uni.lu
    from_addr: docker@myself.com
    to_addr: christophe.trefois@uni.lu

test:
    <<: *common
    loglevel: debug
    debug: true
    smtp_host: smtp.uni.lu
    from_addr: docker@myself.com
    to_addr: christophe.trefois@uni.lu
root@docker-r3lab:~/dockerstorage#
dmp42 commented 9 years ago

Hey @Trefex

Can you copy your docker registry logs?

jgangemi commented 9 years ago

i've seen this happen if the X-Registry-Auth header is missing from the request, are you sure that's getting through?

Trefex commented 9 years ago

@dmp42

Here it is, to me it doesn't look very promising

172.17.42.1 - - [10/Apr/2015:19:45:20 +0000] "GET /v2/ HTTP/1.0" 404 233 "-" "Go 1.1 package http"
172.17.42.1 - - [10/Apr/2015:19:45:20 +0000] "GET /v1/_ping HTTP/1.0" 200 1517 "-" "Go 1.1 package http"
172.17.42.1 - - [10/Apr/2015:19:45:20 +0000] "GET /v1/_ping HTTP/1.0" 200 1517 "-" "Go 1.1 package http"
10/Apr/2015:19:45:20 +0000 DEBUG: args = {'namespace': u'trefex', 'repository': u'toto'}
172.17.42.1 - - [10/Apr/2015:19:45:20 +0000] "PUT /v1/repositories/trefex/toto/ HTTP/1.0" 200 2 "-" "docker/1.5.0 go/go1.4.1 git-commit/a8a31ef kernel/3.2.0-61-generic os/linux arch/amd64"
10/Apr/2015:19:45:20 +0000 DEBUG: args = {'image_id': u'ea13149945cb6b1e746bf28032f02e9b5a793523481a0a18645fc77ad53c4ea2'}
10/Apr/2015:19:45:20 +0000 DEBUG: api_error: Image not found
10/Apr/2015:19:45:20 +0000 DEBUG: args = {'image_id': u'4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8125'}
10/Apr/2015:19:45:20 +0000 DEBUG: api_error: Image not found
172.17.42.1 - - [10/Apr/2015:19:45:20 +0000] "GET /v1/images/4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8125/json HTTP/1.0" 404 28 "-" "docker/1.5.0 go/go1.4.1 git-commit/a8a31ef kernel/3.2.0-61-generic os/linux arch/amd64"
172.17.42.1 - - [10/Apr/2015:19:45:20 +0000] "GET /v1/images/ea13149945cb6b1e746bf28032f02e9b5a793523481a0a18645fc77ad53c4ea2/json HTTP/1.0" 404 28 "-" "docker/1.5.0 go/go1.4.1 git-commit/a8a31ef kernel/3.2.0-61-generic os/linux arch/amd64"
10/Apr/2015:19:45:20 +0000 DEBUG: args = {'image_id': u'df7546f9f060a2268024c8a230d8639878585defcc1bc6f79d2728a13957871b'}
10/Apr/2015:19:45:20 +0000 DEBUG: api_error: Image not found
172.17.42.1 - - [10/Apr/2015:19:45:20 +0000] "GET /v1/images/df7546f9f060a2268024c8a230d8639878585defcc1bc6f79d2728a13957871b/json HTTP/1.0" 404 28 "-" "docker/1.5.0 go/go1.4.1 git-commit/a8a31ef kernel/3.2.0-61-generic os/linux arch/amd64"
10/Apr/2015:19:45:20 +0000 DEBUG: args = {'image_id': u'511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158'}
10/Apr/2015:19:45:20 +0000 DEBUG: api_error: Image is being uploaded, retry later
172.17.42.1 - - [10/Apr/2015:19:45:20 +0000] "GET /v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/json HTTP/1.0" 400 49 "-" "docker/1.5.0 go/go1.4.1 git-commit/a8a31ef kernel/3.2.0-61-generic os/linux arch/amd64"
10/Apr/2015:19:45:20 +0000 DEBUG: args = {'image_id': u'511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158'}
172.17.42.1 - - [10/Apr/2015:19:45:20 +0000] "PUT /v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/json HTTP/1.0" 200 4 "-" "docker/1.5.0 go/go1.4.1 git-commit/a8a31ef kernel/3.2.0-61-generic os/linux arch/amd64"
dmp42 commented 9 years ago

Anything in nginx logs?

Trefex commented 9 years ago

@dmp42

==> /var/log/nginx/access.log <==
10.4.0.51 - - [10/Apr/2015:21:59:40 +0200] "GET /v2/ HTTP/1.1" 404 233 "-" "Go 1.1 package http"
10.4.0.51 - - [10/Apr/2015:21:59:40 +0200] "GET /v1/_ping HTTP/1.1" 200 1517 "-" "Go 1.1 package http"
10.4.0.51 - - [10/Apr/2015:21:59:40 +0200] "GET /v1/_ping HTTP/1.1" 200 1517 "-" "Go 1.1 package http"
10.4.0.51 - - [10/Apr/2015:21:59:40 +0200] "PUT /v1/repositories/trefex/toto/ HTTP/1.1" 200 2 "-" "docker/1.5.0 go/go1.4.1 git-commit/a8a31ef kernel/3.2.0-61-generic os/linux arch/amd64"
10.4.0.51 - - [10/Apr/2015:21:59:40 +0200] "GET /v1/images/df7546f9f060a2268024c8a230d8639878585defcc1bc6f79d2728a13957871b/json HTTP/1.1" 404 28 "-" "docker/1.5.0 go/go1.4.1 git-commit/a8a31ef kernel/3.2.0-61-generic os/linux arch/amd64"
10.4.0.51 - - [10/Apr/2015:21:59:40 +0200] "GET /v1/images/ea13149945cb6b1e746bf28032f02e9b5a793523481a0a18645fc77ad53c4ea2/json HTTP/1.1" 404 28 "-" "docker/1.5.0 go/go1.4.1 git-commit/a8a31ef kernel/3.2.0-61-generic os/linux arch/amd64"
10.4.0.51 - - [10/Apr/2015:21:59:40 +0200] "GET /v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/json HTTP/1.1" 400 49 "-" "docker/1.5.0 go/go1.4.1 git-commit/a8a31ef kernel/3.2.0-61-generic os/linux arch/amd64"
10.4.0.51 - - [10/Apr/2015:21:59:40 +0200] "GET /v1/images/4986bf8c15363d1c5d15512d5266f8777bfba4974ac56e3270e7760f6f0a8125/json HTTP/1.1" 404 28 "-" "docker/1.5.0 go/go1.4.1 git-commit/a8a31ef kernel/3.2.0-61-generic os/linux arch/amd64"
10.4.0.51 - - [10/Apr/2015:21:59:40 +0200] "PUT /v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/json HTTP/1.1" 200 4 "-" "docker/1.5.0 go/go1.4.1 git-commit/a8a31ef kernel/3.2.0-61-generic os/linux arch/amd64"
10.4.0.51 - - [10/Apr/2015:22:00:10 +0200] "PUT /v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/layer HTTP/1.1" 400 0 "-" "docker/1.5.0 go/go1.4.1 git-commit/a8a31ef kernel/3.2.0-61-generic os/linux arch/amd64"
dmp42 commented 9 years ago

No errors of any kind, neither in registry or nginx? If the client connection is closed, that should show somewhere...

Trefex commented 9 years ago

@dmp42

Here some more verbose log for the PUT part after switching on debug in nginx

What surprises me is the fact that it says connection closed prematurely by client, but I don't see why this could be.

Also, the first layer is only like 1 MB or something.

2015/04/10 22:04:50 [debug] 5933#0: *16 http request line: "PUT /v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/layer HTTP/1.1"
2015/04/10 22:04:50 [debug] 5933#0: *16 http uri: "/v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/layer"
2015/04/10 22:04:50 [debug] 5933#0: *16 http args: ""
2015/04/10 22:04:50 [debug] 5933#0: *16 http exten: ""
2015/04/10 22:04:50 [debug] 5933#0: *16 http process request header line
2015/04/10 22:04:50 [debug] 5933#0: *16 http header: "Host: docker-r3lab.uni.lu"
2015/04/10 22:04:50 [debug] 5933#0: *16 http header: "User-Agent: docker/1.5.0 go/go1.4.1 git-commit/a8a31ef kernel/3.2.0-61-generic os/linux arch/amd64"
2015/04/10 22:04:50 [debug] 5933#0: *16 http header: "Transfer-Encoding: chunked"
2015/04/10 22:04:50 [debug] 5933#0: *16 http header: "Authorization: Basic Og=="
2015/04/10 22:04:50 [debug] 5933#0: *16 http header: "Content-Type: application/octet-stream"
2015/04/10 22:04:50 [debug] 5933#0: *16 http header: "Accept-Encoding: gzip"
2015/04/10 22:04:50 [debug] 5933#0: *16 http header: "Client-ip: 10.79.2.28"
2015/04/10 22:04:50 [debug] 5933#0: *16 http header: "X-Forwarded-For: 10.79.2.28"
2015/04/10 22:04:50 [debug] 5933#0: *16 http header: "Via: https/1.1 aratis.uni.lu[FE80000000000000025056FFFE844D03] (ApacheTrafficServer/3.2.4)"
2015/04/10 22:04:50 [debug] 5933#0: *16 http header done
2015/04/10 22:04:50 [debug] 5933#0: *16 event timer del: 17: 1428696350062
2015/04/10 22:04:50 [debug] 5933#0: *16 generic phase: 0
2015/04/10 22:04:50 [debug] 5933#0: *16 rewrite phase: 1
2015/04/10 22:04:50 [debug] 5933#0: *16 test location: "/"
2015/04/10 22:04:50 [debug] 5933#0: *16 test location: "v1/_ping"
2015/04/10 22:04:50 [debug] 5933#0: *16 test location: "v1/search"
2015/04/10 22:04:50 [debug] 5933#0: *16 using configuration "/"
2015/04/10 22:04:50 [debug] 5933#0: *16 http cl:-1 max:0
2015/04/10 22:04:50 [debug] 5933#0: *16 rewrite phase: 3
2015/04/10 22:04:50 [debug] 5933#0: *16 post rewrite phase: 4
2015/04/10 22:04:50 [debug] 5933#0: *16 generic phase: 5
2015/04/10 22:04:50 [debug] 5933#0: *16 generic phase: 6
2015/04/10 22:04:50 [debug] 5933#0: *16 generic phase: 7
2015/04/10 22:04:50 [debug] 5933#0: *16 access phase: 8
2015/04/10 22:04:50 [debug] 5933#0: *16 access phase: 9
2015/04/10 22:04:50 [debug] 5933#0: *16 access phase: 10
2015/04/10 22:04:50 [debug] 5933#0: *16 post access phase: 11
2015/04/10 22:04:50 [debug] 5933#0: *16 posix_memalign: 00000000008A9450:4096 @16
2015/04/10 22:04:50 [debug] 5933#0: *16 http request body chunked filter
2015/04/10 22:04:50 [debug] 5933#0: *16 malloc: 00000000008AA460:8192
2015/04/10 22:04:50 [debug] 5933#0: *16 http read client request body
2015/04/10 22:04:50 [debug] 5933#0: *16 SSL_read: -1
2015/04/10 22:04:50 [debug] 5933#0: *16 SSL_get_error: 2
2015/04/10 22:04:50 [debug] 5933#0: *16 http client request body recv -2
2015/04/10 22:04:50 [debug] 5933#0: *16 http client request body rest 3
2015/04/10 22:04:50 [debug] 5933#0: *16 event timer add: 17: 60000:1428696350070
2015/04/10 22:04:50 [debug] 5933#0: *16 http finalize request: -4, "/v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/layer?" a:1, c:2
2015/04/10 22:04:50 [debug] 5933#0: *16 http request count:2 blk:0
2015/04/10 22:05:20 [debug] 5933#0: *16 post event 000000000086C240
2015/04/10 22:05:20 [debug] 5933#0: *16 delete posted event 000000000086C240
2015/04/10 22:05:20 [debug] 5933#0: *16 http run request: "/v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/layer?"
2015/04/10 22:05:20 [debug] 5933#0: *16 http read client request body
2015/04/10 22:05:20 [debug] 5933#0: *16 SSL_read: 0
2015/04/10 22:05:20 [debug] 5933#0: *16 SSL_get_error: 5
2015/04/10 22:05:20 [debug] 5933#0: *16 peer shutdown SSL cleanly
2015/04/10 22:05:20 [debug] 5933#0: *16 http client request body recv 0
2015/04/10 22:05:20 [info] 5933#0: *16 client prematurely closed connection, client: 10.4.0.51, server: docker-r3lab-server.uni.lu, request: "PUT /v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/layer HTTP/1.1", host: "docker-r3lab.uni.lu"
2015/04/10 22:05:20 [debug] 5933#0: *16 http finalize request: 400, "/v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/layer?" a:1, c:1
2015/04/10 22:05:20 [debug] 5933#0: *16 http terminate request count:1
2015/04/10 22:05:20 [debug] 5933#0: *16 http terminate cleanup count:1 blk:0
2015/04/10 22:05:20 [debug] 5933#0: *16 http posted request: "/v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/layer?"
2015/04/10 22:05:20 [debug] 5933#0: *16 http terminate handler count:1
2015/04/10 22:05:20 [debug] 5933#0: *16 http request count:1 blk:0
2015/04/10 22:05:20 [debug] 5933#0: *16 http close request
2015/04/10 22:05:20 [debug] 5933#0: *16 http log handler
2015/04/10 22:05:20 [debug] 5933#0: *16 free: 00000000008AA460
2015/04/10 22:05:20 [debug] 5933#0: *16 free: 00000000008A13B0, unused: 4
2015/04/10 22:05:20 [debug] 5933#0: *16 free: 00000000008A9450, unused: 2293
2015/04/10 22:05:20 [debug] 5933#0: *16 close http connection: 17
2015/04/10 22:05:20 [debug] 5933#0: *16 SSL_shutdown: 1
2015/04/10 22:05:20 [debug] 5933#0: *16 event timer del: 17: 1428696350070
2015/04/10 22:05:20 [debug] 5933#0: *16 reusable connection: 0
2015/04/10 22:05:20 [debug] 5933#0: *16 free: 0000000000895740
2015/04/10 22:05:20 [debug] 5933#0: *16 free: 0000000000893ED0, unused: 0
2015/04/10 22:05:20 [debug] 5933#0: *16 free: 0000000000893FE0, unused: 72
2015/04/10 22:05:54 [debug] 5933#0: *1 event timer del: 11: 1428696354803
2015/04/10 22:05:54 [debug] 5933#0: *1 http keepalive handler
2015/04/10 22:05:54 [debug] 5933#0: *1 close http connection: 11
2015/04/10 22:05:54 [debug] 5933#0: *1 SSL_shutdown: 1
2015/04/10 22:05:54 [debug] 5933#0: *1 reusable connection: 0
2015/04/10 22:05:54 [debug] 5933#0: *1 free: 0000000000000000
2015/04/10 22:05:54 [debug] 5933#0: *1 free: 0000000000000000
2015/04/10 22:05:54 [debug] 5933#0: *1 free: 0000000000839E40, unused: 0
2015/04/10 22:05:54 [debug] 5933#0: *1 free: 000000000086A060, unused: 56
2015/04/10 22:05:54 [debug] 5933#0: *1 free: 0000000000842400, unused: 144
2015/04/10 22:05:54 [debug] 5933#0: *3 event timer del: 12: 1428696354958
2015/04/10 22:05:54 [debug] 5933#0: *3 http keepalive handler
2015/04/10 22:05:54 [debug] 5933#0: *3 close http connection: 12
2015/04/10 22:05:54 [debug] 5933#0: *3 SSL_shutdown: 1
2015/04/10 22:05:54 [debug] 5933#0: *3 reusable connection: 0
2015/04/10 22:05:54 [debug] 5933#0: *3 free: 0000000000000000
2015/04/10 22:05:54 [debug] 5933#0: *3 free: 0000000000000000
2015/04/10 22:05:54 [debug] 5933#0: *3 free: 00000000008431D0, unused: 0
2015/04/10 22:05:54 [debug] 5933#0: *3 free: 00000000008A2B00, unused: 56
2015/04/10 22:05:54 [debug] 5933#0: *3 free: 00000000008A2E70, unused: 144
2015/04/10 22:05:54 [debug] 5933#0: *5 event timer del: 13: 1428696354963
2015/04/10 22:05:54 [debug] 5933#0: *5 http keepalive handler
2015/04/10 22:05:54 [debug] 5933#0: *5 close http connection: 13
2015/04/10 22:05:54 [debug] 5933#0: *5 SSL_shutdown: 1
2015/04/10 22:05:54 [debug] 5933#0: *5 reusable connection: 0
2015/04/10 22:05:54 [debug] 5933#0: *5 free: 0000000000000000
2015/04/10 22:05:54 [debug] 5933#0: *5 free: 0000000000000000
2015/04/10 22:05:54 [debug] 5933#0: *5 free: 0000000000848EC0, unused: 0
2015/04/10 22:05:54 [debug] 5933#0: *5 free: 0000000000848FD0, unused: 56
2015/04/10 22:05:54 [debug] 5933#0: *5 free: 00000000008A23C0, unused: 144
2015/04/10 22:05:54 [debug] 5933#0: *7 event timer del: 14: 1428696354966
2015/04/10 22:05:54 [debug] 5933#0: *7 http keepalive handler
2015/04/10 22:05:54 [debug] 5933#0: *7 close http connection: 14
2015/04/10 22:05:54 [debug] 5933#0: *7 SSL_shutdown: 1
2015/04/10 22:05:54 [debug] 5933#0: *7 reusable connection: 0
2015/04/10 22:05:54 [debug] 5933#0: *7 free: 0000000000000000
2015/04/10 22:05:54 [debug] 5933#0: *7 free: 0000000000000000
2015/04/10 22:05:54 [debug] 5933#0: *7 free: 00000000008A6FA0, unused: 0
2015/04/10 22:05:54 [debug] 5933#0: *7 free: 00000000008A70B0, unused: 32
2015/04/10 22:05:54 [debug] 5933#0: *7 free: 00000000008A78F0, unused: 144
2015/04/10 22:05:54 [debug] 5933#0: *12 event timer del: 15: 1428696354974
2015/04/10 22:05:54 [debug] 5933#0: *12 http keepalive handler
2015/04/10 22:05:54 [debug] 5933#0: *12 close http connection: 15
2015/04/10 22:05:54 [debug] 5933#0: *12 SSL_shutdown: 1
2015/04/10 22:05:54 [debug] 5933#0: *12 reusable connection: 0
2015/04/10 22:05:54 [debug] 5933#0: *12 free: 0000000000000000
2015/04/10 22:05:54 [debug] 5933#0: *12 free: 0000000000000000
2015/04/10 22:05:54 [debug] 5933#0: *12 free: 00000000008A74E0, unused: 0
2015/04/10 22:05:54 [debug] 5933#0: *12 free: 00000000008A75F0, unused: 56
2015/04/10 22:05:54 [debug] 5933#0: *12 free: 00000000008A7700, unused: 144
2015/04/10 22:05:55 [debug] 5933#0: *14 event timer del: 16: 1428696355041
2015/04/10 22:05:55 [debug] 5933#0: *14 http keepalive handler
2015/04/10 22:05:55 [debug] 5933#0: *14 close http connection: 16
2015/04/10 22:05:55 [debug] 5933#0: *14 SSL_shutdown: 1
2015/04/10 22:05:55 [debug] 5933#0: *14 reusable connection: 0
2015/04/10 22:05:55 [debug] 5933#0: *14 free: 0000000000000000
2015/04/10 22:05:55 [debug] 5933#0: *14 free: 0000000000000000
2015/04/10 22:05:55 [debug] 5933#0: *14 free: 00000000008933E0, unused: 0
2015/04/10 22:05:55 [debug] 5933#0: *14 free: 0000000000893BD0, unused: 32
2015/04/10 22:05:55 [debug] 5933#0: *14 free: 0000000000897A10, unused: 144
dmp42 commented 9 years ago

Hey @Trefex

From our irc chitchat, you mentioned that your proxy was responsible for the messup?

So, is this still an issue or can we close this?

Trefex commented 9 years ago

The error seems to have come from our proxy indeed.

dmg3 commented 8 years ago

@Trefex I'm experiencing a similar issue, how did you determine that it was your proxy at fault?

luxifr commented 8 years ago

I'd also be interested in the details of the resolution here

Trefex commented 8 years ago

@dmg3 I won't be able to be super helpful, but I did debug this by not using the proxy, which means changing the DNS, opening ports temporarily internally and so on.

Without proxy, everything was fine, with the proxy, all hell broke loose.

@luxifr I'm not sure what was done on the proxy side, because I'm not in control of that component :(