Push just stops - api_error: Image not found

vitorarins commented 9 years ago

I am trying to tag and push a repository by doing: docker tag mongodb-docker my.registry.com/mongodb-docker docker push my.registry.com/mongodb-docker

I have the registry begind nginx auth, reverse proxying localhost:5000 to my.registry.com. But it seems like the registry is trying to PUT tags before the image is uploaded

Here is the relevant log:

172.17.42.1 - - [08/Dec/2014:13:06:31 +0000] "GET /v1/_ping HTTP/1.0" 200 1420 "-" "Go 1.1 package http"
172.17.42.1 - - [08/Dec/2014:13:06:31 +0000] "GET /v1/_ping HTTP/1.0" 200 1420 "-" "Go 1.1 package http"
08/Dec/2014:13:06:31 +0000 DEBUG: args = {'namespace': 'library', 'repository': u'mongodb-docker'}
172.17.42.1 - - [08/Dec/2014:13:06:31 +0000] "PUT /v1/repositories/mongodb-docker/ HTTP/1.0" 200 2 "-" "docker/1.3.1 go/go1.3.3 git-commit/4e9bbfa kernel/3.16.4-tinycore64 os/linux arch/amd64"
08/Dec/2014:13:06:31 +0000 DEBUG: args = {'image_id': u'511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158'}
08/Dec/2014:13:06:31 +0000 DEBUG: api_error: Image not found
172.17.42.1 - - [08/Dec/2014:13:06:31 +0000] "GET /v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/json HTTP/1.0" 404 28 "-" "docker/1.3.1 go/go1.3.3 git-commit/4e9bbfa kernel/3.16.4-tinycore64 os/linux arch/amd64"
08/Dec/2014:13:06:31 +0000 DEBUG: args = {'image_id': u'511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158'}
172.17.42.1 - - [08/Dec/2014:13:06:31 +0000] "PUT /v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/json HTTP/1.0" 200 4 "-" "docker/1.3.1 go/go1.3.3 git-commit/4e9bbfa kernel/3.16.4-tinycore64 os/linux arch/amd64"

dmp42 commented 9 years ago

Can you:

try pushing and pulling without nginx
copy the information listed here: (https://github.com/docker/docker-registry/blob/master/DEBUGGING.md)
copy your docker daemon logs (in debug mode -D)
copy your registry logs in debug (DEBUG=true)

vitorarins commented 9 years ago

docker push gives:

docker push 192.168.110.11:5000/spring-boot
2014/12/08 19:12:32 Error: Invalid registry endpoint https://192.168.110.11:5000/v1/: Get https://192.168.110.11:5000/v1/_ping: EOF. If this private registry supports only HTTP or HTTPS with an unknown CA certificate, please add `--insecure-registry 192.168.110.11:5000` to the daemon's arguments. In the case of HTTPS, if you have access to the registry's CA certificate, no need for the flag; simply place the CA certificate at /etc/docker/certs.d/192.168.110.11:5000/ca.crt

docker info:

[debug] server.go:1181 Calling GET /info
[info] GET /v1.15/info
[ce52d9a4] +job info()
[ce52d9a4] +job subscribers_count()
[ce52d9a4] -job subscribers_count() = OK (0)
[ce52d9a4] -job info() = OK (0)
Containers: 5
Images: 94
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 104
Execution Driver: native-0.2
Kernel Version: 3.2.0-4-amd64
Operating System: Debian GNU/Linux 7 (wheezy)
Debug mode (server): true
Debug mode (client): false
Fds: 16
Goroutines: 15
EventsListeners: 0
Init Path: /usr/bin/docker
WARNING: No memory limit support
WARNING: No swap limit support

docker version:

Client version: 1.3.2
Client API version: 1.15
Go version (client): go1.3.3
Git commit (client): 39fa2fa
OS/Arch (client): linux/amd64
[debug] server.go:1181 Calling GET /version
[info] GET /v1.15/version
[ce52d9a4] +job version()
[ce52d9a4] -job version() = OK (0)
Server version: 1.3.2
Server API version: 1.15
Go version (server): go1.3.3
Git commit (server): 39fa2fa

docker logs:

[debug] server.go:1181 Calling GET /containers/{name:.*}/json
[info] GET /v1.15/containers/registry/json
[ce52d9a4] +job container_inspect(registry)
[ce52d9a4] -job container_inspect(registry) = OK (0)
[debug] server.go:1181 Calling GET /containers/{name:.*}/logs
[info] GET /v1.15/containers/registry/logs?stderr=1&stdout=1&tail=all
[ce52d9a4] +job container_inspect(registry)
[ce52d9a4] -job container_inspect(registry) = OK (0)
[ce52d9a4] +job logs(registry)
2014-12-08 21:10:42 [1] [INFO] Starting gunicorn 19.1.0
2014-12-08 21:10:42 [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
2014-12-08 21:10:42 [1] [INFO] Using worker: gevent
2014-12-08 21:10:42 [14] [INFO] Booting worker with pid: 14
[ce52d9a4] -job logs(registry) = OK (0)
2014-12-08 21:10:43 [15] [INFO] Booting worker with pid: 15
2014-12-08 21:10:43 [16] [INFO] Booting worker with pid: 16
2014-12-08 21:10:43 [17] [INFO] Booting worker with pid: 17
2014-12-08 21:10:43 [1] [INFO] 4 workers
08/Dec/2014:21:10:43 +0000 WARNING: Cache storage disabled!
08/Dec/2014:21:10:43 +0000 WARNING: LRU cache disabled!
08/Dec/2014:21:10:43 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
08/Dec/2014:21:10:43 +0000 WARNING: Cache storage disabled!
08/Dec/2014:21:10:43 +0000 WARNING: LRU cache disabled!
08/Dec/2014:21:10:43 +0000 WARNING: Cache storage disabled!
08/Dec/2014:21:10:43 +0000 WARNING: LRU cache disabled!
08/Dec/2014:21:10:44 +0000 WARNING: Cache storage disabled!
08/Dec/2014:21:10:44 +0000 WARNING: LRU cache disabled!
08/Dec/2014:21:10:44 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
08/Dec/2014:21:10:44 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
08/Dec/2014:21:10:44 +0000 DEBUG: Will return docker-registry.drivers.file.Storage

docker registry with nginx:

docker login https://registry.com.br
Username (vitor.arins):     
Login Succeeded

docker tag spring-boot registry.com.br/spring-boot

docker push registry.com.br/spring-boot
The push refers to a repository [registry.com.br/spring-boot] (len: 1)
Sending image list
Pushing repository registry.com.br/spring-boot (1 tags)
511136ea3c5a: Pushing [==================================================>] 1.536 kB/1.536 kB
2014/12/08 19:17:54

docker logs with nginx:

[debug] server.go:1181 Calling GET /containers/{name:.*}/json
[info] GET /v1.15/containers/registry/json
[ce52d9a4] +job container_inspect(registry)
[ce52d9a4] -job container_inspect(registry) = OK (0)
[debug] server.go:1181 Calling GET /containers/{name:.*}/logs
[info] GET /v1.15/containers/registry/logs?stderr=1&stdout=1&tail=all
[ce52d9a4] +job container_inspect(registry)
[ce52d9a4] -job container_inspect(registry) = OK (0)
[ce52d9a4] +job logs(registry)
2014-12-08 21:10:42 [1] [INFO] Starting gunicorn 19.1.0
2014-12-08 21:10:42 [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
2014-12-08 21:10:42 [1] [INFO] Using worker: gevent
2014-12-08 21:10:42 [14] [INFO] Booting worker with pid: 14
2014-12-08 21:10:43 [15] [INFO] Booting worker with pid: 15
2014-12-08 21:10:43 [16] [INFO] Booting worker with pid: 16
2014-12-08 21:10:43 [17] [INFO] Booting worker with pid: 17
2014-12-08 21:10:43 [1] [INFO] 4 workers
08/Dec/2014:21:10:43 +0000 WARNING: Cache storage disabled!
08/Dec/2014:21:10:43 +0000 WARNING: LRU cache disabled!
08/Dec/2014:21:10:43 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
08/Dec/2014:21:10:43 +0000 WARNING: Cache storage disabled!
08/Dec/2014:21:10:43 +0000 WARNING: LRU cache disabled!
08/Dec/2014:21:10:43 +0000 WARNING: Cache storage disabled!
08/Dec/2014:21:10:43 +0000 WARNING: LRU cache disabled!
08/Dec/2014:21:10:44 +0000 WARNING: Cache storage disabled!
08/Dec/2014:21:10:44 +0000 WARNING: LRU cache disabled!
08/Dec/2014:21:10:44 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
08/Dec/2014:21:10:44 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
08/Dec/2014:21:10:44 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
172.17.42.1 - - [08/Dec/2014:21:16:44 +0000] "GET /v1/_ping HTTP/1.0" 200 1420 "-" "Go 1.1 package http"
172.17.42.1 - - [08/Dec/2014:21:16:44 +0000] "GET /v1/_ping HTTP/1.0" 200 1420 "-" "Go 1.1 package http"
172.17.42.1 - - [08/Dec/2014:21:16:44 +0000] "GET /v1/users/ HTTP/1.0" 200 4 "-" "docker/1.3.1 go/go1.3.3 git-commit/4e9bbfa kernel/3.16.4-tinycore64 os/linux arch/amd64"
172.17.42.1 - - [08/Dec/2014:21:17:21 +0000] "GET /v1/_ping HTTP/1.0" 200 1420 "-" "Go 1.1 package http"
172.17.42.1 - - [08/Dec/2014:21:17:21 +0000] "GET /v1/_ping HTTP/1.0" 200 1420 "-" "Go 1.1 package http"
172.17.42.1 - - [08/Dec/2014:21:17:37 +0000] "GET /v1/_ping HTTP/1.0" 200 1420 "-" "Go 1.1 package http"
172.17.42.1 - - [08/Dec/2014:21:17:37 +0000] "GET /v1/_ping HTTP/1.0" 200 1420 "-" "Go 1.1 package http"
08/Dec/2014:21:17:37 +0000 DEBUG: args = {'namespace': 'library', 'repository': u'spring-boot'}
172.17.42.1 - - [08/Dec/2014:21:17:37 +0000] "PUT /v1/repositories/spring-boot/ HTTP/1.0" 200 2 "-" "docker/1.3.1 go/go1.3.3 git-commit/4e9bbfa kernel/3.16.4-tinycore64 os/linux arch/amd64"
08/Dec/2014:21:17:37 +0000 DEBUG: args = {'image_id': u'511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158'}
08/Dec/2014:21:17:37 +0000 DEBUG: api_error: Image is being uploaded, retry later
172.17.42.1 - - [08/Dec/2014:21:17:37 +0000] "GET /v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/json HTTP/1.0" 400 49 "-" "docker/1.3.1 go/go1.3.3 git-commit/4e9bbfa kernel/3.16.4-tinycore64 os/linux arch/amd64"
08/Dec/2014:21:17:37 +0000 DEBUG: args = {'image_id': u'511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158'}
[ce52d9a4] -job logs(registry) = OK (0)
172.17.42.1 - - [08/Dec/2014:21:17:37 +0000] "PUT /v1/images/511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158/json HTTP/1.0" 200 4 "-" "docker/1.3.1 go/go1.3.3 git-commit/4e9bbfa kernel/3.16.4-tinycore64 os/linux arch/amd64"

Thank you for the quick response!

dmp42 commented 9 years ago

Ok... my 2 cents:

something went wrong during the upload, and it failed (silently?) (this is why it seems from the registry logs that the tags are being put "before" the image)
the image is now in a broken state (you now get a 400)
the reason it broke is possibly nginx configuration (maybe chunk size)

To start over:

manually delete the image from the storage
see if you can reproduce
if you can reproduce, please copy over your nginx configuration and version

dmp42 commented 9 years ago

Hi, any update on this?

docker-archive / docker-registry

Push just stops - api_error: Image not found #828