"FATA[0005] HTTP code 500 while uploading metadata: invalid character '<' looking for beginning of value" ; Cannot push

[jalev]

We've recently gotten this behaviour as of yesterday when trying to push images. It was caused when one of the images failed to upload, possibly due to a time-out. Subsequent pushing of that image failed with this error.

I'm using Docker 1.4.1

My setup is: Registry behind an Nginx reverse proxy, using Redis as a LRU cache.

From the registry side when trying to push the image which caused this problem:

2015-02-25 09:32:23,850 ERROR: Exception on /v1/images/0f01bd768760ce4ab5847e1cc5a10fe38d2af8e13f23e38a91ec2be46fea11e8/json [PUT]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 280, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 37, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/images.py", line 375, in put_image_json
    layers.generate_ancestry(image_id, parent_id)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/layers.py", line 56, in generate_ancestry
    data = store.get_json(store.image_ancestry_path(parent_id))
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/core/driver.py", line 185, in get_json
    return json.loads(self.get_unicode(path))
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/core/driver.py", line 191, in get_unicode
    return self.get_bytes(path).decode('utf8')
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/core/driver.py", line 197, in get_bytes
    return self.get_content(path)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/core/lru.py", line 94, in wrapper
    content = f(*args)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/drivers/file.py", line 55, in get_content
    raise exceptions.FileNotFoundError('%s is not there' % path)
FileNotFoundError: /registry/images/5ba9dab47459d81c0037ca3836a368a4f8ce5050505ce89720e1fb8839ea048a/ancestry is not there
25/Feb/2015:09:32:23 +0000 ERROR: Exception on /v1/images/0f01bd768760ce4ab5847e1cc5a10fe38d2af8e13f23e38a91ec2be46fea11e8/json [PUT]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 280, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 37, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/images.py", line 375, in put_image_json
    layers.generate_ancestry(image_id, parent_id)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/layers.py", line 56, in generate_ancestry
    data = store.get_json(store.image_ancestry_path(parent_id))
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/core/driver.py", line 185, in get_json
    return json.loads(self.get_unicode(path))
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/core/driver.py", line 191, in get_unicode
    return self.get_bytes(path).decode('utf8')
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/core/driver.py", line 197, in get_bytes
    return self.get_content(path)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/core/lru.py", line 94, in wrapper
    content = f(*args)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/drivers/file.py", line 55, in get_content
    raise exceptions.FileNotFoundError('%s is not there' % path)
FileNotFoundError: /registry/images/5ba9dab47459d81c0037ca3836a368a4f8ce5050505ce89720e1fb8839ea048a/ancestry is not there

When trying to push new images to the registry:

2015-02-25 09:43:41,406 ERROR: Exception on /v1/images/fa4fd76b09ce9b87bfdc96515f9a5dd5121c01cc996cf5379050d8e13d4a864b/json [PUT]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 280, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 37, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/images.py", line 375, in put_image_json
    layers.generate_ancestry(image_id, parent_id)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/layers.py", line 56, in generate_ancestry
    data = store.get_json(store.image_ancestry_path(parent_id))
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/core/driver.py", line 185, in get_json
    return json.loads(self.get_unicode(path))
  File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
25/Feb/2015:09:43:41 +0000 ERROR: Exception on /v1/images/fa4fd76b09ce9b87bfdc96515f9a5dd5121c01cc996cf5379050d8e13d4a864b/json [PUT]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 280, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 37, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/images.py", line 375, in put_image_json
    layers.generate_ancestry(image_id, parent_id)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/layers.py", line 56, in generate_ancestry
    data = store.get_json(store.image_ancestry_path(parent_id))
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/core/driver.py", line 185, in get_json
    return json.loads(self.get_unicode(path))
  File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Docker logs when pushing old image:

Feb 25 10:02:07 core-01-01 dockerd[647]: HTTP code 500 while uploading metadata: invalid character '<' looking for beginning of value

Docker logs when pushing new image:

Feb 25 09:48:31 core-01-01 dockerd[647]: time="2015-02-25T09:48:31Z" level="info" msg="POST /v1.16/images/registry.scee.scedev.net/scee/locust/push?tag="
Feb 25 09:48:31 core-01-01 dockerd[647]: time="2015-02-25T09:48:31Z" level="info" msg="+job push(registry.scee.scedev.net/scee/locust)"
Feb 25 09:48:36 core-01-01 dockerd[647]: Error: Status 502 trying to push repository scee/locust: <html>
Feb 25 09:48:36 core-01-01 dockerd[647]: <head><title>502 Bad Gateway</title></head>
Feb 25 09:48:36 core-01-01 dockerd[647]: <body bgcolor="white">
Feb 25 09:48:36 core-01-01 dockerd[647]: <center><h1>502 Bad Gateway</h1></center>
Feb 25 09:48:36 core-01-01 dockerd[647]: <hr><center>nginx/1.4.6 (Ubuntu)</center>
Feb 25 09:48:36 core-01-01 dockerd[647]: </body>
Feb 25 09:48:36 core-01-01 dockerd[647]: </html>

Docker reports the same error for two different behaviours.

[dmp42]

Unfortunately, a failed push usually leaves the image in a "broken" state that the registry can not handle. The only (not so) good recommendation is to manually delete the image from the storage and push it again...

[sgandon]

I have the same issue

FATA[0009] HTTP code 500 while uploading metadata: invalid character '<' looking for beginning of value

when pushing an nginx image to our private registry.

[dmp42]

@sgandon this error is non specific. Please at least provide docker version, docker info, and both docker registry and docker daemon (debug) logs. Also, if this is the same issue, have you tried removing the "broken" image like suggested in my previous comment?

Thanks.

[sgandon]

We have tried to remove the faulty images without success. Our image comes from a simple Dockerfile inheriting from nginx:latest. here is the client log (version 1.5.0)

docker push our-registry:5000/test-dataprep-webapp-nginx:0.0
The push refers to a repository [our-registry:5000/test-dataprep-webapp-nginx] (len: 1)
Sending image list
Pushing repository our-registry:5000/test-dataprep-webapp-nginx (1 tags)
Image 30d39e59ffe2 already pushed, skipping
511136ea3c5a: Image successfully pushed
c90d655b99b2: Pushing
FATA[0009] HTTP code 500 while uploading metadata: invalid character '<' looking for beginning of value

here is the server log (version is 1.3.0)

2015-03-18 14:07:34,292 ERROR: Exception on /v1/images/c90d655b99b2ec5b7e94d38c87f92dce015c17a313caeaae0e980d9b9bed8444/json [PUT]
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 250, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/images.py", line 421, in put_image_json
    layers.generate_ancestry(image_id, parent_id)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/layers.py", line 55, in generate_ancestry
    data = store.get_json(store.image_ancestry_path(parent_id))
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/core/driver.py", line 184, in get_json
    return json.loads(self.get_unicode(path))
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/core/driver.py", line 190, in get_unicode
    return self.get_bytes(path).decode('utf8')
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/core/driver.py", line 196, in get_bytes
    return self.get_content(path)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/drivers/file.py", line 55, in get_content
    raise exceptions.FileNotFoundError('%s is not there' % path)
FileNotFoundError: /registry/images/30d39e59ffe287f29a41a3f8bd70734afc8728329e3289945cbdc5bbf07cd980/ancestry is not there

[hgomez]

Same problem here using docker-registry 0.9.1 and Docker version 1.5.0, build a8a31ef on Mint 17. I'm using Apache HTTPd 2.4 in front.

Image I tried to push is busybox

Does it means ancestry layout should be fetched via docker-registry first ?

[rohitgupta-90]

I faced the same issue with my private registry. ... Image f6808a3e4d9e already pushed, skipping ca49356dbc38: Pushing FATA[0002] HTTP code 500 while uploading metadata: invalid character '<' looking for beginning of value

I found that my private registry was out of space, fixed it and it seems to work fine after that.

FWIW this is the docker version I am using: Server version: 1.3.2 Server API version: 1.15

[Jamlee]

@rocketraman I occur a same problem. Do you solve the problem?

jamlee@jamlee-pc:~/JAM/lab/docker-reg/private-registry$ docker push docker.dev:5000/node The push refers to a repository [docker.dev:5000/node](len: 1) Sending image list Pushing repository docker.dev:5000/node (1 tags) Image 938e3817ad84 already pushed, skipping Image b7820d1ee4ee already pushed, skipping Image 511136ea3c5a already pushed, skipping Image 65688f7c61c4 already pushed, skipping Image d338bb63f151 already pushed, skipping 2b7102d6f8b7: Pushing FATA[0006] HTTP code 500 while uploading metadata: invalid character '<' looking for beginning of value

my docker version:

jamlee@jamlee-pc:~/JAM/lab/docker-reg/private-registry$ docker version Client version: 1.5.0 Client API version: 1.17 Go version (client): go1.4.1 Git commit (client): a8a31ef OS/Arch (client): linux/amd64 Server version: 1.5.0 Server API version: 1.17 Go version (server): go1.4.1 Git commit (server): a8a31ef

[jankoprowski]

Behaves the same when docker repository run out of space.

[zapient]

We are seeing the same issue but we also see the following error:

Received HTTP code 404 while uploading layer: "{\"error\": \"Image not found\"}"

The only work around was to keep resubmitting the push

function docker-push() { docker push $1; while [ $? -ne 0 ]; do docker push $1; done }

Doesnt appear to have anything to do with disk-space for us

Client version: 1.5.0 Client API version: 1.17 Go version (client): go1.3.3 Git commit (client): a8a31ef-dirty OS/Arch (client): linux/amd64 Server version: 1.5.0 Server API version: 1.17 Go version (server): go1.3.3 Git commit (server): a8a31ef-dirty

[dmp42]

I would really encourage everybody to start looking into registry:2 (available as an official image: docker pull registry:2 documentation is here: https://docs.docker.com/registry/ and github repository is here: https://github.com/docker/distribution

The python registry here is not being worked on actively anymore.

[Jamlee]

@dmp42 is it really stable now? how long you use it ?

[dmp42]

@Jamlee it has been released officially last month, and is being used in production by a number of people, including Docker Hub.

[k16wire]

I got the same issues. I can resolve to remove broken image file on the registry server. How about remove 'd338bb63f151' file on the repository and try again!