search

teamhephy / registry

MIT License
0 stars 5 forks source link

`deis pull` on trivially large images fails #4

Open kingdonb opened 5 years ago

kingdonb commented 5 years ago

User reports from slack came in that with the default configuration, deis pull on some trivially large images like wordpress:latest (Docker Hub measures this one at 145MB) will fail.

The relevant errors appear to be showing up in the registry logs. It seems like it's possible that minio is rejecting the images because a layer is too big, but that seems like a pretty low ceiling.

The same procedure applied with the example-dockerfile-http image (4MB according to Docker Hub) succeeds. I have not tried to reproduce this issue yet on a cluster that is not running minio and has Object Storage properly configured, but it seems likely from the errors presenting in registry logs that this change will resolve it.

kingdonb commented 5 years ago

Here is a sample of the errors from registry, not sure I have grabbed the most important relevant bits (the real source of the error is probably buried in minio info or debug logs which are not configured to be displayed by default, minio logs are eerily quiet amidst all of this trouble not reporting anything on success or failure basically all of the time):

time="2018-11-17T15:42:50.598234283Z" level=error msg="response completed with error" err.code=unknown err.detail="s3aws: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: 6CR72G6EM04Z6Z1I" err.message="unknown error" go.version=go1.7.3 http.request.host="127.0.0.1:5555" http.request.id=9a30962f-54a2-45b7-be07-2e027a4a1589 http.request.method=PUT http.request.remoteaddr=10.244.31.1 http.request.uri="/v2/ip/blobs/uploads/5c9fa78d-94a1-4b0e-b8d2-fa75026bd7a9?_state=JJgQ6wfTXdUz4N2ehkO3Xsh4WG8X-GzPoB18g3TD6LB7Ik5hbWUiOiJpcCIsIlVVSUQiOiI1YzlmYTc4ZC05NGExLTRiMGUtYjhkMi1mYTc1MDI2YmQ3YTkiLCJPZmZzZXQiOjIzMjMyOTg1MSwiU3RhcnRlZEF0IjoiMjAxOC0xMS0xN1QxNTo0MjoxM1oifQ%3D%3D&digest=sha256%3A7e49a239bcdc4e9d84b037fe59751f0595bc8d49cf09227cd3a2fbee23b3e1d5" http.request.useragent="docker/18.09.0 go/go1.10.4 git-commit/4d60db4 kernel/4.9.0-8-amd64 os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)" http.response.contenttype="application/json; charset=utf-8" http.response.duration=617.493801ms http.response.status=500 http.response.written=117 instance.id=a32afcd9-1c88-43cd-a796-cb7245c80f4f service=registry vars.name=ip vars.uuid=5c9fa78d-94a1-4b0e-b8d2-fa75026bd7a9 version=v2.6.0
Cryptophobia commented 5 years ago

s3aws: NoSuchKey ? Where is this key coming from. Any idea where it could be defined?

till commented 5 years ago

There must be something else before.

kingdonb commented 5 years ago

I think the message that comes before, is when the upload fails. There is no such key because it has not been uploaded successfully...

Unfortunately I'm not in a position to repro it again at this time, but I believe you should be able to see it erroring on a cluster that still works with minio memory storage backend.

Cryptophobia commented 5 years ago

Ok, I will try to reproduce at some point in the future when I have time to tackle this problem.