push/pull with custom registries in subpath

h3kker commented 5 years ago

Version of Singularity:

3.x

Expected behavior

We have a singularity registry that is reverse proxied in a subpath, like domain.test/sregistry/...

singularity pull shub://domain.test/sregistry/pipeline/minimap2
Progress |===================================| 100.0%
Done. Container is at: /users/heinz.ekker/pipeline-minimap2-latest.simg

In 2.x I could pull images from this registry, but not with 3.x

Actual behavior

WARNING: Authentication token file not found : Only pulls of public images will succeed
FATAL:   Failed to get manifest from Shub: No response received from singularity hub

Steps to reproduce behavior

Difficult to reproduce, you'd have to set up a custom sregistry with a subpath. I'm not a go programmer, but I think the problem is in pkg/client/shub/util.go. If I read the code correctly, shubParseReference splits the URI at slashes and then concats everything except the last to parts again for the hostname of the custom registry. In this case this yields domain.testsregistry, which does not resolve.

WestleyK commented 5 years ago

@h3kker,

As a temporary solution, you could try pulling it with http://, or https://:

$ singularity --debug pull http(s)?://domain.test/sregistry/pipeline/minimap2

If that does not work, can you post the output?

EDIT: This actually may not work... but its worth a try :wink:

Thanks!

h3kker commented 5 years ago

Hi!

As expected in the edit :)

singularity --debug pull https://domain.test/sregistry/pipeline/minimap2:latest
WARNING [U=56203,P=49348]  sylabsToken()                 Authentication token file not found : Only pulls of public images will succeed
DEBUG   [U=56203,P=49348]  DownloadImage()               Pulling from URL: https://domain.test/sregistry/pipeline/minimap2:latest
FATAL   [U=56203,P=49348]  PullNetImage()                The requested image was not found in the library

The https pull only works if I use the image link directly:

singularity pull https://domain.test/sregistry/containers/34/download/c9c7248a-1637-4e02-bb5b-e23f2dba9eea

It seems to me that the shub:// link first tries to get this link via the API (GET /api/container/pipeline/minimap2:latest), at least that's what singularity 2.x does. The https pull just tries to directly GET /pipeline/minimap2, which is a 404.

BTW due to $REASONS I'm working with singularity 3.1, because that's what we've got on our cluster. I can install a more current version locally and try with that, would also give me something to push for an upgrade.

WestleyK commented 5 years ago

Could you try this: https://github.com/sylabs/singularity/pull/4053? Just build by branch locally and run:

$ ./singularity -d pull shub://domain.test/sregistry/pipeline/minimap2

I dont have a custom registry right now, but I did a couple tests and this should fix the missing / in the URL (domain.test/sregistry/api/container/).

h3kker commented 5 years ago

Hi!

Thank you, that works!

singularity --version
singularity version 3.0.1-3382.gd257037
singularity pull shub://domain.test/sregistry/pipeline/minimap2:latest
INFO:    Downloading shub image
 46.78 MiB / ? [-----------------------------=----------------------] 490536.96% 18.79 MiB/s00m02s
FATAL:   While pulling shub image: image received is not the right size. supposed to be: -1  actually: 49299487

It seems that the registry does not send a correct body length header, but this surely is our local problem and anyways unrelated to this issue.

WestleyK commented 5 years ago

Glad its semi working!

Yeah, it looks like you custom registry is not handling the ContentLength correctly. But, I have a idea to handle this better in the code so pull would still work...

WestleyK commented 5 years ago

Okay, can you update my branch WestleyK:custom_shub_registry_fix (may need to re-clone it), re-compile and re-try the pull command again?

It should now print a warning about unknown image length, but the pull should still succeed.

h3kker commented 5 years ago

Great, that works!

It's a bit weird, since I can see the correct Content-Length header when I use curl for the same GET request.

I'm using https://github.com/singularityhub/sregistry, maybe it has something to do with nginx and the sendfile backend that somehow delays the content-length header? I don't know. I'll play around and report it there, if I can find the time.

Meanwhile thanks for the fast fixes!

apptainer / singularity