Fatal error while pulling images with singularity-ce 3.9.x

[brisk022]

When I try to pull an image from sregistry with singularity 3.9.x (tested with 3.9.0, 3.9.6 and 3.9.7), I get the following error.

singularityce Ф❯ singularity --version
singularity-ce version 3.9.7
singularityce Ф❯ singularity pull --disable-cache library://rbrisk/apps/test:2.0
INFO:    Downloading library image
FATAL:   While pulling library image: error fetching image: unable to download image: error downloading image: unexpected HTTP status 302: <nil>

Everything works fine with the older versions of singularity <3.9.0. For example, here is the expected response with 3.7.1.

singularity Ф❯ singularity --version
singularity version 3.7.1
singularity Ф❯ singularity pull --disable-cache library://rbrisk/apps/test:2.0
INFO:    Downloading library image
58.0MiB / 58.0MiB [==============================================================================] 100 % 57.5 MiB/s 0s
WARNING: integrity: signature not found for object group 1
WARNING: Skipping container verification

Here is what I see in the nginx logs for those two requests.

ESC[36mnginx_1      |ESC[0m xxx.xx.xx.xx - - [07/Apr/2022:10:19:38 +0000] "GET /assets/config/config.prod.json HTTP/1.1" 200 517 "-" "Singularity-Ce/3.9.7 (Linux amd64) Go/1.17.8" "-"
ESC[36mnginx_1      |ESC[0m xxx.xx.xx.xx - - [07/Apr/2022:10:19:38 +0000] "GET /v1/images/rbrisk/apps/test:2.0?arch=amd64 HTTP/1.1" 200 695 "-" "Singularity-Ce/3.9.7 (Linux amd64) Go/1.17.8" "-"
ESC[36mnginx_1      |ESC[0m xxx.xx.xx.xx - - [07/Apr/2022:10:19:38 +0000] "GET /version HTTP/1.1" 200 58 "-" "Singularity-Ce/3.9.7 (Linux amd64) Go/1.17.8" "-"
ESC[36mnginx_1      |ESC[0m xxx.xx.xx.xx - - [07/Apr/2022:10:19:38 +0000] "GET /v1/imagefile/rbrisk/apps/test:2.0?arch=amd64 HTTP/1.1" 302 0 "-" "Singularity-Ce/3.9.7 (Linux amd64) Go/1.17.8" "-"
ESC[36mnginx_1      |ESC[0m xxx.xx.xx.xx - - [07/Apr/2022:10:19:50 +0000] "GET /assets/config/config.prod.json HTTP/1.1" 200 517 "-" "Singularity-Ce/3.9.7 (Linux amd64) Go/1.17.8" "-"
ESC[36mnginx_1      |ESC[0m xxx.xx.xx.xx - - [07/Apr/2022:10:25:04 +0000] "GET /v1/images/rbrisk/apps/test:2.0?arch=amd64 HTTP/1.1" 200 695 "-" "Singularity/3.7.1 (Linux amd64) Go/1.15.8" "-"
ESC[36mnginx_1      |ESC[0m xxx.xx.xx.xx - - [07/Apr/2022:10:25:04 +0000] "GET /version HTTP/1.1" 200 58 "-" "Singularity/3.7.1 (Linux amd64) Go/1.15.8" "-"
ESC[36mnginx_1      |ESC[0m xxx.xx.xx.xx - - [07/Apr/2022:10:25:04 +0000] "GET /v1/imagefile/rbrisk/apps/test:2.0?arch=amd64 HTTP/1.1" 302 0 "-" "Singularity/3.7.1 (Linux amd64) Go/1.15.8" "-"

In this particular test, I used 0c88404 but I observed the same behaviour with 9abb177

Is this the right place for the bug or should I submit it to the Singularity-CE developers?

[vsoch]

This is the right place! It would be helpful to ping them and see if they know what changes happened between those versions. It seems like they no longer allow 302, which used to be part of their pull process (to redirect so some signed AWS url).

[brisk022]

As far as I can tell, the problem appeared after SingularityCE switched to concurrent image download in 3.9.0. They have an enhancement request that might address that: sylabs/scs-library-client#130

I created a bug report in their main repo to let them know about the issue: sylabs/singularity#698

I could try a pull request but I do not feel very confident about my golang skills :(

[vsoch]

Thank you for doing that! I am watching the issue you posted, and if there is something we can do to fix it I can also offer to help.

And if you want to take a shot, I think probably I and the other sylabs singularity maintainers could help you :)

[brisk022]

I think that the fix has made it into singularity 3.9.9. I have tested it and it seems to be working properly. So, I am going to close this issue now, if you do not mind.