search

singularityhub / sregistry

server for storage and management of singularity images
https://singularityhub.github.io/sregistry
Mozilla Public License 2.0
103 stars 42 forks source link

Singularity PUSH returns 504 - however partially appears in Registry UI #284

Closed lmcdasm closed 4 years ago

lmcdasm commented 4 years ago

Describe the bug When performing an upload of a built SIF, the upload fails with a 504 HTTP return code. Looking at the Web UI Collections Page shows that the source container was uploaded.

Collection View: image

Looking at the filesystem on the server, (images directory) we see that the three containers are present at the filesystem level.

image

PROBLEM HERE However, when we browse into the Collection Test itself, we see there is only 1 Container listed. (the busybox one.

image

We can see the file being copied into the images/test directory (storage for test collection) over time here (this is coming from images/_upload directories (0-9) after transfer has finished.

image

We can observe that the sregistry_uwsgi_1 container logs the following - with the last line updated once the copy to the images/test directory is completed.

image

To Reproduce A) Create the image as outlined in the build step question below B) start up the same Sregistry setup (i am running on SRegistry 1.1.22 with the add/multipart-upload-404 PR) C) try to push the image as outlined.

Expected behavior A) the client should not be returned a 504 error if the upload is fine. B) the container should show up both in the Collections View (working) as well as in the container list

If applicable, add versions and screenshots to help explain your problem. INFORMATION REQUESTED ABOUT INSTALL METHOD

How do you install Singularity (more than just the version, how you went about it) on the client? Followed the steps marked here https://sylabs.io/guides/3.0/user-guide/installation.html - in a) Installed all necessary centos dependancies using yum - Section "Install Dependanices yum" b) Installed go:

How did you install Sregistry? REFERENCE for Steps/deps: https://singularityhub.github.io/sregistry/docs/install/settings a) installed all linux dependancies b) git clone https://github.com/singularityhub/sregistry c) followed steps to setup HTTPS (docker-compose.xml and config.py from https/ in git project base directory d) followed steps in install instructions to setpu LDAP and use this docker build command to do our build with LDAP enabled: docker build --build-arg ENABLE_LDAP=true -t quay.io/vanessa/sregistry .

How are you building / pulling your larger container? I am building on my machine and pushing to the Registry. Here is the command im using to build singularity build --fakeroot trax_centos:0.0.3 ./trax_centos.sing

Here is the build file i am using.. its just a simple centos wrapped in docker/

START Bootstrap: docker From: centos:latest

%runscript exec /venv/bin/python "$@"

%post export LC_ALL=C export PATH=/bin:/sbin:/usr/bin:/usr/sbin:$PATH

echo "S: Update and install pip + virtualenv packages"
yum update -y
yum install -y python3-devel python3-pip
pip3 install -U virtualenv

echo "S: Create a virtual environment"
virtualenv --system-site-packages -p python3 /venv

echo "S: Upgrade pip"
/venv/bin/pip install --upgrade pip

echo "S: Install tensorflow package"
/venv/bin/pip install --upgrade tensorflow

echo "S: Verify tensorflow"
/venv/bin/python -c "import tensorflow as tf;print(tf.reduce_sum(tf.random.normal([100, 100])))"
echo "S: Install trax"
/venv/bin/pip install --upgrade trax

%post echo "S: Install matplotlib" /venv/bin/pip install --upgrade matplotlib

%labels AUTHOR Daniel Smith END

How do you create your collection? a) login to the UI - we have LDAP enabled to our AD, works fine b) Click Collections, Click New Collections, Name Collection "test" and press Save

What are the the exact commands and result (you can likely use your previous comment here in a new issue) to push?

BUILD PROCESS: singularity build --fakeroot trax_centos:0.0.3 ./traxcentos.sing singularity remote add --no-login SingReg singreg.ourdomain.org singularity remote use SingReg singularity remote login SingReg (provide API Key and copy the $HOME/.sregistry info from ../tokens screen in the Web UI to my machine)_ singularity push -U trax_centos:0.0.3 library://daniel_smith/test/trax_centos:0.0.3

Usually this kind of detailed testing takes me a few hours and it's incredibly detailed work looking through several repos of sylabs, so please set your expectation accordingly. @lmcdasm - no worries, thank you so much for all the help... really great stuff you're doing and super appreciate the PR!_

vsoch commented 4 years ago

okay working to reproduce! I've successfully built your container:

INFO:    Adding labels
INFO:    Adding runscript
INFO:    Creating SIF file...
INFO:    Build complete: trac.sif
vanessa@vanessa-ThinkPad-T490s:/tmp$ ls -l trac.sif 
-rwxr-xr-x 1 vanessa vanessa 1290887168 Mar 24 11:12 trac.sif

It's ~1.3 GB. I've also built and started sregistry - I don't have LDAP so I cannot reproduce that (and if that turns out to be the issue, I'm not sure I could help).

Okay, I've also logged in to my remote, I've created a test collection, and now I'm testing push (I've replaced daniel_smith with vsoch as that's my username):

singularity push -U trac.sif library://vsoch/test/trax_centos:0.0.3

I'm curious, how are you referring to your container as trax_centos:0.0.3 - do you really have a filename without a sif or simg extension? Just want to check here because Singularity cannot support pushing a Docker URI, it's for a complete image. I don't see any issues with pushing:

$ singularity push -U trac.sif library://vsoch/test/trax_centos:0.0.3
WARNING: Skipping container verifying
1.2GiB / 1.2GiB [=========================================] 100 % 619.5 MiB/s 0s

And my web UI has the container too:

image

I am not able to reproduce, My thinking is that the issue here has something to do with your custom LDAP or nginx configuration. Could you share with me again your nginx.conf, and the docker-compose? What we would want to try next is to see if you are able to reproduce what I've done, which possibly means trying to build first without LDAP, and if the error still persists, without LDAP and any other customizations to the nginx.conf (are you using a proxy or something?)

lmcdasm commented 4 years ago

Hey there.

let me get this back to you tomorrow!

Doing some overnight maintenance tonight but will get back to you with nginx and ldap confs and i can try switching off LDAP and pushing and see sure.

more to come and thanks! Daniel

On Tue, 24 Mar 2020 at 14:02, Vanessasaurus notifications@github.com wrote:

okay working to reproduce! I've successfully built your container:

INFO: Adding labels INFO: Adding runscript INFO: Creating SIF file... INFO: Build complete: trac.sif vanessa@vanessa-ThinkPad-T490s:/tmp$ ls -l trac.sif -rwxr-xr-x 1 vanessa vanessa 1290887168 Mar 24 11:12 trac.sif

It's ~1.3 GB. I've also built and started sregistry - I don't have LDAP so I cannot reproduce that (and if that turns out to be the issue, I'm not sure I could help).

Okay, I've also logged in to my remote, I've created a test collection, and now I'm testing push (I've replaced daniel_smith with vsoch as that's my username):

singularity push -U trac.sif library://vsoch/test/trax_centos:0.0.3

I'm curious, how are you referring to your container as trax_centos:0.0.3

  • do you really have a filename without a sif or simg extension? Just want to check here because Singularity cannot support pushing a Docker URI, it's for a complete image. I don't see any issues with pushing:

$ singularity push -U trac.sif library://vsoch/test/trax_centos:0.0.3 WARNING: Skipping container verifying 1.2GiB / 1.2GiB [=========================================] 100 % 619.5 MiB/s 0s

And my web UI has the container too:

[image: image] https://user-images.githubusercontent.com/814322/77460581-e586f280-6dc6-11ea-9783-d52aaba9412e.png

I am not able to reproduce, My thinking is that the issue here has something to do with your custom LDAP or nginx configuration. Could you share with me again your nginx.conf, and the docker-compose? What we would want to try next is to see if you are able to reproduce what I've done, which possibly means trying to build first without LDAP, and if the error still persists, without LDAP and any other customizations to the nginx.conf (are you using a proxy or something?)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/singularityhub/sregistry/issues/284#issuecomment-603413504, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACHMXRFG7YY653WGZSN23QLRJDYSXANCNFSM4LD5XDEQ .

lmcdasm commented 4 years ago

Hello. to answer this question: "I'm curious, how are you referring to your container as trax_centos:0.0.3 - do you really have a filename without a sif or simg extension? Just want to check here because Singularity cannot support pushing a Docker URI, it's for a complete image. I don't see any issues with pushing:"

i am doing as you do. i have a test.SIF file (that is my image on the filesystem - 1.3GB, etc) and im naming it trax_centos:0.0.3 in the library call as you have shown here: "singularity push -U trac.sif library://vsoch/test/trax_centos:0.0.3" ( of course, our other thing about pushing to collections nothing withstanding) more to come. D

On Wed, 25 Mar 2020 at 01:31, Daniel Smith lmcdasm@gmail.com wrote:

Hey there.

let me get this back to you tomorrow!

Doing some overnight maintenance tonight but will get back to you with nginx and ldap confs and i can try switching off LDAP and pushing and see sure.

more to come and thanks! Daniel

On Tue, 24 Mar 2020 at 14:02, Vanessasaurus notifications@github.com wrote:

okay working to reproduce! I've successfully built your container:

INFO: Adding labels INFO: Adding runscript INFO: Creating SIF file... INFO: Build complete: trac.sif vanessa@vanessa-ThinkPad-T490s:/tmp$ ls -l trac.sif -rwxr-xr-x 1 vanessa vanessa 1290887168 Mar 24 11:12 trac.sif

It's ~1.3 GB. I've also built and started sregistry - I don't have LDAP so I cannot reproduce that (and if that turns out to be the issue, I'm not sure I could help).

Okay, I've also logged in to my remote, I've created a test collection, and now I'm testing push (I've replaced daniel_smith with vsoch as that's my username):

singularity push -U trac.sif library://vsoch/test/trax_centos:0.0.3

I'm curious, how are you referring to your container as trax_centos:0.0.3

  • do you really have a filename without a sif or simg extension? Just want to check here because Singularity cannot support pushing a Docker URI, it's for a complete image. I don't see any issues with pushing:

$ singularity push -U trac.sif library://vsoch/test/trax_centos:0.0.3 WARNING: Skipping container verifying 1.2GiB / 1.2GiB [=========================================] 100 % 619.5 MiB/s 0s

And my web UI has the container too:

[image: image] https://user-images.githubusercontent.com/814322/77460581-e586f280-6dc6-11ea-9783-d52aaba9412e.png

I am not able to reproduce, My thinking is that the issue here has something to do with your custom LDAP or nginx configuration. Could you share with me again your nginx.conf, and the docker-compose? What we would want to try next is to see if you are able to reproduce what I've done, which possibly means trying to build first without LDAP, and if the error still persists, without LDAP and any other customizations to the nginx.conf (are you using a proxy or something?)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/singularityhub/sregistry/issues/284#issuecomment-603413504, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACHMXRFG7YY653WGZSN23QLRJDYSXANCNFSM4LD5XDEQ .

vsoch commented 4 years ago

@lmcdasm we just merged the Minio backend which should address most of the issues with push/pull - would you care to test?

vsoch commented 4 years ago

Closing issue, no response.