search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
721 stars 38 forks source link

[REVIEW]: Singularity Registry: Open Source Registry for Singularity Images #426

Closed whedon closed 7 years ago

whedon commented 7 years ago

Submitting author: @vsoch (Vanessa Sochat) Repository: https://www.github.com/singularityhub/sregistry Version: v1.0.0 Editor: @pjotrp Reviewer: @smoe Archive: 10.5281/zenodo.1012531

Status

status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/050362b7e7691d2a5d0ebed8251bc01e"><img src="http://joss.theoj.org/papers/050362b7e7691d2a5d0ebed8251bc01e/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/050362b7e7691d2a5d0ebed8251bc01e/status.svg)](http://joss.theoj.org/papers/050362b7e7691d2a5d0ebed8251bc01e)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer questions

@smoe, please carry out your review in this issue by updating the checklist below (please make sure you're logged in to GitHub). The reviewer guidelines are available here: http://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @pjotrp know.

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 7 years ago

Hello human, I'm @whedon. I'm here to help you with some common editorial tasks for JOSS. @smoe it looks like you're currently assigned as the reviewer for this paper :tada:.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As as reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all JOSS reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands
pjotrp commented 7 years ago

@smoe started review with:

Singularity is important and everything that fosters its acceptance is appreciated.

The setup is technically sound. I like the distributed effort. What seems missing is the concept of mirroring images for long time availability when a site goes down, which then truly preserves setups for scientific reproducibility. For that to happen, we would need identifiers that are not URL-dependent, though. Another feature I am missing is the automated creation of images, which would nicely work together with the description of CWL workflows that features references to software registries which in turn started to have references to Linux distributions (OMICtools now, bio.tools soon, I hope) and Debian at least having references to the registries already.

Not having an exact clue what you ask me to review now, I went through your JOSS author guidelines on http://joss.theoj.org/about#author_guidelines:

The other "requirements for submission" I find kind of redundant and I consider them to be fulfilled.

What I would typically do now is to install this locally and see if this works, which I kind of hope not to be imposed on me because the authors declare to us that they are working with exactly that setup themselves and that what we see is live on https://singularity-hub.org/collections or elsewhere - this is what should be explicitly stated in the paper.md, too.

Otherwise, I hope this was not too obvious, I am a fan.

pjotrp commented 7 years ago

@smoe please tick the boxes, thanks! @vsoch please follow up.

vsoch commented 7 years ago

Thank you @smoe and @pjotrp for starting the discussion! This is my first review with JOSS so please tell me if I am doing something incorrectly. I will follow up here with answers to your questions, and then make a suggested list of changes for the repository. @whedon is asking for the issues to be in the target repository or linked, and so I will address the decided changes there.

Singularity is important and everything that fosters its acceptance is appreciated.

Thank you! I feel the same way.

Mirroring and Backup

The setup is technically sound. I like the distributed effort. What seems missing is the concept of mirroring images for long time availability when a site goes down, which then truly preserves setups for scientific reproducibility.

I have the same sentiment that there is risk in any kind of infrastructure that is not properly mirrored. The challenge into integrating a mirror strategy into Singularity Registry is that it would be very challenging to impose a "one storage fits all" strategy. For example, by way of Docker the entire software can be brought up with:

docker-compose up -d

and this includes a postgres database image. However, the suggested deployment is to use a (non container) postgres image with something like a hot standby for backup, and then a mirror to some external storage (e.g., local filesystem) or cloud (e.g., object storage). What I think is in the scope of Singularity Registry is to provide the core software for organization, but leave the details about how backup is done to the user. This fits well with the two use cases that I see:

In other words, the responsibility of ensuring the replication of the registry is going to differ based on the deployment, and is understood to be the choice of the deploying institution. What I've done to address back up, from the base that I can control (singularity-hub) is to have a deployment with a mirrored database, and all images in backed up cloud storage. If the web server goes down and the postgres and/or its backup endure, then it's not an issue to bring up the application again. If the web server and both databases go down, the entire set of containers and metadata in Storage can be used to repopulate a new database, and users are linked via the Github repos.

Identifiers

For that to happen, we would need identifiers that are not URL-dependent, though.

Yes this is definitely the case! When a user references a Singularity Hub / Registry image, it might look something like this:

singularity pull shub://vsoch/hello-world

implicitly said in the above is the default url for the Singularity Hub registry - this produces the same outcome:

singularity pull shub://singularity-hub.org/vsoch/hello-world

and so if another registry was deployed with the same image uri, the registry name would need to be specified for the image. 

singularity pull shub:// <registry> /vsoch/hello-world

@smoe do you have suggestions for how this could be better? For Singularity Hub, the namespace is well maintained to correspond with Github. For a Singularity Registry, the naming schema is up to the owner. For example, if I were running a cluster, I might have namespace correspond to user names and/or groups, and then projects owned by them. Each image does carry it's unique version (a hash of the file) however it wouldn't be intuitive to search and find. What (I think) you are hinting at is some sort of global wrapper to registries (akin to a package manager) that can find an image based on criteria, no matter where it is! This is a great idea, although would be a separate project (that uses the registries).

Building

Another feature I am missing is the automated creation of images, which would nicely work together with the description of CWL workflows that features references to software registries which in turn started to have references to Linux distributions (OMICtools now, bio.tools soon, I hope) and Debian at least having references to the registries already.

Building is really important! And actually, I bit off more than I could chew a few years ago when my original vision for Singularity Hub was centered around the workflows. I realized we needed just storage and management before that.

To best answer this question, I want to make a distinction between Singularity Registry, and it's original base (the current Singularity Hub) that does in fact provide a building service. Singularity Hub was originally created to build images for users, as a free service, directly from a Github repository, and then have the images available immediately via the Singularity command line tool (for example, here is the repository that builds the image example above, when I push the build is done in the cloud automatically) and here is a quick image to show how it works:

figure1-shub-flow

It's thus very possible for some higher level workflow tool to use Singularity Hub (or Singularity Registry) API to get the command to download an image. What is out of scope is actually running the workflow, and how it plugs into a workflow tool. Here is an example with nextflow.

The Registry is a little bit of a different use case. I created the Registry because institutions / users came to me and said "I don't want to use Github, and I don't want even to share. I just want a manager for my institution images that I can push after build, and then pull from and be able to use on my resource." So Singularity Registry does not have builders or any cloud integrations - it's storage is a local file system as opposed to a cloud resource.

However, what I'm hoping to do for Singularity Hub and a new addition called SCI-F apps (and this is probably outside of the scope of this review) is to have easy templates / buttons so that a user can easily plug it into a workflow. There are many workflows so likely I would start with major cloud providers. For apps, the idea is to provide easy templates (for example, something to produce a SLURM job file) that helps with container integration.

Singularity Hub vs. Registry vs. Github

Docker Hub

Should users of docker also consider singularityhub instead of hub.docker.io? The two registries are similar in that they produce a container, however Docker Hub is fundamentally different in that it serves layers (.tar.gz) and not entire images. If one layer changes or is lost, the same image cannot be built. Singularity Registry and Hub serve entire images, no assembly required. I don't see the two as competitors, because many choose to use Docker Hub as a base to build a new image. I would say the decision of which to use is up to the user. I commonly build Docker images and then just pull them as singularity:

singularity run docker://godlovedc/lolcow

Any Docker image can be immediately converted to Singularity by adding the .tar.gz layers to an empty image base. As far as I know, Docker doesn't have a nice conversion to get a Docker image from a Singularity one. Some users (myself included) find the Singularity recipe format to be more intuitive than Docker, and make the decision based on that.

Are there features of singularity that are particularly supported by singularityhub?

Singularity aims to support scientific containers, so Singularity Hub and Registry take an extra step to serve metadata about the containers via the API. It's important to know about usage (downloads and stars) but also software, environment variables, labels, and runscripts. This supports being able to do more research analytics across containers to better understand how containers (and more broadly software) help answer scientific questions. Given the issues we have with reproducibility, I think this is essential

Citations

Thank you for linking to the citation page. I'd be happy to add more references. Given that the description talks on a high level about Docker, containers, would the most appropriate references be to related web content? It's challenging to find papers that are specifically about container software and infrastructure - there are mostly analyses that use them.

Ongoing Research Projects

I can definitely say that community members are using it, reflected in discussion on the list and Github issues, but I can't comment on specific projects.

Thank you for your thorough review! I can address the above points after you've taken a look, and I'll try to organize the issues and then fixes in context of the check boxes.

smoe commented 7 years ago

Dear @vsoch, please rest assured that I did not expect you to add any extra functionality to your hub. I need to thank you for your detailed replies to my comments - I hope you could copy some good parts from elsewhere. It is also my first review for JOSS - interesting :)

Easiest first: Concerning references I suggest you reference/cite whatever that you want to reference. It is the least that JOSS can do for you.

Concerning the writeup in paper.md, I tend to think that JOSS gives "us" more freedom than the many very short papers in JOSS are suggesting that the casual reader finds. For instance I found some early submission by our editor himself that has a huge picture - yes, you can have diagrams in your JOSS paper or whatever you like, if you like.

I propose to have the summary in paper.md in two parts. The first paragraph would be something alike "Introduction to containers, why we need them and why we share them" and that you have estsablished a (the first?) user-installable non-centralized Open Source infrastructure to facilitate that exchange. You run a public instance at URL. The text that you already have can then basically follow as it is. Let 90% of the readers stop reading at that point - they either know that they want it and go to the repository to read the documentation, that they do not want it, or they follow the link to the instance. The ones that are interested but do not want to read the documentation shall then get a second paragraph with selected lines of what you explained above to explain how you did it, how different parts relate to each other ... and whatever you want people to know that turn around in their chairs to tell their office mates about what they have just read.

@pjotrp , you are the boss, direct me if you see this differently, but I am truly excited about @vsoch's writing above. It has a bit of a conference coffee-break atmosphere in which you give a summary of what you are doing. I can well imagine that many readers would appreciate a bit of that in that second paragraph of your paper.md summary. This is also a bit of a criticism of a few articles in JOSS that treat JOSS like a "it is in the docs" DOI-machine. But frankly, a singuarity-hub is more than an R API of some C library and should get a bit of a special treatment.

pjotrp commented 7 years ago

Mind: The paper should not be lengthy, according to the JOSS guidelines. It is better to have real information in a README or other doc in the repo (which can be referenced from the paper). Still the paper is a point of call for readers, so be concise but informative. A figure usually is a good idea.

pjotrp commented 7 years ago

BTW have you thought of using IPFS as a backend? That could solve the redundancy question. We are experimenting with IPFS in several projects.

smoe commented 7 years ago

Right, not long. Dear @vsoch, you will know what to do and I will be happy with it. That I have not checked everything above, yet, is just because I did not find the time, yet. It has nothing to do with your work.

vsoch commented 7 years ago

Dear @smoe, I have done ample changes to the paper.md and will move them (from our conversation above) here.

Submission Changes

@pjotrp - I don't know much about the interplanetary file system, but it sounds interesting! I will take a look. I think backup for storage is important and something we can think about and test with various users, and eventually provide a "best practice" suggestion. To foreshadow that, I added a section to the FAQ

pjotrp commented 7 years ago

@vsoch I like the new version of the paper better. I think it still can be improved a bit. First of all, don't keep the reader guessing (as a rule). I would put main section 3 first. Sections 1 and 2 are long for what they need to say, better to shorten them to a few lines each and embed them in section 3. I mean, we can assume that people understand reproducibility. In the context of containers just reference a paper that discusses that. As for what a container is, I would keep that simple too and avoid words like hazard, hitch, ideal, storm, etc. which are not often seen in scientific papers.

How about something like:

Singularity Registry is a non-centralized Free and Open Source infrastructure to facilitate sharing of containers that can be installed at the institutional or personal level and facilitates caching of container objects.

A container is a bundle of binary deployed software that can be run consistently if the platform supports it. It is an aid in reproducibility [citation] because different researchers can run the exact same software stack on different underlying (Linux Intel) systems. Docker has become popular as a general container system because it allows software to be bundled with administrator privileges without actually having root and it allows the running of tools with restricted privileges which has impact on security. Docker, however, is known to be complex software and that has prevented admission in HPC environments. Singularity offers similar features to Docker for software deployment but requires less privileges than Docker and is therefore easier accepted in HPC environments since its introduction. So far, Singularity has been deployed in 45 HPC environments (right?).

We have created a solution etc. ....

vsoch commented 7 years ago

@pjotrp I shorted the introduction quite a bit, and moved the bulk of the background into the docs introduction. Here is the updated paper. Your suggested introduction is fantastic, and I only tweaked it slightly (Docker in fact has huge security issues so it was a non starter for HPC).

I left the remainder of the description with pictures assuming that interested readers will continue, and dis-interested will have been satisfied by the early description and move on.

pjotrp commented 7 years ago

@vsoch I like it. Do people not have root-like privileges inside the singularity container? It may read that way now. I think you mean that singularity does not actually use root privileges to run the container - it stays in userland. Correct? After this niggle, @smoe can you sign off on your review and we'll take it to the next level? Thanks!

vsoch commented 7 years ago

@pjotrp that is correct - inside a Singularity container you are the same user as outside the container:

$ whoami
vanessa
$ singularity shell docker://ubuntu
Docker image path: index.docker.io/library/ubuntu:latest
Cache folder set to /home/vanessa/.singularity/docker
Creating container runtime...
Singularity: Invoking an interactive shell within container...

Singularity ubuntu:~/Documents/Dropbox/Code/sregistry/sregistry> whoami
vanessa

And thus if you run as root, you are root inside as well!

$ sudo whoami
root
sudo singularity shell docker://ubuntu
Docker image path: index.docker.io/library/ubuntu:latest
Cache folder set to /root/.singularity/docker
[5/5] |===================================| 100.0% 
Creating container runtime...
Singularity: Invoking an interactive shell within container...

Singularity ubuntu:/home/vanessa/Documents/Dropbox/Code/sregistry/sregistry> whoami
root
pjotrp commented 7 years ago

OK, I think that means someone needs to be root to create containers? That would be an HPC administrator? How would apt or rpm work otherwise?

vsoch commented 7 years ago

@pboesu yes, the typical workflow is to build on a resource (a personal work station,for example) with root, and then run the container anywhere. The user running the container cannot escalate to root on the host system, as would be possible with Docker. It matches nicely to a typical scientific workflow, where you write and test scripts on a personal computer, and then run at scale. The workflow typically look like this (with Singularity 2.3 and lower)

image

For a hosted registry, the three general use cases I have in mind are the following:

For using, meaning running, executing, shelling (all the functions important for working on a cluster) the user's have the same permissions as they would as their normal user. Binding the host to the container is done automatically for some folders (eg, tmp, the present working directory, the user's home) and on request with --bind.

pjotrp commented 7 years ago

Good. Have a look at your text to see if that is reflected. On to @smoe ...

vsoch commented 7 years ago

Thanks @pjotrp, I don't in fact think the use cases were properly reflected, and they are important to give users a sense of "who cares?" I've added a use cases section to the docs to address this.

smoe commented 7 years ago

Dear @vsoch,

I am happy, except that I have not yet installed it - and I really need to do something else today, I am afraid - hey, I have installed docker on a remote server already. @pjotrp, would it be OK to skip that as I have no doubt at all that this will either work or be immediately fixed if not should I run into a local issue?

Something that I should possibly comment on are the references. The DOIs are not in the bib file. Being a JOSS-newbie myself I can only speculate that this is where our editor would like to see them as described here https://tex.stackexchange.com/questions/3802/how-to-get-doi-links-in-bibliography . Also, I had the impression that there is a syntax glitch with the ARTICLE tags in there. Seen by chance, the @Kurtzer2017-x reference lacks at terminal "j". @pjotrp, please kindly help me out if I got this wrong.

I found your comment to @pjotrp important: There is no chance for a priviledge escalation. It is the special feature why so many HPC environments are prefering Singularity over Docker: It reduces the risks for running foreign images to the general risk of running foreign software, which is the precondition for sharing. Conversely, if you want to share images, you will prefer using what is safe. And that again is why this registry is so important for Singularity to shine.

@pjotrp, I happily sign-off this review to bring the submission to some "next level". Is there anything special I need to do about it?

pjotrp commented 7 years ago

@smoe could you try a quick install to be able to tick the Functionality boxes? It is a requirement rather. No rush, you have been quick so far.

smoe commented 7 years ago

Sigh. How comes I knew this would happen. I ran what I understood to be run and ran into an issue. @vsoch, I get you an account on that machine (regular Debian Stretch) if that is of any help. I followed the instructions on https://github.com/singularityhub/sregistry/blob/master/scripts/prepare_instance.sh with docker already installed.

I successfully run the hello-world as root. As a regular user I get a "permission denied" for building in the fresh sregistry git checkout. I am not fluent enough with docker - should the "docker build ." possibly also be executed with sudo and that was only working on your site without it since you already downloaded everything you need to download? Over here it is quite busy to build images when I first initiated the build.

So, running with sudo, I end up with:

.... many lines omitted .... 
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/rview (rview) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/vi (vi) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/view (view) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/ex (ex) in auto mode
update-alternatives: using /usr/bin/vim.basic to provide /usr/bin/editor (editor) in auto mode
Setting up libtasn1-doc (4.2-3+deb8u3) ...
Processing triggers for systemd (215-17+deb8u4) ...
Processing triggers for libc-bin (2.19-18+deb8u4) ...
/sbin/ldconfig.real: Cannot lstat /usr/lib/x86_64-linux-gnu/libMagickCore-6.Q16.so: No such file or directory
/sbin/ldconfig.real: Cannot lstat /usr/lib/x86_64-linux-gnu/libMagickWand-6.Q16.so: No such file or directory
/sbin/ldconfig.real: Cannot lstat /usr/lib/x86_64-linux-gnu/libcairo-gobject.so: No such file or directory
/sbin/ldconfig.real: Cannot lstat /usr/lib/x86_64-linux-gnu/libcairo-script-interpreter.so: No such file or directory
/sbin/ldconfig.real: Cannot lstat /usr/lib/x86_64-linux-gnu/libcairo.so: No such file or directory
/sbin/ldconfig.real: Cannot lstat /usr/lib/x86_64-linux-gnu/libgdk_pixbuf-2.0.so: No such file or directory
/sbin/ldconfig.real: Cannot lstat /usr/lib/x86_64-linux-gnu/libgdk_pixbuf_xlib-2.0.so: No such file or directory
/sbin/ldconfig.real: Cannot lstat /usr/lib/x86_64-linux-gnu/libgio-2.0.so: No such file or directory
/sbin/ldconfig.real: Cannot lstat /usr/lib/x86_64-linux-gnu/libglib-2.0.so: No such file or directory
/sbin/ldconfig.real: Cannot lstat /usr/lib/x86_64-linux-gnu/libgmodule-2.0.so: No such file or directory
/sbin/ldconfig.real: Cannot lstat /usr/lib/x86_64-linux-gnu/libgobject-2.0.so: No such file or directory
/sbin/ldconfig.real: Cannot lstat /usr/lib/x86_64-linux-gnu/libgthread-2.0.so: No such file or directory
/sbin/ldconfig.real: Cannot lstat /usr/lib/x86_64-linux-gnu/librsvg-2.so: No such file or directory
 ---> e320dd67f67f
Removing intermediate container 9d4fc33d83e1
Step 4/21 : RUN git clone https://www.github.com/singularityware/singularity.git
 ---> Running in acfb57909644
Cloning into 'singularity'...
 ---> 4fa9bdac1522
Removing intermediate container acfb57909644
Step 5/21 : WORKDIR singularity
 ---> 4dd86eced674
Removing intermediate container c81613b8fbe6
Step 6/21 : RUN ./autogen.sh && ./configure --prefix=/usr/local && make && make install
 ---> Running in eb2b8cd2c10b
+autoreconf -i -f
aclocal: error: aclocal: file '/usr/share/aclocal/gsettings.m4' does not exist
autoreconf: aclocal failed with exit status: 1
The command '/bin/sh -c ./autogen.sh && ./configure --prefix=/usr/local && make && make install' returned a non-zero code: 1

If I get this right, then this was all executed in the docker environment, so it should not have mattered that I did not have the autoconf etc. package installed. The missing gsettings.m4 is in the libglic2.0-dev package:

$ apt-file search gsettings.m4
gnome-builder: /usr/share/gnome-builder/plugins/autotools_templates/resources/m4/gsettings.m4
libglib2.0-dev: /usr/share/aclocal/gsettings.m4

For the better overview of yours, starting the process again, I get:

moeller@server2:~/sregistry$ docker build  .
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.32/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&session=d35d88477cc93e251cb5dae3c5fbf624c49cccdf9a0aaf06c4875279daeb60d3&shmsize=0&target=&ulimits=null: dial unix /var/run/docker.sock: connect: permission denied
moeller@server2:~/sregistry$ ^Ccker build  .
moeller@server2:~/sregistry$ sudo docker build -t vanessa/sregistry .
Sending build context to Docker daemon  22.77MB
Step 1/21 : FROM python:3.5.1
 ---> a00e9008965a
Step 2/21 : ENV PYTHONUNBUFFERED 1
 ---> Using cache
 ---> 41200125058b
Step 3/21 : RUN apt-get update && apt-get install -y     anacron     libopenblas-dev     gfortran     pkg-config     libxml2-dev     libxmlsec1-dev     libhdf5-dev     libgeos-dev     build-essential     openssl     wget     vim
 ---> Using cache
 ---> e320dd67f67f
Step 4/21 : RUN git clone https://www.github.com/singularityware/singularity.git
 ---> Using cache
 ---> 4fa9bdac1522
Step 5/21 : WORKDIR singularity
 ---> Using cache
 ---> 4dd86eced674
Step 6/21 : RUN ./autogen.sh && ./configure --prefix=/usr/local && make && make install
 ---> Running in c4bc5a22558f
+autoreconf -i -f
aclocal: error: aclocal: file '/usr/share/aclocal/gsettings.m4' does not exist
autoreconf: aclocal failed with exit status: 1
The command '/bin/sh -c ./autogen.sh && ./configure --prefix=/usr/local && make && make install' returned a non-zero code: 1

On a sidenote, the python3.5.1 image you ask for is based on Debian Jessie. That is the oldstable one - maintained for another while, but ... it should be the current release "Stretch", not "Jessie". I just checked the images for docker and among the readily prepared images it seems like that python 3.6.3 is the closest choice that uses Stretch (https://store.docker.com/images/python). Frankly, though, so much gets installed for that image, that it does not really matter if there is a Python already shipping or not. I propose you just get an empty Debian image and add python 3.5 with "apt-get install python3" - as 3.5 is still the default python3 installation (https://packages.qa.debian.org/p/python3-defaults.html).

So, after taking some deep breath I propose we just get this [the build error, the version of the distro is on you] fixed. My offer to get you an account on the machine still holds (it is not doing anything but helping Einstein@Home these days, so don't worry and send me a public ssh key if you like).

vsoch commented 7 years ago

Hey @smoe ! Sorry for your troubles! There are two things here - preparing your host to use Docker and compose, and then running the image. I

Preparing Host

It's best to follow the instructions for your distribution here. Note that I provided instructions for an Ubuntu host. There are detailed instructions provided by Docker for non ubuntu, along with docker-compose.

Important you need to, after you add your user to Docker's group, log out and back in before you continue. The call to docker run hello-world should succeed.

Bringing up Registry

You shouldn't need to build the image, but if you want to:

docker build -t vanessa/sregistry .

Note that I didn't run just docker build . as you did. However, the entire image is provided pre-build on Docker Hub:

https://hub.docker.com/r/vanessa/sregistry/builds/

so you should be able to do:

docker-compose up -d

Let's walk through these steps, one at a time, and discuss when you hit a snag (and please don't move on). I've had two others deploy without much issue, so I think it comes down to your installation of docker and compose. I will update the docs to explicitly say that the prepare script example that I provided is for an ubuntu host.

vsoch commented 7 years ago

hey @smoe ! I updated the docs to very specifically say to install docker an docker-compose for your particular choice, along with needing to log in and out.

If you want to see one of my colleagues deployed sregistry, check out randomroad

Please let me know what is (and isn't clear) and I'll improve the instructions in the docs. I want to make sure they are user friendly for a wide range of audiences.

smoe commented 7 years ago

I had used the wrong open shell for the build in which I had just added me to the docker user group. So, thank you for that. Indeed, I can now run as a regular user for which I had to use sudo before. Good news first: The docker-compose just worked fine. For @pjotrp:

moeller@server2:~/sregistry$ docker-compose up -d
Pulling redis (redis:latest)...
latest: Pulling from library/redis
065132d9f705: Pull complete
be9835c27852: Pull complete
f4a0d1212c38: Pull complete
43be9e9f0fb9: Pull complete
a1bca8e532ec: Pull complete
382eae952932: Pull complete
Digest: sha256:ebb396dc3ac00e8eb4a64c1c022ef41ef16801f31ff98b16916a77fdc7252e67
Status: Downloaded newer image for redis:latest
Pulling db (postgres:latest)...
latest: Pulling from library/postgres
219d2e45b4af: Pull complete
87b4d6274d77: Pull complete
2569a32ee6dd: Pull complete
23b4d0fc3192: Pull complete
8275aae461c7: Pull complete
45087ee6fc31: Pull complete
2d2265f720a6: Pull complete
2860cdb4889c: Pull complete
597205e436ef: Pull complete
d445130f222d: Pull complete
def74255ce78: Pull complete
4a3042ea21df: Pull complete
686ac5adc42e: Pull complete
Digest: sha256:242a4569b89cda783c164380aaa202dc0332d79c49b856a0554c3a94de8a4dfa
Status: Downloaded newer image for postgres:latest
Pulling uwsgi (vanessa/sregistry:latest)...
latest: Pulling from vanessa/sregistry
51f5c6a04d83: Already exists
65e9ddd8bd7a: Already exists
c41545ebedf5: Already exists
04aed1875617: Already exists
cf8af6bdf113: Already exists
b1dde6937fab: Already exists
c8bc9f75687c: Already exists
a91d9d9365df: Pull complete
99c405b8c55e: Pull complete
0e4bb0a0976e: Pull complete
b225dcdce7e1: Pull complete
ab1b2e6030c9: Pull complete
9ac9ee44eb32: Pull complete
7c87a686bfaa: Pull complete
fafbd6f79b25: Pull complete
1006ffb0701c: Pull complete
a065aeb43100: Pull complete
bbe9db7595f1: Pull complete
af848c89cfcb: Pull complete
ec8d609a6d9b: Pull complete
d0110b169e20: Pull complete
Digest: sha256:57be162740e34e8bd415b957c137a9b98116c5b23d2be36ec8947d2602f74dde
Status: Downloaded newer image for vanessa/sregistry:latest
Pulling nginx (nginx:latest)...
latest: Pulling from library/nginx
afeb2bfd31c0: Pull complete
7ff5d10493db: Pull complete
d2562f1ae1d0: Pull complete
Digest: sha256:af32e714a9cc3157157374e68c818b05ebe9e0737aac06b55a09da374209a8f9
Status: Downloaded newer image for nginx:latest
Creating sregistry_redis_1
Creating sregistry_db_1
Creating sregistry_uwsgi_1
Creating sregistry_nginx_1
Creating sregistry_worker_1

Executed with sudo or after properly being accepted as a member of the docker group on the system, the error remains the same:

moeller@server2:~/sregistry$ docker build -t vanessa/sregistry .
Sending build context to Docker daemon  22.79MB
Step 1/21 : FROM python:3.5.1
 ---> a00e9008965a
Step 2/21 : ENV PYTHONUNBUFFERED 1
 ---> Using cache
 ---> 41200125058b
Step 3/21 : RUN apt-get update && apt-get install -y     anacron     libopenblas-dev     gfortran     pkg-config     libxml2-dev     libxmlsec1-dev     libhdf5-dev     libgeos-dev     build-essential     openssl     wget     vim
 ---> Using cache
 ---> e320dd67f67f
Step 4/21 : RUN git clone https://www.github.com/singularityware/singularity.git
 ---> Using cache
 ---> 4fa9bdac1522
Step 5/21 : WORKDIR singularity
 ---> Using cache
 ---> 4dd86eced674
Step 6/21 : RUN ./autogen.sh && ./configure --prefix=/usr/local && make && make install
 ---> Running in 849c7875abcc
+autoreconf -i -f
aclocal: error: aclocal: file '/usr/share/aclocal/gsettings.m4' does not exist
autoreconf: aclocal failed with exit status: 1
The command '/bin/sh -c ./autogen.sh && ./configure --prefix=/usr/local && make && make install' returned a non-zero code: 1

Does the build still work for you? Concerning the documentation, I just followed https://github.com/singularityhub/sregistry/blob/master/scripts/prepare_instance.sh to which your documentation links and that does the build. But then again, our dear editor wants me to build it, so, the pointer to prepare_instance.sh was fine to this point. I had not even seen install.md before your kind indirect pointer on https://github.com/singularityhub/sregistry/issues/35 .

You are right in that you are granting me the opportunity to learn some docker over the process. Hey, I had to install it when what I really wanted to install was Singularity :) Please allow that I continue with my learning tomorrow, night time over here. Many thanks for all your friendly and quick help.

vsoch commented 7 years ago

It looks like it's having trouble with singularity itself (and not the registry). I updated the image to clone from development, if you want to try building again. Here is also a complete build I did from scratch to confirm that it works for me: https://asciinema.org/a/141559?speed=3

Do you think it's too easy to find the prepare_instance.sh script (over the install instructions?) If so, let's definitely fix that.

No worries, chat tomorrow! Do

smoe commented 7 years ago

Sigh. I have removed all images (and running containers), and even after your change to the Dockerfile, I still get

/sbin/ldconfig.real: Cannot lstat /usr/lib/x86_64-linux-gnu/libglib-2.0.so: No such file or directory
/sbin/ldconfig.real: Cannot lstat /usr/lib/x86_64-linux-gnu/libgmodule-2.0.so: No such file or directory
/sbin/ldconfig.real: Cannot lstat /usr/lib/x86_64-linux-gnu/libgobject-2.0.so: No such file or directory
/sbin/ldconfig.real: Cannot lstat /usr/lib/x86_64-linux-gnu/libgthread-2.0.so: No such file or directory
/sbin/ldconfig.real: Cannot lstat /usr/lib/x86_64-linux-gnu/librsvg-2.so: No such file or directory
 ---> dbba45aef6b8
Removing intermediate container cbc044f24d7c
Step 4/21 : RUN git clone https://www.github.com/singularityware/singularity.git
 ---> Running in fdcc79b73821
Cloning into 'singularity'...
 ---> dd68929297c4
Removing intermediate container fdcc79b73821
Step 5/21 : WORKDIR singularity
 ---> 2087f0109a29
Removing intermediate container 137c1d022d4d
Step 6/21 : RUN ./autogen.sh && ./configure --prefix=/usr/local && make && make install
 ---> Running in c2bda8ebff01
+autoreconf -i -f
aclocal: error: aclocal: file '/usr/share/aclocal/gsettings.m4' does not exist
autoreconf: aclocal failed with exit status: 1
The command '/bin/sh -c ./autogen.sh && ./configure --prefix=/usr/local && make && make install' returned a non-zero code: 1

but (and this is a big but) https://github.com/singularityhub/sregistry/pull/39 fixed that for me. In my commit note I speculated that the mirrors of Debian may differ slightly in their punctuality of updates. Or do you have a web cache for the pip installation?

Anyway, I can proceed to the configuration now.

vsoch commented 7 years ago

Your fix is added! I think it might also be missing cmake? It could be a different version related to mirrors. I've merged your additional library, and also just added cmake. It's very strange, because it works for me (and the few others that have installed it) so hard to pin down what it might be. I also read that the error pops up when package config fails to link the file.

smoe commented 7 years ago

Ok ... is there a place for me to look when encounering an "internal server error" that is displayed as the only content? To be seen on server2.ffoh.de.

smoe commented 7 years ago

A missing cmake was not complained about.

Is there a chance to have you migrate to Debian Stretch? I volunteer to perform the test build.

vsoch commented 7 years ago

Try

docker-compose logs uwsgi

and please tell me what you find!

vsoch commented 7 years ago

You shouldn't need to worry about building on your particular host. I can't explain the issues that you are having, but since there is a pre-built container provided --> https://hub.docker.com/r/vanessa/sregistry/builds/ that should be more than sufficient to reproduce the application exactly as intended. Building on your own, actually, would produce a different image (as we have seen) given that different mirrors / other versions could be obtained with installation commands.

vsoch commented 7 years ago

I think it may be the case that you need to follow the instructions in setup first before you start the image. Did you see the page here?

https://singularityhub.github.io/sregistry/install.html

That configuration needs to be done before you start the image. If you are using the one from docker hub, the files are mounted so they will be updated when you bring up the image.

smoe commented 7 years ago

I had disabled all the remote authentications - could that be the culprit? Is there a logfile that would indicate what the problem truly is? I would need to attach to your image for that, right?

vsoch commented 7 years ago

Hi @smoe - please see my instruction above:

docker-compose logs uwsgi

that will show the logs of the main server, which is the image called uwsgi. You do need to follow the instructions and choose some method to authenticate. The other option is that I could allow you (the user) to bring up a registry that you are unable to login to. Please run that command and show me the output (which will correspond to the server error). Minimally we can set up dummy variables so the server comes up without error, albeit you probably won't be able to do much without the configurations.

smoe commented 7 years ago

It says that the SECRET_KEY would be empty - which it is not on my end. I also failed to see at which point you forward that file into your containers - but I have complete trust that we can settle that independently from this review. So I pressed the last button.

I personally like the idea of a "display only" image server very much where users could anonymously download the images they want. This could possibly come very handy e.g. for the CWL community to automatically prepare images for whatever favorite pipelines of theirs. Feedback on those images could be with some github solution elsewhere.

I had another look at your infrastructure and you indeed set up everything with python3 on Debian Jessie. The prior noted location dependency is presumably caused by the automated mirror finding with httpredir. I find this somewhat shocking, I must admit.

~/sregistry$ docker run -it  python:3.5.1 bash
root@717f5c53a176:/# pip --version
pip 8.1.2 from /usr/local/lib/python3.5/site-packages (python 3.5)
root@717f5c53a176:/# cat /etc/apt/sources.list
deb http://httpredir.debian.org/debian jessie main
deb http://httpredir.debian.org/debian jessie-updates main
deb http://security.debian.org jessie/updates main

To help your new users it would be nice if you could catch the one or other error and come up with a minimal system. 

uwsgi_1   | Wed Oct 11 11:48:47 2017 - *** Operational MODE: single process ***
uwsgi_1   | Traceback (most recent call last):
uwsgi_1   |   File "shub/wsgi.py", line 7, in <module>
uwsgi_1   |     application = get_wsgi_application()
uwsgi_1   |   File "/usr/local/lib/python2.7/dist-packages/django/core/wsgi.py", line 13, in get_wsgi_application
uwsgi_1   |     django.setup(set_prefix=False)
uwsgi_1   |   File "/usr/local/lib/python2.7/dist-packages/django/__init__.py", line 22, in setup
uwsgi_1   |     configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
uwsgi_1   |   File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 56, in __getattr__
uwsgi_1   |     self._setup(name)
uwsgi_1   |   File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 41, in _setup
uwsgi_1   |     self._wrapped = Settings(settings_module)
uwsgi_1   |   File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 110, in __init__
uwsgi_1   |     mod = importlib.import_module(self.SETTINGS_MODULE)
uwsgi_1   |   File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
uwsgi_1   |     __import__(name)
uwsgi_1   |   File "./shub/__init__.py", line 2, in <module>
uwsgi_1   |     from .celery import shubcelery as celery_app
uwsgi_1   |   File "./shub/celery.py", line 45, in <module>
uwsgi_1   |     from opbeat.contrib.django.models import (
uwsgi_1   |   File "/usr/local/lib/python2.7/dist-packages/opbeat/contrib/django/models.py", line 226, in <module>
uwsgi_1   |     if 'opbeat.contrib.django' in django_settings.INSTALLED_APPS:
uwsgi_1   |   File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 56, in __getattr__
uwsgi_1   |     self._setup(name)
uwsgi_1   |   File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 41, in _setup
uwsgi_1   |     self._wrapped = Settings(settings_module)
uwsgi_1   |   File "/usr/local/lib/python2.7/dist-packages/django/conf/__init__.py", line 129, in __init__
uwsgi_1   |     raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
uwsgi_1   | django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.
uwsgi_1   | Wed Oct 11 11:48:48 2017 - unable to load app 0 (mountpoint='') (callable not found or import error)
uwsgi_1   | Wed Oct 11 11:48:48 2017 - *** no app loaded. going in full dynamic mode ***
uwsgi_1   | Wed Oct 11 11:48:48 2017 - *** uWSGI is running in multiple interpreter mode ***
uwsgi_1   | Wed Oct 11 11:48:48 2017 - spawned uWSGI master process (pid: 48)
uwsgi_1   | Wed Oct 11 11:48:48 2017 - spawned uWSGI worker 1 (pid: 52, cores: 1)
uwsgi_1   | Wed Oct 11 11:48:55 2017 - --- no python application found, check your startup logs for errors ---
vsoch commented 7 years ago

hey @smoe! The secret key is a django application detail, and you must have missed where I wrote about it in the instructions to generate one. It's important for security features for any django application, and as you discovered, it will get angry at you if you don't have it. Would it be useful to step back and go back to the instructions and go through them to make sure you set everything up proper?

I personally like the idea of a "display only" image server very much where users could anonymously download the images they want.

Yes! And actually a registry can serve as exactly this. This doesn't do away with the need of someone (some admin) setting it up and managing the images. And we can do slightly better because we don't only download anonymously, we can track the number of uses (anonymously) to better understand which software is most useful.

This could possibly come very handy e.g. for the CWL community to automatically prepare images for whatever favorite pipelines of theirs. Feedback on those images could be with some github solution elsewhere.

Yes I had this in mind when I created push - images could be pushed automatically after a pull request (PR) and discussion on a version controlled base like Github.

I had another look at your infrastructure and you indeed set up everything with python3 on Debian Jessie. The prior noted location dependency is presumably caused by the automated mirror finding with httpredir. I find this somewhat shocking, I must admit.

I am using the Docker provided python image, which likely uses those mirrors. I don't have much opinion, but if you want to avoid building all together I can suggest just pulling from Docker Hub.

smoe commented 7 years ago

Hi @vsoch, admittedly, my very initial run was without the secret key since the prepare_instance.sh script did not generate it. But a next invocation, with all images removed if I did not mess it up, had it in place. And this is how it still is:

moeller@server2:~/sregistry$ cat shub/settings/secret.py
SECRET_KEY='djangoSecretKeytyw!b^d#elqz2d@c!m)o)g=pfnkilc07mof=pa8f@zszl5^fe=forSregistry'
moeller@server2:~/sregistry$ ls -l shub/settings/secret.py
-rw-r--r-- 1 moeller moeller 91 Oct 10 17:43 shub/settings/secret.py

I had used the django generator you referred to create it.

Concerning the python image I am still a bit unhappy. It seems like this comes with the autotools already installed, and it even has git. Why does have python image feature all that?

If you do not mind too much, please add those bits in - "back in" for autoconf and friends. git \ autoconf \ automake \ libtool

This is just because it is needed and might not work once you upgrade to a different version of that image. Btw, I had a building version with Debian Stretch instead of Debian Jessie-based python:3.5.1.

vsoch commented 7 years ago

hey @smoe! Are you still have trouble with the secret key? You shouldn't need to remove or rebuild images, just restart with docker compose (and this is true whenever a change is done and the images are running):

docker-compose restart

I added more verbose documentation about this to this section, in case you missed it.

I can't comment on why the base python chooses to include core git (likely a choice of the developer) but per your request, I've added back in the libraries you listed.

smoe commented 7 years ago 
moeller@server2:~/sregistry$ docker rmi vanessa/sregistry
Error: No such image: vanessa/sregistry
moeller@server2:~/sregistry$ docker-compose up -d
Pulling uwsgi (vanessa/sregistry:latest)...
latest: Pulling from vanessa/sregistry
51f5c6a04d83: Already exists
65e9ddd8bd7a: Already exists
c41545ebedf5: Already exists
04aed1875617: Already exists
cf8af6bdf113: Already exists
b1dde6937fab: Already exists
c8bc9f75687c: Already exists
7ec2ccb4b7ef: Pull complete
45aa57212348: Pull complete
bbe3ffa886f4: Pull complete
ae4988fdcaba: Pull complete
88774b19f593: Pull complete
a9fbd563a04b: Pull complete
d66e99c8cb84: Pull complete
03c3b0adfe39: Pull complete
4e7c6ac411f1: Pull complete
d2ffd7d5720b: Pull complete
0ed82f0bc059: Pull complete
7a302558f857: Pull complete
af4abfaab25d: Pull complete
e9347ba1dfcd: Pull complete
57c07257b2a4: Pull complete
Digest: sha256:12d7c6ddc801f07813dd068308048b76637488255d3219bb7e91906531b35e87
Status: Downloaded newer image for vanessa/sregistry:latest
Creating sregistry_redis_1
Creating sregistry_db_1
Creating sregistry_uwsgi_1
Creating sregistry_nginx_1
Creating sregistry_worker_1
moeller@server2:~/sregistry$ pwd
/home/moeller/sregistry
moeller@server2:~/sregistry$ cat shub/settings/secret.py
SECRET_KEY='djangoSecretKeytyw!b^d#elqz2d@c!m)o)g=pfnkilc07mof=pa8f@zszl5^fe=forSregistry'
...
uwsgi_1   | Wed Oct 11 15:24:11 2017 - your server socket listen backlog is limited to 100 connections
uwsgi_1   | Wed Oct 11 15:24:11 2017 - your mercy for graceful operations on workers is 60 seconds
uwsgi_1   | Wed Oct 11 15:24:11 2017 - mapped 145536 bytes (142 KB) for 1 cores
uwsgi_1   | Wed Oct 11 15:24:11 2017 - *** Operational MODE: single process ***
uwsgi_1   | Traceback (most recent call last):
uwsgi_1   |   File "shub/wsgi.py", line 7, in <module>
uwsgi_1   |     application = get_wsgi_application()
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/core/wsgi.py", line 13, in get_wsgi_application
uwsgi_1   |     django.setup(set_prefix=False)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/__init__.py", line 22, in setup
uwsgi_1   |     configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/conf/__init__.py", line 56, in __getattr__
uwsgi_1   |     self._setup(name)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/conf/__init__.py", line 41, in _setup
uwsgi_1   |     self._wrapped = Settings(settings_module)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/conf/__init__.py", line 110, in __init__
uwsgi_1   |     mod = importlib.import_module(self.SETTINGS_MODULE)
uwsgi_1   |   File "/usr/local/lib/python3.5/importlib/__init__.py", line 126, in import_module
uwsgi_1   |     return _bootstrap._gcd_import(name[level:], package, level)
uwsgi_1   |   File "<frozen importlib._bootstrap>", line 986, in _gcd_import
uwsgi_1   |   File "<frozen importlib._bootstrap>", line 969, in _find_and_load
uwsgi_1   |   File "<frozen importlib._bootstrap>", line 944, in _find_and_load_unlocked
uwsgi_1   |   File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
uwsgi_1   |   File "<frozen importlib._bootstrap>", line 986, in _gcd_import
uwsgi_1   |   File "<frozen importlib._bootstrap>", line 969, in _find_and_load
uwsgi_1   |   File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
uwsgi_1   |   File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
uwsgi_1   |   File "<frozen importlib._bootstrap_external>", line 662, in exec_module
uwsgi_1   |   File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
uwsgi_1   |   File "./shub/__init__.py", line 2, in <module>
uwsgi_1   |     from .celery import shubcelery as celery_app
uwsgi_1   |   File "./shub/celery.py", line 45, in <module>
uwsgi_1   |     from opbeat.contrib.django.models import (
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/opbeat/contrib/django/models.py", line 226, in <module>
uwsgi_1   |     if 'opbeat.contrib.django' in django_settings.INSTALLED_APPS:
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/conf/__init__.py", line 56, in __getattr__
uwsgi_1   |     self._setup(name)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/conf/__init__.py", line 41, in _setup
uwsgi_1   |     self._wrapped = Settings(settings_module)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/conf/__init__.py", line 129, in __init__
uwsgi_1   |     raise ImproperlyConfigured("The SECRET_KEY setting must not be empty.")
uwsgi_1   | django.core.exceptions.ImproperlyConfigured: The SECRET_KEY setting must not be empty.
uwsgi_1   | Wed Oct 11 15:24:13 2017 - unable to load app 0 (mountpoint='') (callable not found or import error)
uwsgi_1   | Wed Oct 11 15:24:13 2017 - *** no app loaded. going in full dynamic mode ***
uwsgi_1   | Wed Oct 11 15:24:13 2017 - *** uWSGI is running in multiple interpreter mode ***
uwsgi_1   | Wed Oct 11 15:24:13 2017 - spawned uWSGI master process (pid: 47)
uwsgi_1   | Wed Oct 11 15:24:13 2017 - spawned uWSGI worker 1 (pid: 51, cores: 1)
uwsgi_1   | Wed Oct 11 15:24:13 2017 - --- no python application found, check your startup logs for errors ---
uwsgi_1   | [pid: 51|app: -1|req: -1/1] 185.62.205.145 () {38 vars in 582 bytes} [Wed Oct 11 15:24:13 2017] GET / => generated 21 bytes in 0 msecs (HTTP/1.1 500) 2 headers in 83 bytes (0 switches on core 0)

Big sigh.

moeller@server2:~/sregistry$ docker-compose stop
Stopping sregistry_nginx_1 ... done
Stopping sregistry_uwsgi_1 ... done
Stopping sregistry_redis_1 ... done
Stopping sregistry_db_1 ... done

Hm.

$ sudo useradd vsoch
$ sudo mkdir /home/vsoch
$ chown vsoch.vsoch !$
$ sudo chown vsoch.vsoch /home/vsoch
$ sudo vim /etc/group
$ grep docker /etc/group
docker:x:999:moeller,vsoch

Dear @vsoch, If you do not mind too much, please send me public ssh key of yours and then have a look yourself.

vsoch commented 7 years ago

hey @smoe ! You named your file secret.py. It needs to be named secrets.py, per the instruction!

https://singularityhub.github.io/sregistry/install.html#secrets

The reason it isn't finding it is because it looks specifically for that file, and defaults to importing "bogus_secrets.py"

We will get this running, don't worry! You are doing great! :)

smoe commented 7 years ago

Wow! A classic one. Many thanks for spotting that. No sure if I deserve that praise, though. Now it is the "502 Bad Gateway" and

uwsgi_1   | No changes detected
uwsgi_1   | Operations to perform:
uwsgi_1   |   Apply all migrations: auth
uwsgi_1   | Running migrations:
uwsgi_1   |   Applying contenttypes.0001_initial... OK
uwsgi_1   |   Applying contenttypes.0002_remove_content_type_name... OK
uwsgi_1   |   Applying auth.0001_initial... OK
uwsgi_1   |   Applying auth.0002_alter_permission_name_max_length... OK
uwsgi_1   |   Applying auth.0003_alter_user_email_max_length... OK
uwsgi_1   |   Applying auth.0004_alter_user_username_opts... OK
uwsgi_1   |   Applying auth.0005_alter_user_last_login_null... OK
uwsgi_1   |   Applying auth.0006_require_contenttypes_0002... OK
uwsgi_1   |   Applying auth.0007_alter_validators_add_error_messages... OK
uwsgi_1   |   Applying auth.0008_alter_user_username_max_length... OK
uwsgi_1   | Traceback (most recent call last):
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
uwsgi_1   |     return self.cursor.execute(sql, params)
uwsgi_1   | psycopg2.ProgrammingError: relation "users_user" does not exist
uwsgi_1   | LINE 1: ...gree_terms", "users_user"."agree_terms_date" FROM "users_use...
uwsgi_1   |                                                              ^
uwsgi_1   |
uwsgi_1   |
uwsgi_1   | The above exception was the direct cause of the following exception:
uwsgi_1   |
uwsgi_1   | Traceback (most recent call last):
uwsgi_1   |   File "manage.py", line 8, in <module>
uwsgi_1   |     execute_from_command_line(sys.argv)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
uwsgi_1   |     utility.execute()
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/core/management/__init__.py", line 355, in execute
uwsgi_1   |     self.fetch_command(subcommand).run_from_argv(self.argv)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
uwsgi_1   |     self.execute(*args, **cmd_options)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
uwsgi_1   |     output = self.handle(*args, **options)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/core/management/commands/migrate.py", line 227, in handle
uwsgi_1   |     self.verbosity, self.interactive, connection.alias, apps=post_migrate_apps, plan=plan,
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/core/management/sql.py", line 53, in emit_post_migrate_signal
uwsgi_1   |     **kwargs
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/dispatch/dispatcher.py", line 193, in send
uwsgi_1   |     for receiver in self._live_receivers(sender)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/dispatch/dispatcher.py", line 193, in <listcomp>
uwsgi_1   |     for receiver in self._live_receivers(sender)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/guardian/management/__init__.py", line 33, in create_anonymous_user
uwsgi_1   |     User.objects.get(**lookup)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/db/models/manager.py", line 85, in manager_method
uwsgi_1   |     return getattr(self.get_queryset(), name)(*args, **kwargs)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 374, in get
uwsgi_1   |     num = len(clone)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 232, in __len__
uwsgi_1   |     self._fetch_all()
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 1105, in _fetch_all
uwsgi_1   |     self._result_cache = list(self._iterable_class(self))
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py", line 53, in __iter__
uwsgi_1   |     results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 886, in execute_sql
uwsgi_1   |     raise original_exception
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 876, in execute_sql
uwsgi_1   |     cursor.execute(sql, params)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
uwsgi_1   |     return self.cursor.execute(sql, params)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/db/utils.py", line 94, in __exit__
uwsgi_1   |     six.reraise(dj_exc_type, dj_exc_value, traceback)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
uwsgi_1   |     raise value.with_traceback(tb)
uwsgi_1   |   File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
uwsgi_1   |     return self.cursor.execute(sql, params)
uwsgi_1   | django.db.utils.ProgrammingError: relation "users_user" does not exist
uwsgi_1   | LINE 1: ...gree_terms", "users_user"."agree_terms_date" FROM "users_use...
uwsgi_1   |                                                              ^
uwsgi_1   |
uwsgi_1   | Migrations for 'users':
uwsgi_1   |   shub/apps/users/migrations/0001_initial.py
uwsgi_1   |     - Create model User
uwsgi_1   | Migrations for 'main':
uwsgi_1   |   shub/apps/main/migrations/0001_initial.py
uwsgi_1   |     - Create model Collection
uwsgi_1   |     - Create model Container
uwsgi_1   |     - Create model Demo
uwsgi_1   |     - Create model Label
uwsgi_1   |     - Create model Share
uwsgi_1   |     - Create model Star
uwsgi_1   |     - Alter unique_together for star (1 constraint(s))
uwsgi_1   |     - Alter unique_together for share (1 constraint(s))
uwsgi_1   |     - Alter unique_together for label (1 constraint(s))
uwsgi_1   |     - Alter unique_together for container (1 constraint(s))
uwsgi_1   | Migrations for 'api':
uwsgi_1   |   shub/apps/api/migrations/0001_initial.py
uwsgi_1   |     - Create model ImageFile
uwsgi_1   | Migrations for 'logs':
uwsgi_1   |   shub/apps/logs/migrations/0001_initial.py
uwsgi_1   |     - Create model APIRequestCount
uwsgi_1   |     - Create model APIRequestLog
uwsgi_1   | Operations to perform:
uwsgi_1   |   Apply all migrations: admin, api, auth, authtoken, contenttypes, djcelery, guardian, logs, main, sessions, social_django, taggit, users
uwsgi_1   | Running migrations:
uwsgi_1   |   Applying users.0001_initial... OK
vsoch commented 7 years ago

Actually I think you got it working :) When it first comes up and tries to do all the migrations, it's common to see that blurp, but then as you see below, it finishes the migrations. If the 502 persists ( try refreshing ) then just do a restart. I would bet if you refresh after you see the uwsgi running it will show up. Give it a try - we are close!

smoe commented 7 years ago

Indeed. Amazing. I leave it up for @pjotrp to toy around with it. For all review purposes I tend to think that together with your public instance this is just fine now. If I continue, then we soon have all possible and impossible images gathering various tools for biomedical workflows in Debian Med. And I need to do some other work :-7

I am right in my understanding that you do not need to bootstrap from Docker but that regular Singularity images are just as fine as a start, right? Is there any significant difference in performance when you start from either? Or from a security perspective - it is the root permissions that you need to bootstrap/build/change anything that SingularityHub takes from us since those images are rightly executable, right? That point is trivial but maybe you find a place for it.

vsoch commented 7 years ago

@smoe great to hear you got it working! You are correct that Singularity does not have a dependency on Docker. A lot of work (e.g., biocontainers) has gone into Docker, so an early addition was to ensure that a Docker container could be imported seamlessly into Singularity. We have several other bases, including the mirror urls for each distribution (debootstrap, arch linux, and more here). If you want to get your hands into Singularity I would wait a few days - we are about to release 2.4 and the building commands are a bit more intuitive than the previous.

For security, Docker has huge issues. Installation on a shared resource is a non starter. This is why Singularity has taken the HPC world by storm. It does not carry these same issues. As for performance, there have been a few papers that have compared Docker vs Singularity (if you do a Google search a few blog posts will come up) and (also in my experience) Singularity is a bit closer to "bare metal." It also is more seamlessly integrated into the environment, unlike Docker that is isolated.

But I digress! The Singularity software proper is separate from a Registry to serve it's images (Singularity Registry). This is also different from Singularity Hub (which is a build service). I hope I made these distinctions clear in the documentation, please let me know if you would want to see more.

This (I think) has been the most satisfying review I've ever done! To actually have the reviewers test and give feedback on setup, install, and documentation, is fantastic.

smoe commented 7 years ago

Oh, too late, I had my first hands-on experience with Singularity somewhen in July and liked it a lot. It is basically what saves Debian Med in the HPC world where otherwise everyone is much after BioConda.

Please let us digress a bit more - just a tiny, tiny bit. On the Debian Med mailing list, Mr. CWL (Michael Crusoe) just asked about an automated Docker image build service for Debian packages for the biocontainers to adopt when there is no BioConda package for it (https://lists.debian.org/debian-med/2017/10/msg00029.html). I personally think that we should have Debian Med autobuilt for Singularity in some reasonable manner - "reasonable" as in not having a separate image for every tiny package. And if I get this all right, then Singularity Hub could do that for us?!? Is there something that you feel like replying to Michael on the Debian Med list? Should we not somehow investigate how much packages share in terms of runtime dependencies and use that information together with workflow information to decide what images to build?

Thank you for your kind words about your reviewing experience with me. I tend to think I have been luckier with my reviews "on your side of the review" in the past. But I certainly never had anyone so active and positive to direct me towards getting bits and pieces to run like you, @vsoch . Many thanks for that. Please contact me for anything I can help with on the Debian front of yours.

vsoch commented 7 years ago

Oh, too late, I had my first hands-on experience with Singularity somewhen in July and liked it a lot. It is basically what saves Debian Med in the HPC world where otherwise everyone is much after BioConda.

Don't worry - most of it is the same! We are adding a lot more security to the image format that is built, and a single interface for interacting with it (basically a command called build instead of bootstrap) so the sampling folk from statistics can have their term back :) There will be ample documentation and examples to get you re-acquainted.

Please let us digress a bit more - just a tiny, tiny bit. On the Debian Med mailing list, Mr. CWL (Michael Crusoe) just asked about an automated Docker image build service for Debian packages for the biocontainers to adopt when there is no BioConda package for it (https://lists.debian.org/debian-med/2017/10/msg00029.html).

Just taking a look, this would work easily with Singularity Registry (if someone wants to host) or the (soon to be released) Singularity hub. WIth Singularity 2.4 you have two options for new bases - either another local Singularity image file:

Bootstrap: localimage
From:  my-container-base.simg

or if it's hosted at Singularity Hub / Registry, the same image from there:

Bootstrap: localimage
From:  shub://library/debian

and so the entire build recipe would look like this:

Bootstrap: localimage
From:  shub://library/debian

%post
apt-get update && apt-get install -y <special-software>

and unlike Docker, if you use a singularity image as a base, it isn't assembled from layers. It's an image. With Singularity Hub 2.0, the managers can even freeze the images, which is a guarantee that it cannot change if you specify a specific version.

I personally think that we should have Debian Med autobuilt for Singularity in some reasonable manner - "reasonable" as in not having a separate image for every tiny package.

As an academic turned software engineer, I'm all for using free services with minimal extra. Here I would advocate for using Github for recipes, and then Singularity Hub to build the images (the next release).

And if I get this all right, then Singularity Hub could do that for us?!? Is there something that you feel like replying to Michael on the Debian Med list?

For sure! But maybe let me finish up this version 2.0 first - I have at most a week or two to getting everything just right and tested. It's actually up and running now at a secret url :)

Should we not somehow investigate how much packages share in terms of runtime dependencies and use that information together with workflow information to decide what images to build?

oh, would you be interested in doing this sort of formally? It's exactly the kind of "analysis" that I love - I wrote an entire paper just on similarity metrics for containers that is based on comparing contents. One of the reasons I am under the gun for Singularity Hub is that I got a very good review back, but don't want to move forward until I make the next version as flawless as I can.

Thank you for your kind words about your reviewing experience with me. I tend to think I have been luckier with my reviews "on your side of the review" in the past. But I certainly never had anyone so active and positive to direct me towards getting bits and pieces to run like you, @vsoch . Many thanks for that. Please contact me for anything I can help with on the Debian front of yours.

Awesome!! I take it you are one of the debian gurus? I've used a debian flavor (ubuntu) since 2009 and never looked back :)

smoe commented 7 years ago

I am one of the Debian Developers (moeller@debian.org), but there are more guru-ish folks out there that are non-developers :) For the formal bits, one could tap into the Unified Debian Database (udd.debian.org) that has a table on packages and its dependencies. So this would create us a directed acyclic (should be) graph to annotate with sizes and do some clustering in. I can help on that, but would uttermost happily leave all the lead on that to you.

We are currently adding tags to Debian packages to link them to OMICtools, bio.tools and the SciCrunch RRIDs. It looks like this https://anonscm.debian.org/cgit/debian-med/bowtie2.git/tree/debian/upstream/metadata . This way we could assign references to Singularity images, too.

People you may be interested to contact are the ones from snapshot.debian.org. It provides legacy packages - you basically give it a date and it finds you all the packages you want in the version they were available at that date. You get a dynamic apt repository from which you could debootstrap your images. This should be helpful for reproductive sciene or for investigating the differences that advances of software bring. I do not know if this would be used often or if this only sounds nice to have.

vsoch commented 7 years ago

It's so great to make your acquaintance! I think references to Singularity images is a fantastic idea. I am definitely going to check out these resources, and @gmk @godlovedc and @bauerm97 see the discussion above!

smoe commented 7 years ago

That thread on the Debian Med was rather fruitful - the NeuroDebian folks created a singularity image, see https://lists.debian.org/debian-med/2017/10/msg00037.html .

pjotrp commented 7 years ago

@smoe I gather you are happy as a reviewer and completed review. Great work both of you! @arfon we are ready to R&R!

vsoch commented 7 years ago

Fantastic! Thanks @pjotrp @smoe and @arfon! :)