Singularity build

[suujia]

shub:bcgsc/orca shub:suujia/orca

Not sure how to trigger a rebuild for the bcgsc/orca shub. The error message I got with a build on suujia/orca shub was:

+ su -c brew install perl -v linuxbrew
Error: /home/linuxbrew/.linuxbrew/Homebrew is not writable. You should change the
ownership and permissions of /home/linuxbrew/.linuxbrew/Homebrew back to your
user account:
sudo chown -R $(whoami) /home/linuxbrew/.linuxbrew/Homebrew
Error: Cannot write to /home/linuxbrew/.linuxbrew
ABORT: Aborting with RETVAL=255

I think I'll start with getting it to build locally first and updating singularity file to align with orca dockerfile.

[suujia]

I was able to use overlay to install nano (apt-get) and it's only available within this overlay.

sudo singularity shell --overlay my-overlay.img orca.simg

However, when installing anything with brew inside (using eg brew install nextflow):

Error: Running Homebrew as root is extremely dangerous and no longer supported.
As Homebrew does not drop privileges on installation you would be giving all
build scripts full access to your system.

[sjackman]

Excellent! That's good progress. The goal with using Singularity is that it shouldn't require root access for the user to run the image. It does I believe require root access to build the original Singularity image, but hopefully not for the user. Can you it without sudo?

[suujia]

singularity shell --overlay my-overlay.img orca.simg

Yes, after running this command, in directory where I don't have sudo access on the outside, I was able to brew install __ things into the overlay image. This was with a small orca.simg I built. I am still debugging to build the whole orca singularity image. After it's built, I can start writing a script that sets up the user, most importantly: singularity image.create --size 3000 $USER_overlay.simg, (note that our Singularity file is prebuilt into orca.simg, which can be run with no setup time)

Users would just have to:

ssh orca01
singularity shell --overlay $USER_overlay.simg orca.simg

Users not affiliated with bcgsc can just clone singularity locally, and then:

singularity pull shub://bcgsc/orca

====== current build: https://www.singularity-hub.org/containers/3403/log

[sjackman]

Excellent progress! Thanks, Susie.

singularity image.create --size 3000 $USER_overlay.simg

I'd suggest storing the overlay at $HOME/.orca/overlay.simg

[sjackman]

+ su -c brew update linuxbrew
bash: brew: command not found

https://www.singularity-hub.org/containers/3403/log

See if you're able to remove su from this command, and instead set up the permissions so that the user has write access to /home/linuxbrew/.linuxbrew. You can do that by making the directory owned by user linuxbrew and group linuxbrew (also written linuxbrew:linuxbrew or for short linuxbrew:) and making sure the directories and files have group write permissions (chmod -R g+w /home/linuxbrew/.linuxbrew). An alternative would be to make the /home/linuxbrew/.linuxbrew directory structure owned by the user rather than by linuxbrew, such as sjackman:linuxbrew so that group write permissions are not needed. I have a preference for linuxbrew:linuxbrew if it can work.

[suujia]

Brew can't be run as sudo (due to error above) which is why it's ran as linuxbrew user:

su -c 'brew update' linuxbrew

I have added chown -R linuxbrew: /home/linuxbrew (I can change it later to add .linuxbrew), and it is rebuilding to same link. It is still building currently. I was running into the problem of individual sub directories requiring a chmod earlier. If it doesn't pass I will try with a chmod.

[sjackman]

Brew can't be run as sudo (due to error above) which is why it's ran as linuxbrew user:

Ah, got it. Can the whole script be run as the Linuxbrew user rather than root? Or separate it into two parts, the root part and then then linuxbrew part, and switch users between those two parts. See here for an example in a Dockerfile. I'm not sure how this works in Singularity.

https://github.com/Linuxbrew/brew/blob/master/Dockerfile#L18

[suujia]

Yes, do linuxbrew user things in su -c '__' linuxbrew and just run the root commands outside with no wrappers. We are already doing that in this our Singularity file, with the brew commands run as linuxbrew user and everything else without it. Which other commands did you want to run under sudo?

This is a really good example: UofU bioBakery

edit: oh do you mean change the permissions back after the installs? I believe that can be done.

[sjackman]

Excellent find with the bioBakery Singularity file! That's a very useful resource.

su -c '…' linuxbrew is fine. I misunderstood what was going on.

Putting the option at the end may be more readable.

su linuxbrew -c '…'

[suujia]

Bind Mounts:

Map directories on host system to directories within container (read/write data on host system). The following should be added to config file of singularity:

ENABLE OVERLAY (boolean, default=’no’)  --  set to yes 
USER BIND CONTROL (boolean, default=’yes’) -- already yes 

==================== pip3 install ruamel.yaml fixes:

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-kobo8rt4/ruamel.yaml/

cwlref-runner \ here

cwltool 1.0.20180711112827 has requirement ruamel.yaml<0.15,>=0.12.4, but you'll have ruamel-yaml 0.15.44 which is incompatible.

and

    pysam/libchtslib.c:15077:12: error: too many arguments to function ‘(PyObject * (*)(PyObject *, PyObject * const*, Py_ssize_t))meth’
         return (*((__Pyx_PyCFunctionFast)meth)) (self, args, nargs, NULL);

r probems:

ARGUMENT '~+~~+~~+~~+~"ggplot2",~+~\' __ignored__

ARGUMENT '~+~~+~~+~~+~"knitr",~+~\' __ignored__

ARGUMENT '~+~~+~~+~~+~"rmarkdown",~+~\' __ignored__

ARGUMENT '~+~~+~~+~~+~"tidyverse"));~+~\' __ignored__

ARGUMENT '~+~~+~~+~~+~source("https://bioconductor.org/biocLite.R");~+~biocLite()' __ignored__

Error: unexpected input in "install.packages(repos = c(CRAN = "https://cran.rstudio.com"), c( \"
Execution halted
ABORT: Aborting with RETVAL=255

[suujia]

+ pip2 install biopython python-igraph
Failed building wheel for python-igraph

Try with easy_install because it doesn't use wheel

Other error

+ pip3 install biopython
Cache entry deserialization failed, entry ignored

+ pip3 install pysam
Failed building wheel for pysam
Command "/home/linuxbrew/.linuxbrew/opt/python/bin/python3.7 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-7t4j166i/pysam/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('

+ pip3 install python-igraph
Failed building wheel for python-igraph

[suujia]

$PREFIX/etc/ singularity/singularity.conf.(owned by root) should have following configurations set up. These configurations should be set up to control all the users using singularity command. However, these configs do not apply for root.

allow container squashfs = yes      # read only 
allow container extfs = no               # writable container
allow container dir = no                   # sandbox container - chroot  

limit container owners = gmk  # only use containers owned by user

limit container paths = /scratch, /tmp     # only able to use containers in this path

enable overlay = yes        # default is no, bind points within containers 

bind path = /opt/singularity/orca.simg      # bind points extended from host system into the container

[suujia]

Documentation

I will be updating singularity_manual with proper instructions.

To put files in the container, use ${SINGULARITY_ROOTFS}.
E.g. to put files to container's /usr/local, put it to ${SINGULARITY_ROOTFS}/usr/local

Permissions

The user has the same permission level inside the container as they are outside. Therefore within the container, users are not able to escalate their privileges if they were not sudo outside.

Workflow

$ singularity pull --name orca.simg shub://suujia/orca:latest
$ singularity shell --overlay overlay.img orca.simg

Singularity orca.simg:~> java --version
java 9.0.4
Java(TM) SE Runtime Environment (build 9.0.4+11)

Singularity orca.simg:~> brew install nextflow 

Sidenote

It is much easier to test singularity hub through making a docker build on dockerhub. Dockerhub runs the full file and tells you which point of the file produces the errors, whereas singularityhub exits the build as soon as it comes upon one error, therefor only ever producing one error message. It takes much longer to debug this way.

[suujia]

Here's the progress so far for singularityhub. This build has around 300 tools, takes a long time to build but I'm still slowly adding tools to it. I removed the biggest tools to speed up the builds and will add it back later, as well as simplify and clean it up. (Also I will change linuxbrew/.linuxbrew writable actually) https://www.singularity-hub.org/containers/3905

Just need to run

1. singularity pull --name orca.simg shub://suujia/orca
   • this step takes about 3-5 minutes
2. singularity image.create --size 3000 myoverlay.img
   • takes ~20 seconds
3. singularity shell --overlay myoverlay.img orca.simg
   • immediate. launches you in the shell

[sjackman]

That's excellent progress! Thanks, Susie. Can you try making a version of https://github.com/bcgsc/orca/blob/master/scripts/orca that uses Singularity rather than Docker? Once you're happy with the image, we'll need to move it from shub://suujia/orca to shub://bcgsc/orca.

[suujia]

Final Update

Brendan fixed some of the permissions and we ended up getting singularity fully on the orca servers fully. It is on permissions 777 because the shub was installing as linuxbrew user and not linuxbrew group. And we can use sudo commands at all for singularity therefore we can't change the identity of user to linuxbrew user to perform the brew install (and we removed the sudo brew install alias). However, the individuals are only modifying/installing things in their own overlay, in their own home directory, which should not be a problem giving the image permissions 777. (You can double check).

All the tools installed in the big singularity image works. Just need to do a:

singularity pull shub://bcgsc/orca         (add @___COMMIT ID________)

into where you'd like the base image to be stored. The thing about singularity hub is that every rebuild overrides the previous, unless you freeze it (similar to tagging it). You can ensure you get that version, even if its same branch, by adding a commit id to the pull.

Oh the current singularity login script doesn't include the bind (for group folders) that Brendan wanted to add, too.

Let me know if you had any more questions or need help, I will be on github :)

[sjackman]

Excellent. Thanks, Susie!

The current build on Singularity Hub says that it has been running since Tue Aug 28 (one full week). So I'm guessing that something is not quite right there. I'd prefer to build the image on Singularity Hub (or a hosted service), but if that's not possible, we can build it locally and upload the image. See https://www.singularity-hub.org/collections/1223