c3se problems

[JohnHellgren]

Hello,

I noticed that the 1.0 release is out, but I cannot download the singularity container for this version

I tired this command:

singularity pull --name nfcore-rnaseq-1.0.img docker://nfcore/rnaseq:1.0

But the I got this

WARNING: pull for Docker Hub is not guaranteed to produce the
WARNING: same image on repeated pull. Use Singularity Registry
WARNING: (shub://) to pull exactly equivalent images.
Docker image path: index.docker.io/nfcore/rnaseq:1.0
ERROR MANIFEST_UNKNOWN: manifest unknown
Cleaning up...
ERROR: pulling container failed!

running the pipeline without pulling the container first doesnt work for c3se

Caused by:
  Failed to pull singularity image
  command: singularity pull --name nfcore-rnaseq-1.0.img docker://nfcore/rnaseq:1.0 > /dev/null
  status : 1
  message:
    WARNING: pull for Docker Hub is not guaranteed to produce the
   WARNING: same image on repeated pull. Use Singularity Registry
    WARNING: (shub://) to pull exactly equivalent images.
 ERROR MANIFEST_UNKNOWN: manifest unknown
    ERROR: pulling container failed!

I ran the pipeline like this: sbatch rna_seq_script.sh rna_seq_script.sh

Best, John

[apeltzer]

Hi John, we had some issues with building on Docker Hub and I had to set up Docker Cloud to get this to run. I will have a look and push a 1.0 image in a couple of minutes.

[apeltzer]

Started pushing the image, this will take a while to finish.

[apeltzer]

You should be able to get the image now - let me know if otherwise, I will close this now!

[ewels]

Yes, apologies for this - we have the docker build setup automated so I expected it to magically appear in the night after the release. We only noticed this morning that it hadn't triggered due to a confusing mix of docker web infrastructures. All fixed (and hopefully future-proofed) now! Let us know if you run into any problems.

[JohnHellgren]

Thanks

I still have the bind point issue

Command error:
  WARNING: Non existent bind point (directory) in container: '/c3se'
  WARNING: Non existent bind point (directory) in container: '/apps'
  WARNING: Non existent bind point (directory) in container: '/usr/share/lmod/lmod'
  WARNING: Non existent bind point (directory) in container: '/var/hasplm'
  WARNING: Non existent bind point (directory) in container: '/local'
  WARNING: Could not chdir to home: /c3se/users/johnhe/Hebbe

Do I have to add it manually this time as well? Last time I solved it by asking the support at our computer cluster Do you have a link to the singularity container? Like with the last pipeline: https://hub.docker.com/r/scilifelab/ngi-rnaseq/builds/

[ewels]

Hi John,

Yes, if you can't enable overlays then you will need to manually add those bind points for every new singularity image that you want to use.

x-ref: https://github.com/SciLifeLab/NGI-RNAseq/issues/174#issuecomment-386239084 and https://github.com/SciLifeLab/NGI-RNAseq/issues/174#issuecomment-386571706

The nfcore/rnaseq v1.0 tagged docker container is visible here: https://cloud.docker.com/swarm/nfcore/repository/docker/nfcore/rnaseq/tags

So your Singularity bootstrap file should look something like this:

Bootstrap: docker
From: nfcore/rnaseq:1.0
%post
    mkdir ${SINGULARITY_ROOTFS}/c3se
    mkdir ${SINGULARITY_ROOTFS}/local
    mkdir ${SINGULARITY_ROOTFS}/apps
    mkdir ${SINGULARITY_ROOTFS}/usr/share/lmod/lmod
    mkdir ${SINGULARITY_ROOTFS}/var/hasplm
    mkdir ${SINGULARITY_ROOTFS}/var/opt/thinlinc

$ singularity build ngi-rnaseq-hebbe.simg Singularity

Note that we're aware that this process is a pain and we have been actively trying to develop a helper tool to make this process easier, see https://github.com/nf-core/tools/issues/69 - hopefully the next time there's a release we'll have a helper tool to automate this process for you.

Phil

[ewels]

Hi @JohnHellgren,

@remiolsen also suggested trying to use the -B option with Singularity. However, we're not sure if this works or not on systems were overlayFS is not enabled. We don't have access to any such systems, so this is tough for us to test!

If you're interested, it would be great if you could give it a try...

1) Create a nextflow config file, eg. singularity-c3se.config.txt:

    singularity {
      runOptions = '-B /c3se -B /local -B /apps -B /usr/share/lmod/lmod -B /var/hasplm -B /var/opt/thinlinc'
    }

2) Run the pipeline with the original Singularity container, but specifying the above config:

    nextflow run nf-core/rnaseq -c singularity-c3se.config.txt ....

* Modified launch script: [rna_seq_script.txt](https://github.com/nf-core/rnaseq/files/2313892/rna_seq_script.txt) (requires you to create `/c3se/NOBACKUP/users/johnhe/rnaseq/nf-core/config/singularity-c3se.config.txt` from above).

If this does work, then there's no need for the creation of a custom Singularity container, which would be much better. We can also add all of this to the hebbe profile in the pipeline, so you won't need to specify it in the future (and it will work for all hebbe users).

Let me know what you think!

Cheers,

Phil