best way to run rabix/bunny on HPC?

[davidroberson]

Is there some magic to get rabix to work in an environment hostile to docker? Should we run bunny inside a singularity container or is it possible to bootstrap the docker images inside many singularity steps?

Thank you.

[milos-ljubinkovic]

config/core.properties file has a field executor.override.command where you can specify a template for the command you want to be used to execute individual steps.

So you can set it to:

executor.override.command=singularity exec {{#volumes}}--bind {{path}}:{{location}} {{/volumes}} --pwd {{workingDir}} docker://{{image}} /bin/sh -c '{{#escape}}{{command}}{{/escape}}'

in order to just exec the steps' docker image using Singularity.

This kinda works, but if you need environmental variables from cwl you need to expand it into something like this:

executor.override.command=echo \"Bootstrap: docker\nFrom: {{image}}\n\n%runscript\n\n {{#escape}}{{command}}{{/escape}}\n\n%environment\n {{#env}}\n export {{key}}={{#escape}}{{value}}{{/escape}};{{/env}}\" > /tmp/script-{{jobId}} && sudo singularity build ./container-{{jobId}} /tmp/script-{{jobId}} && singularity run {{#volumes}}--bind {{path}}:{{location}} {{/volumes}} --pwd {{workingDir}} ./container-{{jobId}}

Where a singularity build file is created and built with all the needed info.

You can also write templates to use uDocker (no-sudo replacement for dockerd) instead of singularity or prefix the commands with your HPC's submit commands.

ex: executor.override.command=srun -N1 -l singularity exec {{#volu....

I've tested this only on local vm installations of Slurm and LSF with singularity and uDocker so I'd be interested in hearing if you got it to work in an actual working env.

[davidroberson]

great. We are trying to transfer a workflow to another collaborator and use their HPC. I may loop them into to this thread.

[davidroberson]

Will rabix/bunny work inside a singularity container if all the dependencies for the cwl application are present and we use a "no docker" flag?

That might be the easiest way to package something for HPC.

[milos-ljubinkovic]

It should probably work but I don't think I tried the no-docker flag recently, might be some issues.

[davidroberson]

was able to get rabix/bunny working inside a docker container with "no container" flag.

Having problems running it inside the same bootstrapped from docker singularity container.

[davidroberson]

dna2protein example is working

Singularity cgrlab-typeseqer:latest.simg:~/rabix-cli-1.0.4/examples/dna2protein> ../../rabix --no-container dna2protein.cwl.json i>
[2018-03-12 02:25:01.448] [INFO] Job root.Transcribe has started
[2018-03-12 02:25:01.480] [INFO] Local execution (no container) has started
[2018-03-12 02:25:01.572] [INFO] Running command line: python transcribe_argparse.py -d /home/ubuntu/rabix-cli-1.0.4/examples/dna2protein/data/input.txt --verbose > /home/ubuntu/rabix-cli-1.0.4/examples/dna2protein/dna2protein.cwl-2018-03-12-022459.833/root/Transcribe/rna.txt
[2018-03-12 02:25:02.596] [INFO] Job root.Transcribe has completed
[2018-03-12 02:25:02.698] [INFO] Job root.Translate has started
[2018-03-12 02:25:02.710] [INFO] Local execution (no container) has started
[2018-03-12 02:25:02.718] [INFO] Running command line: python translate.py -r /home/ubuntu/rabix-cli-1.0.4/examples/dna2protein/dna2protein.cwl-2018-03-12-022459.833/root/Transcribe/rna.txt > /home/ubuntu/rabix-cli-1.0.4/examples/dna2protein/dna2protein.cwl-2018-03-12-022459.833/root/Translate/protein.txt
[2018-03-12 02:25:03.732] [INFO] Job root.Translate has completed
{
  "output_protein" : {
    "basename" : "protein.txt",
    "checksum" : "sha1$55adf0ec2ecc6aee57a774d48216ac5a97d6e5ba",
    "class" : "File",
    "contents" : null,
    "dirname" : "/home/ubuntu/rabix-cli-1.0.4/examples/dna2protein/dna2protein.cwl-2018-03-12-022459.833/root/Translate",
    "format" : null,
    "location" : "file:///home/ubuntu/rabix-cli-1.0.4/examples/dna2protein/dna2protein.cwl-2018-03-12-022459.833/root/Translate/protein.txt",
    "metadata" : null,
    "nameext" : ".txt",
    "nameroot" : "protein",
    "path" : "/home/ubuntu/rabix-cli-1.0.4/examples/dna2protein/dna2protein.cwl-2018-03-12-022459.833/root/Translate/protein.txt",
    "secondaryFiles" : [ ],
    "size" : 9
  }
}

[davidroberson]

wait...i think it is working... just weird file permission issues..

[davidroberson]

closing this issue because using

rabix --no-container

inside a singularity container is completely working for me and even is doing all the scatter / gather correctly..etc etc.

I like this solution better because now the only requirement is singularity. Java 8, Rabix/bunny, and cwl file are inside the container. We could still pipe in a new cwl file for a quick correction of a script.

Thank you