search

jts / ncov-tools

Small collection of tools for performing quality control on coronavirus sequencing data and genomes
MIT License
47 stars 16 forks source link

Singularity containerisation to enable use in conda-limited environments #49

Closed fmaguire closed 3 years ago

fmaguire commented 3 years ago

HPC/Compute Canada doesn't support conda so running the workflow is non-trivial.

They can use singularity so the options would be:

  1. Create a container that has all the dependencies installed and have host install of snakemake executing rules in that environment (easier but less portable)
  2. Create a container encompassing the entire tool (more portable but more complex).

Once done, this build can be triggered on a new release in github actions e.g. https://github.com/jaleezyy/covid-19-signal/blob/master/.github/workflows/dockerpublish.yml

  1. is probably the better choice using the conda docker container as a base image e.g. singularity build of https://github.com/jaleezyy/covid-19-signal/blob/master/resources/Dockerfile_pipeline installing mamba on this and then just scripting the README install instructions SHOULD work although the nested pangolin workflow might be a slight tripping point
poquirion commented 3 years ago

Hello, I am with the C3G and Compute Canada.

A reasonable solution would be to have two containers, one with the dependencies, with fixed versions of the tools minus snakemake, and one with the ncov-tools scripts that depend on the previous container. Having a version number for all the tools in the environment file could also speed up the build speed and help reproducibility (https://www.anaconda.com/blog/understanding-and-improving-condas-performance)

The goal for us will be to run the pipeline using snakemake --use-singularity --use-conda paradigm, where snakemake is installed on the host (not in a container).

I can fork the repo and propose something.

jts commented 3 years ago

Thanks @poquirion, I'd be happy to take a look if you want to fork and propose some changes. You may wish to have a look at issue #47 as we're discussing some other changes to the workflow that may effect the environments.

jts commented 3 years ago

@poquirion just a head's up that I merged the first stage of the refactor - I suggest you update your fork before starting on the singularity changes

jts commented 3 years ago

Resolved by @poquirion