Provide Dockerfile & prebuilt/automatically built container image

[Masterxilo]

This software should be available in an automatically build container image with all dependencies already installed and ready to run with a well defined interface.

The container image should be available via quay or docker hub.

This was done manually at some point but the automatic pipeline has not been established yet: https://quay.io/repository/dryak/v-pipe?tab=tags

[Masterxilo]

Or the sofware could be made available like bwa on anaconda cloud and bioshadock:

https://anaconda.org/bioconda/bwa https://docker-ui.genouest.org/app/#/container/bioconda/bwa

[Masterxilo]

more modern: https://biocontainers.pro/#/

[sposadac]

In general it is a good idea. However, we support distribution with rule-specific conda environments, which we find to be suitable for HPC. A containerised solution could help users to get started, but for most applications, it might fall short (e.g. sponsoring containers within containers is not trivial). For cloud computing, it could prove useful, but such solution would depend on the particular cloud provider.

[DrYak]

Hi Paul, hope you're going well.

If you plan to use V-pipe further in your work, we could flesh out better the first draft solution that you mention we did a long ago. We should discuss your exact needs so we get a good idea how to best serve them.

(Giant "katamari"-style docker wit everything? (i.e.: with the environment of every last single snakemake rule included) Specific dockers for different uses? (Different docker, each with only a select collection of rules' environment specific for specific use-cases ?), etc.)

Regarding a "docker-in-docker" possibility, I would also second Susana's opinion: while doable, it's not a very stable and solid construction.

Regarding bioconda/biocontainers: currently every single component we use has containers automatically produced out of the conda recipes (e.g.: ShoRAH ). But as Susana mentioned, currently V-pipe is designed to use the "per-rule environment" mode of snakemake, so these aren't directly usable.

The "pre-install rules' environment inside a single Docker" approach we used in the draft seems to me to be the most sensible.

[Masterxilo]

Hey Ivan, nice to hear from you.

I think the "all dependencies in one container, one-size-fits all" solution is the best as you say. There should just be an automatic pipeline somewhere which automatically builds the latest version of this container from this repository.

Dockerhub does it for free and circleci and travisci and other automatic build tools are also free on public github projects like this, so it should be doable with no extra cost.

I will use the existing old container until then, but this would be a nice and very useful improvement to this pipeline.

[DrYak]

Not quite Dockerfile yet, but the new SARS-CoV-2 version comes with an installer, that can:

• pre populate a working directory distinct from the V-pipe source
• has a vpipe.conf which is preset for a specific pipeline
• and in the tutorial there are instruction how to pre-install all the dependencies

Basically all the pieces are there to make the "sars-cov2" branch into a ready-to-use Docker. Next step I am attacking on: CI/CD. That should help use automate the Docker generation, at least for branches that have a clear set of default config (e.g.: for SARS-CoV-2, that is bwa aligner, shorah caller).

[DrYak]

Hi! Long time no see...

As of master commit 2497d0aa4b23e95e3a4b5b65cbc38a0917da6e6d , the support for docker is finally here.

• Since #102 we have a new more modern configuration system, that supports per-virus base configurations. The master branch is fully agnostic, but has base configuration for HIV (used to be testdata for HIV in older version) and SARS-CoV-2 (adapted from the sars-cov-2 branch), and provides the necessary files (References, gene definitions, etc.) as standard snakemake workflow resources.
• The Docker is populated by downloading the conda env for all the standard steps of the provided viruses' base configs. ( we have both bwa and ngshmmalign as aligners, only predicthaplo is provided as haplotype engine).
• This resulting Docker is usable as-is offline for most common tasks, or would require an internet connection to fetch additionnal component for corner cases not covered by the base configs.

• The whole Docker creation is automated, and a new up-to-date is create for each new push to master and for future tag releases. Check the package section of the repository.

  A big thank to @uweschmitt from SIS who helped setup the CI/CD workflows and some of the Docker work, and @kpj who helpted some more with Docker workflow automation.

Hope that this helps you!

[Masterxilo]

Nice @DrYak great job. We will look into it, we are still interested in setting up vpipe as one of our ngs processing pipelines.