search

EBI-Metagenomics / genomes-catalogue-pipeline

MGnify genome analysis pipeline
Other
101 stars 21 forks source link

Fetch singularity images for offline use #76

Closed lam-c closed 10 months ago

lam-c commented 11 months ago

Hi, thank you for sharing us this great pipeline! I wonder if there any solution for fetching all the images needed before executing the pipeline?

I've tried with nf-core download (which may not a proper use for this project), and it has ended up with errors below. Look forward for your response.

$ nf-core download --revision v2.2.0 --outdir genomes-pipeline_v2.2.0 -s singularity EBI-Metagenomics/genomes-pipeline
INFO    Process workflow revision v2.2.0, found 24 container images in total
Pulling singularity images
ERROR   Please try to rerun 
            "Singularity pull --name ..../quay.io-biocontainers-bracken-2.8--py...img quay.io-biocontainers-bracken-2.8--py...img" manually with a different registry.f

Software & system information:

System: Red Hat 8.5.0-10
Singularity-ce: version 3.10.2-1.el8
Nextflow: version 23.10.0 build 5889
nf-core: version 2.10
KateSakharova commented 11 months ago

Hi @lam-c, have you tried launching the pipeline? All containers should be pulled automatically during launching/running pipeline steps.

Best, Kate

mberacochea commented 11 months ago

Hi, thank you for sharing us this great pipeline! I wonder if there any solution for fetching all the images needed before executing the pipeline?

I've tried with nf-core download (which may not a proper use for this project), and it has ended up with errors below. Look forward for your response.

$ nf-core download --revision v2.2.0 --outdir genomes-pipeline_v2.2.0 -s singularity EBI-Metagenomics/genomes-pipeline
INFO    Process workflow revision v2.2.0, found 24 container images in total
Pulling singularity images
ERROR   Please try to rerun 
            "Singularity pull --name ..../quay.io-biocontainers-bracken-2.8--py...img quay.io-biocontainers-bracken-2.8--py...img" manually with a different registry.f

Software & system information:

System: Red Hat 8.5.0-10
Singularity-ce: version 3.10.2-1.el8
Nextflow: version 23.10.0 build 5889
nf-core: version 2.10

This pipeline doesn't follow the nf-core conventions, so the nf-core tool won't work with it.

But I was able to pull all the containers using nextflow inspect, you can create a small script to pull the contains from this file.

{
    "processes": [
        {
            "name": "GAP:PREPARE_DATA:FILTER_QS50",
            "container": "quay.io/microbiome-informatics/genomes-pipeline.python3base:v1.1"
        },
        {
            "name": "GAP:PREPARE_DATA:RENAME_FASTA",
            "container": "quay.io/microbiome-informatics/genomes-pipeline.python3base:v1.1"
        },
        {
            "name": "GAP:GENOME_SUMMARY_JSON",
            "container": "quay.io/microbiome-informatics/genomes-pipeline.python3base:v1.1"
        },
        {
            "name": "GAP:KRAKEN_SWF:BRACKEN_150",
            "container": "quay.io/biocontainers/bracken:2.8--py310h30d9df9_0"
        },
        {
            "name": "GAP:KRAKEN_SWF:BRACKEN_250",
            "container": "quay.io/biocontainers/bracken:2.8--py310h30d9df9_0"
        },
        {
            "name": "GAP:PROCESS_MANY_GENOMES:PANAROO",
            "container": "quay.io/biocontainers/panaroo:1.3.2--pyhdfd78af_0"
        },
        {
            "name": "GAP:KRAKEN_SWF:KRAKEN2_POSTPROCESSING",
            "container": null
        },
        {
            "name": "GAP:ANNOTATE:EGGNOG_MAPPER_ANNOTATIONS",
            "container": "quay.io/microbiome-informatics/genomes-pipeline.eggnog-mapper:v2.1.11"
        },
        {
            "name": "GAP:GTDBTK_AND_METADATA:METADATA_TABLE",
            "container": "quay.io/microbiome-informatics/genomes-pipeline.python3base:v1.1"
        },
        {
            "name": "GAP:ANNOTATE:DETECT_RRNA",
            "container": "quay.io/microbiome-informatics/genomes-pipeline.detect_rrna:v3.1"
        },
        {
            "name": "GAP:FUNCTIONAL_ANNOTATION_SUMMARY",
            "container": "quay.io/microbiome-informatics/genomes-pipeline.python3base:v1.1"
        },
        {
            "name": "GAP:KRAKEN_SWF:KRAKEN2_BUILD",
            "container": "quay.io/biocontainers/kraken2:2.1.2--pl5321h9f5acd7_2"
        },
        {
            "name": "GAP:ANNOTATE:SANNTIS",
            "container": "quay.io/microbiome-informatics/sanntis:0.9.3.2"
        },
        {
            "name": "GAP:DETECT_NCRNA",
            "container": "quay.io/microbiome-informatics/genomes-pipeline.detect_rrna:v3"
        },
        {
            "name": "GAP:MMSEQ_SWF:MMSEQ_100",
            "container": "quay.io/biocontainers/mmseqs2:13.45111--h2d02072_0"
        },
        {
            "name": "GAP:DREP_SWF:SPLIT_DREP",
            "container": "quay.io/microbiome-informatics/genomes-pipeline.python3base:v1.1"
        },
        {
            "name": "GAP:ANNOTATE:PER_GENOME_ANNONTATION_GENERATOR",
            "container": null
        },
        {
            "name": "GAP:PROCESS_SINGLETON_GENOMES:COLLECT_FAILED_GUNC",
            "container": null
        },
        {
            "name": "GAP:KRAKEN_SWF:KRAKEN2_BUILD_LIBRARY",
            "container": "quay.io/biocontainers/kraken2:2.1.2--pl5321h9f5acd7_2"
        },
        {
            "name": "GAP:PROCESS_MANY_GENOMES:CORE_GENES",
            "container": "quay.io/microbiome-informatics/genomes-pipeline.python3base:v1.1"
        },
        {
            "name": "GAP:CRISPRCAS_FINDER",
            "container": "quay.io/microbiome-informatics/genomes-pipeline.crisprcasfinder:4.3.2"
        },
        {
            "name": "GAP:KRAKEN_SWF:BRACKEN_50",
            "container": "quay.io/biocontainers/bracken:2.8--py310h30d9df9_0"
        },
        {
            "name": "GAP:MASH_SKETCH",
            "container": "quay.io/biocontainers/mash:2.3--hd3113c8_4 "
        },
        {
            "name": "GAP:IQTREE_BAC",
            "container": "quay.io/biocontainers/iqtree:2.2.0.3--hb97b32f_1"
        },
        {
            "name": "GAP:DREP_SWF:CLASSIFY_CLUSTERS",
            "container": "quay.io/microbiome-informatics/genomes-pipeline.python3base:v1.1"
        },
        {
            "name": "GAP:ANNOTATE:IPS",
            "container": "quay.io/microbiome-informatics/genomes-pipeline.ips:5.62-94.0"
        },
        {
            "name": "GAP:KRAKEN_SWF:BRACKEN_100",
            "container": "quay.io/biocontainers/bracken:2.8--py310h30d9df9_0"
        },
        {
            "name": "GAP:MASH_TO_NWK",
            "container": "quay.io/microbiome-informatics/genomes-pipeline.mash2nwk:v1"
        },
        {
            "name": "GAP:KRAKEN_SWF:KRAKEN2_PREPARE_GTDBTK_TAX",
            "container": "quay.io/biocontainers/perl-bio-procedural:1.7.4--pl5321h9ee0642_0"
        },
        {
            "name": "GAP:KRAKEN_SWF:BRACKEN_200",
            "container": "quay.io/biocontainers/bracken:2.8--py310h30d9df9_0"
        },
        {
            "name": "GAP:PROCESS_MANY_GENOMES:PROKKA",
            "container": "quay.io/biocontainers/prokka:1.14.6--pl5321hdfd78af_5"
        },
        {
            "name": "GAP:PREPARE_DATA:GENERATE_EXTRA_WEIGHT",
            "container": "quay.io/microbiome-informatics/genomes-pipeline.python3base:v1.1"
        },
        {
            "name": "GAP:IQTREE_AR",
            "container": "quay.io/biocontainers/iqtree:2.2.0.3--hb97b32f_1"
        },
        {
            "name": "GAP:ANNOTATE:EGGNOG_MAPPER_ORTHOLOGS",
            "container": "quay.io/microbiome-informatics/genomes-pipeline.eggnog-mapper:v2.1.11"
        },
        {
            "name": "GAP:MMSEQ_SWF:MMSEQ_50",
            "container": "quay.io/biocontainers/mmseqs2:13.45111--h2d02072_0"
        },
        {
            "name": "GAP:AMRFINDER_PLUS",
            "container": "quay.io/biocontainers/ncbi-amrfinderplus:3.11.4--h6e70893_0"
        },
        {
            "name": "GAP:PROCESS_SINGLETON_GENOMES:PROKKA",
            "container": "quay.io/biocontainers/prokka:1.14.6--pl5321hdfd78af_5"
        },
        {
            "name": "GAP:PROCESS_SINGLETON_GENOMES:GUNC",
            "container": "quay.io/microbiome-informatics/genomes-pipeline.gunc:1.0.3"
        },
        {
            "name": "GAP:MMSEQ_SWF:MMSEQ_90",
            "container": "quay.io/biocontainers/mmseqs2:13.45111--h2d02072_0"
        },
        {
            "name": "GAP:GTDBTK_AND_METADATA:GTDBTK",
            "container": "quay.io/microbiome-informatics/genomes-pipeline.gtdb-tk:v2.3.0"
        },
        {
            "name": "GAP:ANNONTATE_GFF",
            "container": "quay.io/microbiome-informatics/genomes-pipeline.python3base:v1.1"
        },
        {
            "name": "GAP:GENE_CATALOGUE",
            "container": "quay.io/biocontainers/seqtk:1.3--h7132678_4"
        },
        {
            "name": "GAP:ANNOTATE:PROTEIN_CATALOGUE_STORE_ANNOTATIONS",
            "container": null
        },
        {
            "name": "GAP:MMSEQ_SWF:MMSEQ_95",
            "container": "quay.io/biocontainers/mmseqs2:13.45111--h2d02072_0"
        },
        {
            "name": "GAP:GTDBTK_AND_METADATA:PHYLO_TREE",
            "container": "quay.io/microbiome-informatics/genomes-pipeline.python3base:v1.1"
        },
        {
            "name": "GAP:DREP_SWF:DREP",
            "container": "quay.io/biocontainers/drep:3.2.2--pyhdfd78af_0"
        },
        {
            "name": "GAP:INDEX_FNA",
            "container": "quay.io/biocontainers/samtools:1.9--h10a08f8_12"
        }
    ]
}
lam-c commented 11 months ago

Thank you all very much for helping. I pull every image needed according to code snippet above, and ready for the pipeline.