search

epi2me-labs / wf-human-variation

Other
86 stars 41 forks source link

Bamstats version + load container images #194

Open hgingras opened 5 days ago

hgingras commented 5 days ago

Ask away!

Hi, I have two questions:

  1. I would like to know bamtats version you are using in the pipeline.

  2. I am trying to load the different container image but I was only successful with these two:

This one I guess is the common: apptainer pull docker://ontresearch/wf-human-variation wf-human-variation_latest.sif

This one is okay: apptainer pull docker://ontresearch/wf-human-variation:sha2b856c1f358ddf1576217a336bc0e9864b6dc0ed wf-human-variation_sha2b856c1f358ddf1576217a336bc0e9864b6dc0ed.sif

This one and all the other failed: apptainer pull docker://ontresearch/wf-human-variation:sha17e686336bf6305f9c90b36bc52ff9dd1fa73ee9

FATAL: While making image from oci registry: error fetching image to cache: failed to get checksum for docker://ontresearch/wf-human-variation:sha17e686336bf6305f9c90b36bc52ff9dd1fa73ee9: reading manifest sha17e686336bf6305f9c90b36bc52ff9dd1fa73ee9 in docker.io/ontresearch/wf-human-variation: manifest unknown

Here is the full list from nextflow.config:

// container sha e2l_base_tag = "sha2b856c1f358ddf1576217a336bc0e9864b6dc0ed" e2l_snp_tag = "sha17e686336bf6305f9c90b36bc52ff9dd1fa73ee9" e2l_sv_tag = "shac591518dd32ecc3936666c95ff08f6d7474e9728" e2l_mod_tag = "sha0253e9e9ba92aacc852ba376edefe8ff0932f71a" cnv_tag = "sha428cb19e51370020ccf29ec2af4eead44c6a17c2" str_tag = "shaa2f49ce57886426516eadd4048b6fdf9c22c7437" spectre_tag = "sha5a2890023dc7a7899f47585103b4f5762fb9d1b3" snpeff_tag = "sha313729d816872d70b410752001a0e2654eb88807" common_sha = "sha338caea0a2532dc0ea8f46638ccc322bb8f9af48"

Note that I cannot use this command to run the workflow. I do not have internet access on the system I am working with: nextflow run epi2me-labs/wf-human-variation

This is why I need to load all the images.

As a "B" plan, I am trying to load all the software required to work local. This is why I am asking for bamstats version. If you have a list of all the software needed that would be helpful.

Thanks for helping me. Best regards, Helene

hgingras commented 4 days ago

I succeeded to download the .sif files:

apptainer pull docker://ontresearch/wf-human-variation-snp:sha17e686336bf6305f9c90b36bc52ff9dd1fa73ee9 apptainer pull docker://ontresearch/wf-human-variation-sv:shac591518dd32ecc3936666c95ff08f6d7474e9728 apptainer pull docker://ontresearch/modkit:shae137452eb41f5f12b790774cafe15bc97f48d4d0 apptainer pull docker://ontresearch/wf-cnv:sha428cb19e51370020ccf29ec2af4eead44c6a17c2 apptainer pull docker://ontresearch/snpeff:sha313729d816872d70b410752001a0e2654eb88807 apptainer pull docker://ontresearch/wf-common:sha338caea0a2532dc0ea8f46638ccc322bb8f9af48 apptainer pull docker://ontresearch/spectre:sha49a9fe474da9860f84f08f17f137b47a010b1834 apptainer pull docker://ontresearch/wf-human-variation:sha2b856c1f358ddf1576217a336bc0e9864b6dc0ed

Could it still be possible to have a list of all the requirements?

Best regards,

SamStudio8 commented 4 days ago

Hi Helene, I am glad you have now been able to download the SIF files for the workflow. You can inspect these containers for the dependencies, but we do not recommend or support running workflows in custom local environments outside of the containers provided. You'll encounter particular difficulties doing this as several of the workflow dependencies conflict with eachother, necessitating the various different containers used by the workflow.

Thanks, Sam

On Wed, 26 Jun 2024, 21:28 Hélène Gingras, @.***> wrote:

I succeeded to download the .sif files:

apptainer pull docker://ontresearch/wf-human-variation-snp:sha17e686336bf6305f9c90b36bc52ff9dd1fa73ee9 apptainer pull docker://ontresearch/wf-human-variation-sv:shac591518dd32ecc3936666c95ff08f6d7474e9728 apptainer pull docker://ontresearch/modkit:shae137452eb41f5f12b790774cafe15bc97f48d4d0 apptainer pull docker://ontresearch/wf-cnv:sha428cb19e51370020ccf29ec2af4eead44c6a17c2 apptainer pull docker://ontresearch/snpeff:sha313729d816872d70b410752001a0e2654eb88807 apptainer pull docker://ontresearch/wf-common:sha338caea0a2532dc0ea8f46638ccc322bb8f9af48 apptainer pull docker://ontresearch/spectre:sha49a9fe474da9860f84f08f17f137b47a010b1834 apptainer pull docker://ontresearch/wf-human-variation:sha2b856c1f358ddf1576217a336bc0e9864b6dc0ed

Could it still be possible to have a list of all the requirements?

Best regards,

— Reply to this email directly, view it on GitHub https://github.com/epi2me-labs/wf-human-variation/issues/194#issuecomment-2192569604, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIN6OUWOTGSBVGNKXUEX7LZJMP73AVCNFSM6AAAAABJ4R4TKWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJSGU3DSNRQGQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

hgingras commented 3 days ago

Hi Sam, thanks for your comment.

I was able to run the pipeline with test data with apptainer. If it can help others, here is what I have modified:

I added this section in nextflow.config """profile""" section:

// using apptainer instead of docker
apptainer {
    apptainer {
        enabled = true
        autoMounts = true
        //envWhitelist = "" // if your cluster sets a variable to indicate which GPU has been assigned you will want to allow it here
    }
process."withLabel:gpu".containerOptions = "--nv"
}

Here is the sbatch script to run on our HPC system:

!/bin/bash

SBATCH --account=def-account

SBATCH --cpus-per-task=8

SBATCH --time=0-02:00

SBATCH --mem=40G

module load nextflow/23.10.0 apptainer/1.2.4

export NXF_APPTAINER_CACHEDIR="/scratch/$USER/apptainer/cache" export NXF_APPTAINER_TMPDIR="/scratch/$USER/apptainer/tmp"

nextflow run main.nf \ --bam 'wf-human-variation-demo/demo.bam' \ --basecaller_cfg 'dna_r10.4.1_e8.2_400bps_hac_prom' \ --mod \ --ref 'wf-human-variation-demo/demo.fasta' \ --sample_name 'DEMO' \ --snp \ --sv \ -profile apptainer

You can close the issue.

Have a good one,

Hélène