Open claumer opened 1 year ago
Hi Chris,
not an expert with the singularity stuff here but it seems as if the image is not being pulled because of some naming problem.
We run TransPi with conda and tend not to get conflicts except for one step. Do you face conflicts creating all environments or only one?
Sergio
Hi Sergio,
Thanks for the fast response. When running the same command in a fresh environment with -profile conda, it halts with the following:
Error executing process > 'evigene (SRR8641368_R)'
Caused by:
Failed to create Conda environment
command: conda create --mkdir --yes --quiet --prefix /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/users/chril1/TransPiDBs/condaEnv/env-0761ea60e49e247680763dddcc8ef240 -c conda-forge bioconda::cd-hit=4.8.1 bioconda::exonerate=2.4 bioconda::blast=2.11.0
status : 1
message:
Building graph of deps: 0%| | 0/8 [00:00<?, ?it/s]
Examining @/linux-64::cuda==11.6=0: 0%| | 0/8 [00:00<?, ?it/s]
Examining bioconda::blast=2.11.0: 12%|█▎ | 1/8 [00:00<00:00, 8439.24it/s]
Examining @/linux-64::glibc==2.31=0: 25%|██▌ | 2/8 [00:00<00:00, 131.25it/s]
Examining bioconda::cd-hit=4.8.1: 38%|███▊ | 3/8 [00:00<00:00, 195.25it/s]
Examining @/linux-64::linux==5.4.0=0: 50%|█████ | 4/8 [00:00<00:00, 110.94it/s]
Examining @/linux-64::archspec==1=x86_64: 62%|██████▎ | 5/8 [00:00<00:00, 138.28it/s]
Examining bioconda::exonerate=2.4: 75%|███████▌ | 6/8 [00:00<00:00, 165.61it/s]
Examining @/linux-64::__unix==0=0: 88%|████████▊ | 7/8 [00:00<00:00, 79.24it/s]
Determining conflicts: 0%| | 0/8 [00:00<?, ?it/s]
Examining conflict for exonerate blast __glibc cd-hit: 0%| | 0/8 [00:00<?, ?it/s]
Examining conflict for blast exonerate cd-hit: 12%|█▎ | 1/8 [00:00<00:00, 7.35it/s]
Examining conflict for blast exonerate cd-hit: 25%|██▌ | 2/8 [00:00<00:00, 14.70it/s]
Examining conflict for blast exonerate: 25%|██▌ | 2/8 [00:00<00:00, 14.70it/s]
UnsatisfiableError: The following specifications were found to be incompatible with each other:
Output in format: Requested package -> Available versions
Package libzlib conflicts for:
bioconda::cd-hit=4.8.1 -> libzlib[version='>=1.2.11,<1.3.0a0|>=1.2.12,<1.3.0a0']
bioconda::exonerate=2.4 -> glib -> libzlib[version='>=1.2.11,<1.3.0a0|>=1.2.12,<1.3.0a0']
bioconda::cd-hit=4.8.1 -> zlib[version='>=1.2.12,<1.3.0a0'] -> libzlib[version='1.2.11|1.2.11|1.2.11|1.2.12',build='h36c2ea0_1012|h166bdaf_1014|h166bdaf_0|h166bdaf_4|h166bdaf_3|h166bdaf_2|h166bdaf_1|h36c2ea0_1013']
bioconda::blast=2.11.0 -> curl[version='>=7.77.0,<8.0a0'] -> libzlib[version='1.2.11|1.2.11|1.2.11|1.2.12|1.2.12|1.2.12|1.2.12|1.2.12|>=1.2.11,<1.3.0a0|>=1.2.12,<1.3.0a0',build='h36c2ea0_1012|h166bdaf_1014|h166bdaf_0|h166bdaf_4|h166bdaf_3|h166bdaf_2|h166bdaf_1|h36c2ea0_1013']
Package pcre conflicts for:
bioconda::exonerate=2.4 -> pcre[version='>=8.41,<9.0a0|>=8.43,<9.0a0|>=8.44,<9.0a0|>=8.45,<9.0a0']
bioconda::blast=2.11.0 -> pcre[version='>=8.44,<9.0a0']
bioconda::exonerate=2.4 -> glib -> pcre[version='>=8.42,<9.0a0']
Package libgcc-ng conflicts for:
bioconda::blast=2.11.0 -> bzip2[version='>=1.0.8,<2.0a0'] -> libgcc-ng[version='>=10.3.0|>=12|>=7.3.0|>=9.4.0|>=11.2.0|>=4.9|>=7.2.0']
bioconda::blast=2.11.0 -> libgcc-ng[version='>=7.5.0|>=9.3.0']
Package libstdcxx-ng conflicts for:
bioconda::blast=2.11.0 -> pcre[version='>=8.44,<9.0a0'] -> libstdcxx-ng[version='>=7.3.0']
bioconda::blast=2.11.0 -> libstdcxx-ng[version='>=7.5.0|>=9.3.0']The following specifications were found to be incompatible with your system:
- feature:/linux-64::__glibc==2.31=0
- bioconda::blast=2.11.0 -> libgcc-ng[version='>=9.3.0'] -> __glibc[version='>=2.17']
- bioconda::cd-hit=4.8.1 -> libgcc-ng[version='>=9.3.0'] -> __glibc[version='>=2.17']
- bioconda::exonerate=2.4 -> libgcc-ng[version='>=9.3.0'] -> __glibc[version='>=2.17']
Your installed version is: 2.31
Note that strict channel priority may have removed packages required for satisfiability.
Hi Chris,
I checked the code. Specifically, the following line in the error message means that a different docker image is being used in this particular step. Why exactly busco is not included in the transpi image is something I cannot answer because I am not using docker/singularity and cannot test in our hpc anything related to these systems.
Pulling Singularity image docker://null [cache /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/users/chril1/TransPiDBs/singularityCache/null.img]
Now, I think you can solve the issue by adding the following line to your nextflow.config
file:
TransPiContainer {
process {
params.oneContainer = true
params.TPcontainer="rerv/transpi:v1.0.0"
//sevra: fix reported problem with busco container
params.v4container="ezlabgva/busco:v4.0.5_cv1"
}
}
the variable params.v4container
is used in TransPi in the following block:
if (params.oneContainer){ container "${params.v4container}" } else {
container (workflow.containerEngine == 'singularity' && !params.singularity_pull_docker_container ? "https://depot.galaxyproject.org/singularity/busco:4.1.4--py_2" : "ezlabgva/busco:v4.0.5_cv1")
}
The problem right now is that params.v4container
is not defined and set to null
. This explains the docker://null
call in your error. I guess setting the variable to a valid container will solve the issue; no promises, though.
A working block of code, say Transabyss, looks like:
if (params.oneContainer){ container "${params.TPcontainer}" } else {
container (workflow.containerEngine == 'singularity' && !params.singularity_pull_docker_container ? "https://depot.galaxyproject.org/singularity/transabyss:2.0.1--pyh864c0ab_7" : "quay.io/biocontainers/transabyss:2.0.1--pyh864c0ab_7")
}
You see the structure is the same but the TPcontainer is called. This works before params.TPcontainer
is set in the config.
I hope this solves your issue. Please report back to the community and feel free to close the issue if solved. Note however, that other issues may arise from using that particular image, I hope not because it is used if the oneContainer variable is false i.e., you are not using one container with all the stuff but getting multiple images for each process in the pipeline.
cheers
Sergio
Hi there,
I've been trying to use TransPi (love the concept and the benchmarking in the paper looks great!) to re-assemble some older public datasets. Running on an interactive slurm session of an HPC which has singularity installed:
(TransPi) chril1@hpc-gpu-002:~/TransPiDBs$ ./nextflow run ../TransPi/TransPi.nf --all --maxReadLen 150 --k 25,35,55,75,85 --reads '/home/chril1/NERC_PhyloCourse/transcriptome_data/Flagellophora_sp/SRR8641368_R[1,2].fastq.gz' --outdir Results_Flagellophora -profile singularity,TransPiContainer
N E X T F L O W ~ version 21.04.1 Launching
../TransPi/TransPi.nf
[furious_einstein] - revision: 5a26a6e85dTransPi - Transcriptome Analysis Pipeline v1.3.0-rc
TransPi.nf Directory: /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/users/chril1/TransPi/TransPi.nf Launch Directory: /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/users/chril1/TransPiDBs Results Directory: Results_Flagellophora Work Directory: /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/users/chril1/TransPiDBs/work TransPi DBs: /home/chril1/TransPiDBs Uniprot DB: /home/chril1/TransPiDBs/DBs/uniprot_db/uniprot_metazoa_33208.fasta Busco DB: /home/chril1/TransPiDBs/DBs/busco_db/metazoa_odb10 Reads Directory: /home/chril1/NERC_PhyloCourse/transcriptome_data/Flagellophora_sp/SRR8641368_R[1,2].fastq.gz Read Length: 150 Kmers: 25,35,55,75,85
executor > local (2) [6c/876909] process > fasqc (SRR8641368_R) [ 0%] 0 of 1 executor > local (3) [6c/876909] process > fasqc (SRR8641368_R) [ 0%] 0 of 1 executor > local (3) [6c/876909] process > fasqc (SRR8641368_R) [ 0%] 0 of 1 executor > local (3) [6c/876909] process > fasqc (SRR8641368_R) [ 0%] 0 of 1 executor > local (3) [6c/876909] process > fasqc (SRR8641368_R) [ 0%] 0 of 1 executor > local (3) [6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔ executor > local (3) [6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔ executor > local (4) executor > local (5) executor > local (6) executor > local (6) executor > local (6) executor > local (6) executor > local (6) executor > local (6) [6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔ executor > local (8) [6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔ executor > local (9) [6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔ executor > local (9) [6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔ executor > local (10) executor > local (10) executor > local (10) executor > local (10) executor > local (11) executor > local (11) executor > local (11) executor > local (11) executor > local (11) [6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔ executor > local (13) [6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔ executor > local (13) [6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔ [38/2daf75] process > fastp (SRR8641368_R) [100%] 1 of 1 ✔ [09/b36296] process > fastp_stats (SRR8641368_R) [100%] 1 of 1 ✔ [a5/77f26f] process > skip_rrna_removal (SRR8641368_R) [100%] 1 of 1 ✔ [94/71880f] process > normalize_reads (SRR8641368_R) [100%] 1 of 1 ✔ [de/71f78c] process > trinity_assembly (SRR8641368_R) [100%] 1 of 1 ✔ [de/bc6216] process > soap_assembly (SRR8641368_R) [100%] 1 of 1 ✔ [25/457622] process > velvet_oases_assembly (SRR8641368_R) [100%] 1 of 1 ✔ [84/17fe6c] process > rna_spades_assembly (SRR8641368_R) [100%] 1 of 1 ✔ [2e/672733] process > transabyss_assembly (SRR8641368_R) [100%] 1 of 1 ✔ [80/f0f8e6] process > evigene (SRR8641368_R) [ 0%] 0 of 1 [- ] process > rna_quast - [- ] process > mapping_evigene - [- ] process > busco4 - [ca/050e8b] process > mapping_trinity (SRR8641368_R) [ 0%] 0 of 1 [- ] process > summary_evigene_individual - [- ] process > busco4_tri - [47/a485e7] process > skip_busco_dist (SRR8641368_R) [100%] 1 of 1, cached: 1 ✔ executor > local (13) [6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔ [38/2daf75] process > fastp (SRR8641368_R) [100%] 1 of 1 ✔ [09/b36296] process > fastp_stats (SRR8641368_R) [100%] 1 of 1 ✔ [a5/77f26f] process > skip_rrna_removal (SRR8641368_R) [100%] 1 of 1 ✔ [94/71880f] process > normalize_reads (SRR8641368_R) [100%] 1 of 1 ✔ [de/71f78c] process > trinity_assembly (SRR8641368_R) [100%] 1 of 1 ✔ [de/bc6216] process > soap_assembly (SRR8641368_R) [100%] 1 of 1 ✔ [25/457622] process > velvet_oases_assembly (SRR8641368_R) [100%] 1 of 1 ✔ [84/17fe6c] process > rna_spades_assembly (SRR8641368_R) [100%] 1 of 1 ✔ [2e/672733] process > transabyss_assembly (SRR8641368_R) [100%] 1 of 1 ✔ [80/f0f8e6] process > evigene (SRR8641368_R) [ 0%] 0 of 1 [- ] process > rna_quast - [- ] process > mapping_evigene - [- ] process > busco4 - [ca/050e8b] process > mapping_trinity (SRR8641368_R) [ 0%] 0 of 1 [- ] process > summary_evigene_individual - [- ] process > busco4_tri - [47/a485e7] process > skip_busco_dist (SRR8641368_R) [100%] 1 of 1, cached: 1 ✔ [- ] process > summary_busco4_individual - [- ] process > get_busco4_comparison - [- ] process > transdecoder_longorf - [- ] process > transdecoder_diamond - [- ] process > transdecoder_hmmer - [- ] process > transdecoder_predict - [- ] process > swiss_diamond_trinotate - [- ] process > custom_diamond_trinotate - [- ] process > hmmer_trinotate - [- ] process > skip_signalP - [- ] process > skip_tmhmm - [- ] process > skip_rnammer - [- ] process > trinotate - [- ] process > get_GO_comparison - [- ] process > summary_custom_uniprot - [- ] process > skip_kegg - [- ] process > get_transcript_dist - [- ] process > summary_transdecoder_individual - [- ] process > summary_trinotate_individual - [- ] process > get_report - [98/934751] process > get_run_info [100%] 1 of 1 ✔ Pulling Singularity image docker://null [cache /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/users/chril1/TransPiDBs/singularityCache/null.img] Error executing process > 'busco4_tri (SRR8641368_R)'
Caused by: Failed to pull singularity image command: singularity pull --name null.img.pulling.1664887974429 docker://null > /dev/null status : 255 message: FATAL: While making image from oci registry: error fetching image to cache: failed to get checksum for docker://null: Error reading manifest latest in docker.io/library/null: errors: denied: requested access to the resource is denied unauthorized: authentication required
It seems that it gets all the way through building the four assemblies then crashes when trying to pull a singularity image for BUSCO evaluation. I confess I'm not quite sure how to debug this one - do you have any ideas? I've also tried -profile conda and am getting environment creation conflict errors out of that, and so seem to have reached an impasse.
Thanks very much for your attention,
Chris L