Closed liuronghyw closed 1 year ago
Hi @liuronghyw, could you try the following command and then report the output?
singularity pull denglab-viroprofiler-base-v0.2.img docker://denglab/viroprofiler-base:v0.2
Hi, Dr. long. I can pull the image with your command. Since I am not familiar with Nextflow. Would you please tell me how to bulid the pipeline with local image (denglan-viroprofilter-base-v0.2)? Thank you very much! @rujinlong @deng-lab
I guess the original error occurred because I released the new version 10 hours ago, and you executed the Nextflow command simultaneously while the image was not yet successfully created. If you run the Nextflow command now, the error should not occur. Alternatively, you can pull images manually using the following commands, and then execute the Nextflow command:
singularity pull denglab-viroprofiler-base-v0.2.img docker://denglab/viroprofiler-base:v0.2
singularity pull denglab-viroprofiler-vibrant-v0.2.img docker://denglab/viroprofiler-vibrant:v0.2
singularity pull denglab-viroprofiler-binning-v0.2.img docker://denglab/viroprofiler-binning:v0.2
singularity pull denglab-viroprofiler-host-v0.2.img docker://denglab/viroprofiler-host:v0.2
singularity pull denglab-viroprofiler-geneannot-v0.2.img docker://denglab/viroprofiler-geneannot:v0.2
singularity pull denglab-viroprofiler-taxa-v0.2.img docker://denglab/viroprofiler-taxa:v0.2
singularity pull denglab-viroprofiler-abundance-v0.2.img docker://denglab/viroprofiler-abundance:v0.2
singularity pull denglab-viroprofiler-bracken-v0.2.img docker://denglab/viroprofiler-bracken:v0.2
singularity pull denglab-viroprofiler-replicyc-v0.2.img docker://denglab/viroprofiler-replicyc:v0.2
I'm having the same issue:
Error executing process > 'VIROPROFILER:SETUP:DB_VREFSEQ'
Caused by:
Failed to pull singularity image
command: singularity pull --name denglab-viroprofiler-base-v0.2.img.pulling.1681391469512 docker://denglab/viroprofiler-base:v0.2 > /dev/null
status : 1
message:
WARNING: pull for Docker Hub is not guaranteed to produce the
WARNING: same image on repeated pull. Use Singularity Registry
WARNING: (shub://) to pull exactly equivalent images.
ERROR MANIFEST_UNKNOWN: OCI index found, but accept header does not support OCI indexes
ERROR: pulling container failed!
What is the version of singularity? Could you try the following command to update the pipeline,
nextflow pull deng-lab/viroprofiler
and then run with the -r main
parameter,
nextflow run deng-lab/viroprofiler -profile singularity --mode "setup" -r main
Thank you! I did as your said and obtained this error:
Error executing process > 'VIROPROFILER:SETUP:DB_CHECKV'
Caused by:
Failed to pull singularity image
command: singularity pull --name denglab-viroprofiler-base.img.pulling.1681393187226 docker://denglab/viroprofiler-base > /dev/null
status : 1
message:
WARNING: pull for Docker Hub is not guaranteed to produce the
WARNING: same image on repeated pull. Use Singularity Registry
WARNING: (shub://) to pull exactly equivalent images.
ERROR MANIFEST_UNKNOWN: OCI index found, but accept header does not support OCI indexes
ERROR: pulling container failed!
Could you please paste the first two lines of the output here? it should look like the following,
N E X T F L O W ~ version 22.04.0
Launching `https://github.com/deng-lab/viroprofiler` [soggy_elion] DSL2 - revision: 471c32e0ba [main]
Sure!
$ nextflow run deng-lab/viroprofiler -profile singularity --mode "setup" -r main
N E X T F L O W ~ version 22.10.6
Launching `https://github.com/deng-lab/viroprofiler` [hopeful_borg] DSL2 - revision: 471c32e0ba [main]
It's quite strange. The error seems to be caused by singularity. I cannot reproduce this error, but if you can provide additional information, it may help me identify the bug. Could you try as the following,
singularity --version
which singularity
and paste the output here.
The first thing failed and gave this error:
$ singularity pull denglab-viroprofiler-base-v0.2.img docker://denglab/viroprofiler-base:v0.2
ERROR: pull is only supported for shub URIs
Singularity version and path:
$ singularity --version
2.4.2-dist
$ which singularity
/home/olga/miniconda3/bin/singularity
Ok, this might be the reason. Your singularity version is quite old. Could you update to the newest version? The latest on conda is singularity v3.8.7 and singularityce v3.10.0. If possible, ask your sys-admin to install the newest singularity as root, as explained in another issue https://github.com/darcyabjones/pante/issues/4#issuecomment-546830722
I installed a newer version of Singularity.
$ singularity --version
singularity-ce version 3.11.1-jammy
It is now giving a different error:
$ nextflow run deng-lab/viroprofiler -profile singularity --mode "setup" -r main
WorkflowError:
Failed to solve the job scheduling problem with pulp. Please report a bug and use --scheduler greedy as a workaround:
Pulp: Error while trying to execute, use msg=True for more detailscbc
[2023-04-13 19:49 CRITICAL] Command 'snakemake --snakefile /opt/conda/envs/viroprofiler-virsorter2/lib/python3.10/site-packages/virsorter/rules/setup-retry.smk --directory /home/olga/viroprofiler/virsorter2 --quiet --config Skip_deps_install=False --jobs 1 --rerun-incomplete --latency-wait 600 --nolock --use-conda --conda-prefix /home/olga/viroprofiler/virsorter2/conda_envs --conda-frontend mamba ' returned non-zero exit status 1.
Work dir:
/home/olga/work/b9/26db2d6609386b7de7b34fa9ee8433
Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`
Thank you very much
It seems this is caused by snakemake in virsorter2 https://github.com/jiarong/VirSorter2/issues/51 (also see this). I updated the code, please try the following commands,
# update the pipeline
nextflow pull deng-lab/viroprofiler
# delete the old virsorter2 database (in your case, /home/olga/viroprofiler/virsorter2)
rm -rf /path/to/viroprofiler/db/virsorter2
# run the pipeline
nextflow run deng-lab/viroprofiler -profile singularity --mode "setup" -r main
Thank you, I did as you said and running nextflow run deng-lab/viroprofiler -profile singularity --mode "setup" -r main
worked:
executor > local (3)
[05/27b9fd] process > VIROPROFILER:SETUP:DB_CHECKV [100%] 1 of 1, cached: 1 ✔
[ec/7f85af] process > VIROPROFILER:SETUP:DB_VIBRANT [100%] 1 of 1, cached: 1 ✔
[fe/c41c20] process > VIROPROFILER:SETUP:DB_VIRSORTER2 [100%] 1 of 1 ✔
[ef/8e9ed3] process > VIROPROFILER:SETUP:DB_DRAM [100%] 1 of 1 ✔
[df/78f962] process > VIROPROFILER:SETUP:DB_IPHOP [100%] 1 of 1 ✔
[cb/451b85] process > VIROPROFILER:SETUP:DB_VREFSEQ [100%] 1 of 1, cached: 1 ✔
-[ViroProfiler] Pipeline completed successfully-
But then running the test failed:
$ nextflow run deng-lab/viroprofiler -r main -profile singularity,test
This is the error I got:
Error executing process > 'VIROPROFILER:VIRSORTER2'
Caused by:
Process `VIROPROFILER:VIRSORTER2` terminated with an error exit status (1)
Command executed:
virsorter config --set HMMSEARCH_THREADS=2
virsorter run --seqname-suffix-off --viral-gene-enrich-off --prep-for-dramv -i putative_vcontigs_pref1.fasta -w out_vs2 --include-groups dsDNAphage --min-length 3000 --min-score 0.5 -j 2 --provirus-off -d /home/olga/viroprofiler/virsorter2 all
grep '^>' out_vs2/final-viral-combined.fa | sed 's/>//' | sed 's/||.*//' > virus_virsorter2.list
ln -s out_vs2/for-dramv/final-viral-combined-for-dramv.fa .
ln -s out_vs2/for-dramv/viral-affi-contigs-for-dramv.tab .
seqkit fx2tab -n final-viral-combined-for-dramv.fa | sed 's/-cat_/,/g' | csvtk add-header -n Contig,vs2_category > vs2_category.csv
cat <<-END_VERSIONS > versions.yml
"VIROPROFILER:VIRSORTER2":
executor > local (29)
[db/8c0fa0] process > VIROPROFILER:FASTQC (UC30) [100%] 15 of 15, cached: 15 ✔
[d3/91c5e0] process > VIROPROFILER:FASTP (UC30) [100%] 15 of 15, cached: 15 ✔
[87/47afb0] process > VIROPROFILER:SPADES (HT24) [100%] 15 of 15, cached: 14 ✔
[d7/86e681] process > VIROPROFILER:CONTIGLIB [100%] 1 of 1 ✔
[ce/a24c4f] process > VIROPROFILER:CHECKV [100%] 1 of 1 ✔
[3e/545e1b] process > VIROPROFILER:CONTIGLIB_CLUSTER [100%] 1 of 1 ✔
[c4/607316] process > VIROPROFILER:GENEPRED4CTG [100%] 1 of 1 ✔
[46/f2fbf7] process > VIROPROFILER:NRPROT [100%] 1 of 1 ✔
[98/6f1211] process > VIROPROFILER:NRGENE [100%] 1 of 1 ✔
[02/8154e6] process > VIROPROFILER:CONTIGINDEX [100%] 1 of 1 ✔
[6d/32e55f] process > VIROPROFILER:MAPPING2CONTIGS2 (HT24) [100%] 15 of 15 ✔
[28/e6bfaf] process > VIROPROFILER:ABUNDANCE [100%] 1 of 1 ✔
[03/627fa9] process > VIROPROFILER:VIBRANT [100%] 1 of 1 ✔
[1c/6f9093] process > VIROPROFILER:DVF [100%] 1 of 1 ✔
[84/3d432f] process > VIROPROFILER:VIRCONTIGS_PRE [100%] 1 of 1 ✔
[42/6be750] process > VIROPROFILER:VIRSORTER2 [100%] 1 of 1, failed: 1 ✘
[- ] process > VIROPROFILER:DRAMV -
[- ] process > VIROPROFILER:TAXONOMY_VCONTACT -
[- ] process > VIROPROFILER:TAXONOMY_MMSEQS -
[- ] process > VIROPROFILER:TAXONOMY_MERGE -
[f5/24a9a1] process > VIROPROFILER:VIRALHOST_IPHOP [100%] 1 of 1, failed: 1 ✘
[- ] process > VIROPROFILER:BACPHLIP -
[- ] process > VIROPROFILER:RESULTS_TSE -
[- ] process > VIROPROFILER:CUSTOM_DUMPSOFTWAREVERSIONS -
[- ] process > VIROPROFILER:MULTIQC -
Pulling Singularity image docker://denglab/viroprofiler-taxa:v0.2 [cache /home/olga/work/singularity/denglab-viroprofiler-taxa-v0.2.img]
Pulling Singularity image docker://denglab/viroprofiler-replicyc:v0.2 [cache /home/olga/work/singularity/denglab-viroprofiler-replicyc-v0.2.img]
Execution cancelled -- Finishing pending tasks before exit
Any ideas?
There is a log file in work/42/6be750*/.command.log
, could show me the content of that file?
In /home/olga/work/f5/24a9a1c9f030cba84ff25d0ee074cc/.command.log
, there was this:
Pblm, I could not find directory /home/olga/viroprofiler/iphop/Sept_2021_pub. Maybe the host database directory was not specified with the --db_dir and is not the default "db/" directory ?
Your previous setup of the iphop database is not complete. The complete iphop database is ~200G. Please delete the iphop folder, and then run the setup step again, or you can use the --use_iphop false
parameter to skip the host prediction step.
Description of the bug
Dear developers: I can't pull the viroprofilter in cluster with UNIX operate system, the command as follows:
export NXF_SINGULARITY_CACHEDIR="./temp"
nextflow run deng-lab/viroprofiler -profile singularity --mode "setup" -r v0.2 --db "/public/home/**/db/viroprofiler" -without-conda
The error is as follows:![image](https://user-images.githubusercontent.com/42331632/231637753-f4ecce7a-1134-4a37-be7d-e6546fc9754c.png)
Would you please give me some suggestions on how to solve this problem.
Command used and terminal output
Relevant files
No response
System information
N E X T F L O W ~ version 22.10.6
HPC local Signalarity CentOS Linux
viroprofiler v0.2