search

deng-lab / viroprofiler

A containerized bioinformatics pipeline for viral metagenomic data analysis
https://deng-lab.github.io/viroprofiler
MIT License
23 stars 12 forks source link

issue with test #5

Closed XC-Zhai closed 1 year ago

XC-Zhai commented 1 year ago

Description of the bug

Thanks for this one-pool pipeline for virome analysis. I had the issue when I run the test.

################################################################# nextflow run deng-lab/viroprofiler -r main -profile singularity,test

N E X T F L O W ~ version 22.10.6 NOTE: Your local project version looks outdated - a different revision is available in the remote repository [49aeaa6056] Launching https://github.com/deng-lab/viroprofiler [agitated_descartes] DSL2 - revision: 471c32e0ba [v0.2]

                                           __

oooooo oooo o8o 88 88 .o88o. o8o oooo 888. .8'"' 88 ss 88 888 ""' 888 888. .8' oooo oooo d8b .ooooo. 88_88 oooo d8b .ooooo. o888oo oooo 888 .ooooo. oooo d8b 888. .8'888 888""8P d88'88b || 888""8P d88'88b 888 888 888 d88'88b 888""8P 888.8' 888 888 888 888 ||_ 888 888 888 888 888 888 888ooo888 888 888' 888 888 888 888 // || \\ 888 888 888 888 888 888 888 .o 888 8' o888o d888b Y8bod8P' // \\ d888bY8bod8P' o888o o888o o888o `Y8bod8P' d888b ViroProfiler v0.2

Core Nextflow options revision : v0.2 runName : agitated_descartes containerEngine : singularity launchDir : /media/server/95d691e7-0d78-417e-b796-a49c36203e5e/JOSUE/Xichuan28122022 workDir : /media/server/95d691e7-0d78-417e-b796-a49c36203e5e/JOSUE/Xichuan28122022/work projectDir : /home/server/.nextflow/assets/deng-lab/viroprofiler userName : server profile : singularity,test configFiles : /home/server/.nextflow/assets/deng-lab/viroprofiler/nextflow.config

Input/output options mode : all input : https://raw.githubusercontent.com/deng-lab/viroprofiler/main/assets/samplesheet.csv db : /home/server/viroprofiler outdir : output

QC contamref_idx : /home/server/viroprofiler/contamination_refs/hg19/ref

Contig library parameters assemblies : scaffolds

Others replicyc : bacphlip use_iphop : true use_dram : true

Institutional config options config_profile_name : Test profile config_profile_description: Minimal test dataset to check pipeline function

Max job request options max_cpus : 2 max_memory : 6.GB max_time : 24.h

!! Only displaying parameters that differ from the pipeline defaults !!

If you use ViroProfiler for your analysis please cite:

Caused by: Process VIROPROFILER:TAXONOMY_MMSEQS terminated with an error exit status (132)

Command executed:

Run mmseqs taxonomy

mmseqs createdb putative_vcontigs_pref1.fasta qry mmseqs taxonomy qry /home/server/viroprofiler/taxonomy/mmseqs_vrefseq/refseq_viral mmseqsTaxaRst tmp --tax-lineage 1 --majority 0.4 --vote-mode 1 --lca-mode 3 --

report

mmseqs createtsv qry mmseqsTaxaRst mmseqsTaxaRst.tsv mmseqs taxonomyreport /home/server/viroprofiler/taxonomy/mmseqs_vrefseq/refseq_viral mmseqsTaxaRst mmseqsTaxaRst_report.txt --report-mode 0 mmseqs taxonomyreport /home/server/viroprofiler/taxonomy/mmseqs_vrefseq/refseq_viral mmseqsTaxaRst mmseqsTaxaRst_report.html --report-mode 1

cat <<-END_VERSIONS > versions.yml "VIROPROFILER:TAXONOMY_MMSEQS": MMseqs2: $(grep "MMseqs Version" .command.log | head -n1 | sed 's/.* //g') END_VERSIONS

Command exit status: 132

Command output: (empty)

Command error: .command.sh: line 3: 6861 Illegal instruction (core dumped) mmseqs createdb putative_vcontigs_pref1.fasta qry

Work dir: /media/server/95d691e7-0d78-417e-b796-a49c36203e5e/JOSUE/Xichuan28122022/work/b6/1c9bb5c790e6b80ec842d72276a8ac

Tip: when you have fixed the problem you can continue the execution adding the option -resume to the run command line

########################################

Command used and terminal output

is this due to the vs2 or core dumped with mmseqs?

Thanks for your help.

Relevant files

No response

System information

Distributor ID: Ubuntu Description: Ubuntu 16.04.7 LTS Release: 16.04 Codename: xenial

rujinlong commented 1 year ago

Hi @XC-Zhai. Thanks for reporting this. I created a new release v0.2.2 that should have fixed the bug. Please update the pipeline using the following command,

nextflow pull deng-lab/viroprofiler

and then run the pipeline with -r main parameter.

XC-Zhai commented 1 year ago

Thanks a lot. Now I had another issue. Error executing process > 'VIROPROFILER:VIRALHOST_IPHOP'

Caused by: Process VIROPROFILER:VIRALHOST_IPHOP terminated with an error exit status (255)

Command executed:

iphop predict --fa_file putative_vcontigs_pref1.fasta --out_dir out_iphop --db_dir /home/server/viroprofiler/iphop/Sept_2021_pub --num_threads 2

cat <<-END_VERSIONS > versions.yml "VIROPROFILER:VIRALHOST_IPHOP": iPHoP: $(iphop --version | head -n1 | sed 's/iPHoP v//;s/: .*//') seqkit: $( seqkit | sed '3!d; s/Version: //' ) END_VERSIONS

Command exit status: 255

Command output: (empty)

Command error: FATAL: while executing starter: while initializing starter command: /usr/local/libexec/singularity/bin/starter-suid not found, please check your installation

Work dir: /media/server/95d691e7-0d78-417e-b796-a49c36203e5e/work/4c/e08f23797fbe0041aed55fc9331ad6

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named .command.sh

I am using singularity-ce version 3.11.1. Do you have any suggestions for trying it out? Best, Xichuan

rujinlong commented 1 year ago

This problem seems related to the container image. I made a fix to roll back the host prediction module to v0.1. Could you please try with -r fix_iphop parameter and let me know the output?

XC-Zhai commented 1 year ago

-r fix_iphop

Hi Rujin, I tried with command: nextflow run deng-lab/viroprofiler -r fix_iphop -profile singularity,test -resume

[N E X T F L O W ~ version 22.10.6 Launching https://github.com/deng-lab/viroprofiler [grave_montalcini] DSL2 - revision: 4820e9f13f [fix_iphop]]

And got another one with vs2: Error executing process > 'VIROPROFILER:VIRSORTER2'

Caused by: Failed to pull singularity image command: singularity pull --name denglab-viroprofiler-base-v0.2.2.img.pulling.1682416505908 docker://denglab/viroprofiler-base:v0.2.2 > /dev/null status : 255 message: FATAL: While making image from oci registry: error fetching image to cache: failed to get checksum for docker://denglab/viroprofiler-base:v0.2.2: reading manifest v0.2.2 in docker.io/denglab/viroprofiler-base: manifest unknown

Any suggestions to this?

Best, Xichuan

rujinlong commented 1 year ago

Sorry, there is a typo in previous revision. Please update the pipeline and run again with the -r fix_iphop parameter. It should show DSL2 - revision: 7e29324 [fix_iphop] now.

XC-Zhai commented 1 year ago

Sorry, there is a typo in previous revision. Please update the pipeline and run again with the -r fix_iphop parameter. It should show DSL2 - revision: 7e29324 [fix_iphop] now.

Yes, it worked with -r fix_iphop, and it also needs to be setup with this parameter. But I had another following-up issue:

[f8/2313a8] process > VIROPROFILER:VIRALHOST_IPHOP [100%] 1 of 1, failed: 1 ✘ [60/121a7b] process > VIROPROFILER:BACPHLIP [100%] 1 of 1 ✔ [- ] process > VIROPROFILER:RESULTS_TSE - [- ] process > VIROPROFILER:CUSTOM_DUMPSOFTWAREVERSIONS - [- ] process > VIROPROFILER:MULTIQC - Error executing process > 'VIROPROFILER:VIRALHOST_IPHOP'

Caused by: Process VIROPROFILER:VIRALHOST_IPHOP terminated with an error exit status (1)

Command executed:

iphop predict --fa_file putative_vcontigs_pref1.fasta --out_dir out_iphop --db_dir /home/server/viroprofiler/iphop/Sept_2021_pub --num_threads 12

cat <<-END_VERSIONS > versions.yml "VIROPROFILER:VIRALHOST_IPHOP": iPHoP: $(iphop --version | head -n1 | sed 's/iPHoP v//;s/: .*//') seqkit: $( seqkit | sed '3!d; s/Version: //' ) END_VERSIONS

Command exit status: 1

Command output:

Welcome to iPHoP

Command error: Looks like everything is now set up, we will first clean up the input file, and then we will start the host prediction steps themselves [1/1/Run] Running blastn against genomes... [1/3/Run] Get relevant blast matches... The blastn output file was empty, this may be ok, but is still unusual, so you may want to check the blastn log (Wdir/blastgenomes.tsv.log) [2/1/Run] Running blastn against CRISPR... [2/2/Run] Get relevant crispr matches... The blastn output file for the CRISPR spacers was empty, this may be ok, but is still unusual, so you may want to check the blastn log (Wdir/blastcrispr.tsv.log) [3/1/Run] Running WIsH... [3/2/Run] Get relevant WIsH hits...

Welcome to iPHoP

Traceback (most recent call last): File "/opt/conda/bin/iphop", line 10, in sys.exit(cli()) File "/opt/conda/lib/python3.8/site-packages/iphop/iphop.py", line 121, in cli args"func" File "/opt/conda/lib/python3.8/site-packages/iphop/modules/master_predict.py", line 79, in main wish.run_and_parse_wish(args) File "/opt/conda/lib/python3.8/site-packages/iphop/modules/wish.py", line 48, in run_and_parse_wish get_wish_results(args["fasta_file"],args["wishrawresult"],args["wishparsed"],args['messages']) File "/opt/conda/lib/python3.8/site-packages/iphop/modules/wish.py", line 61, in get_wish_results with open(pred_file, newline='') as csvfile: FileNotFoundError: [Errno 2] No such file or directory: 'out_iphop/Wdir/wish_results/prediction.list'

Work dir: /media/server/95d691e7-0d78-417e-b796-a49c36203e5e/work/f8/2313a8ee29316f73333d652253d0fe

Tip: you can replicate the issue by changing to the process work dir and entering the command bash .command.run

I checked the wish.log: ERROR:Cannot open model directory /home/server/viroprofiler/iphop/Sept_2021_pub/db/wish_models.

So there is something missing during the setup.

Any further step that I can manually put this module under the db of iphop?

Best, Xichuan

rujinlong commented 1 year ago

It seems the iphop database is not setup successfully. In the future version, we will add a step to check the completeness of the database. Currently you can manually download the database using the following command,

 wget -c wget https://portal.nersc.gov/cfs/m342/iphop/db/iPHoP_db_Sept21.tar.gz

and then uncompress and put the content to viroprofiler folder. Please let me know if this helps.

XC-Zhai commented 1 year ago

It seems the iphop database is not setup successfully. In the future version, we will add a step to check the completeness of the database. Currently you can manually download the database using the following command,

 wget -c wget https://portal.nersc.gov/cfs/m342/iphop/db/iPHoP_db_Sept21.tar.gz

and then uncompress and put the content to viroprofiler folder. Please let me know if this helps.

I am having the one issue with all the database, previous I used the default setup that database will be downloaded to the $HOME/viroprofiler directory, but now I moved all the database to my work directory due to the huge size of iphop. When I rerun the test pipeline, I got error from the assembling step, and I tried to use: nextflow run deng-lab/viroprofiler -r main -profile singularity --mode "setup" --db /media/server/95d691e7-0d78-417e-b796-a49c36203e5e, but I got error: Command error: mkdir: cannot create directory ‘/media/server/95d691e7-0d78-417e-b796-a49c36203e5e/vibrant’: Read-only file system .

Do you have any suggestion for the configuration.

rujinlong commented 1 year ago

Are /media/server/95d691e7-0d78-417e-b796-a49c36203e5e and your HOME directory in the same hard disk or partition? If not, you need to bind this directory to singularity. see an example in the custom.config file, https://github.com/deng-lab/viroprofiler/blob/222364d5b7416c5aab37045de8bcb97dbded282d/custom.config#L5

mhmism commented 1 year ago

It was not successful for me by adjusting the bind directory for singularity. Can you possibly elaborate more if you have a different directory on another partition?