search

ncbi / vadr

Viral Annotation DefineR: classification and annotation of viral sequences based on RefSeq annotation
Other
97 stars 22 forks source link

ERROR in utl_RunCommand() while executing testfile #24

Closed reymonera closed 4 years ago

reymonera commented 4 years ago

Hi,

Installing and testing VADR I got this output. Apparently it has something to do with how I'm executing the binaries.

Thanks for any answer you could give me.

# v-test.pl :: test VADR scripts [TEST SCRIPT]
# VADR 1.1.1 (July 2020)
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# date:    Wed Aug 26 02:41:29 2020
#
# test file:                                                         /home/marlen/Programas/vadr_install_dir/vadr/testfiles/noro.r10.local.testin
# output directory:                                                  vt-n10-local
# forcing directory overwrite:                                       yes [-f]
# if output files listed in testin file already exist, remove them:  yes [--rmout]
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# Parsing test file                                  ... done. [    0.0 seconds]
##teamcity[testStarted name='annotate-noro-10-local' captureStandardOutput='true']
# Running command  1 [annotate-noro-10-local]        ... /home/marlen/Programas/vadr_install_dir/infernal/binaries/esl-seqstat: 1: /home/marlen/Programas/vadr_install_dir/infernal/binaries/esl-seqstat: Syntax error: "(" unexpected

ERROR in utl_RunCommand(), the following command failed:
/home/marlen/Programas/vadr_install_dir/infernal/binaries/esl-seqstat --dna -a va-noro.r10/va-noro.r10.vadr.in.fa > va-noro.r10/va-noro.r10.vadr.seqstat

done. [    3.0 seconds]
#   checking va-noro.r10/va-noro.r10.vadr.pass.tbl                                                                ... FAIL [output file does not exist]
#   checking va-noro.r10/va-noro.r10.vadr.fail.tbl                                                                ... FAIL [output file does not exist]
#   checking va-noro.r10/va-noro.r10.vadr.sqa                                                                     ... FAIL [output file does not exist]
#   checking va-noro.r10/va-noro.r10.vadr.sqc                                                                     ... FAIL [output file does not exist]
#   checking va-noro.r10/va-noro.r10.vadr.ftr                                                                     ... FAIL [output file does not exist]
#   checking va-noro.r10/va-noro.r10.vadr.sgm                                                                     ... FAIL [output file does not exist]
#   checking va-noro.r10/va-noro.r10.vadr.mdl                                                                     ... FAIL [output file does not exist]
#   checking va-noro.r10/va-noro.r10.vadr.alt                                                                     ... FAIL [output file does not exist]
#   checking va-noro.r10/va-noro.r10.vadr.alc                                                                     ... FAIL [output file does not exist]
##teamcity[testFailed name='annotate-noro-10-local' message='v-test.pl failure']
##teamcity[testFinished name='annotate-noro-10-local']
#
#
# FAIL: 9 of 9 files were not created correctly.
#
# Output printed to screen saved in:                   vt-n10-local.vadr.log
# List of executed commands saved in:                  vt-n10-local.vadr.cmd
# List and description of all output files saved in:   vt-n10-local.vadr.list
#
# All output files created in directory ./vt-n10-local/
#
# Elapsed time:  00:00:03.02
#                hh:mm:ss
# 
[FAIL]

Regards,

ybdong919 commented 4 years ago

It should be caused by infernal binary files. One possible way to solve the issue is to re-compile infernal and replace current binary files in "binaries" with newly compiled files. The details can be found : https://github.com/nawrockie/vadr/issues/23

nawrockie commented 4 years ago

@reymonera Can you please send the output of an 'ls' command on your /home/marlen/Programas/vadr_install_dir/infernal/binaries/ directory?

ybdong919 commented 4 years ago

By the way, the binary files in "ncbi_blast/bin" probably also need to be replaced by re-compiled binary files from blast.

reymonera commented 4 years ago

Hi @nawrockie , here's the output of the ls command:

~/Programas/vadr_install_dir/infernal/binaries$ ls
cmalign      cmconvert  cmpress   cmstat      esl-alimanip  esl-alimerge  esl-alistat     esl-construct  esl-mixdchlet  esl-seqrange  esl-shuffle    esl-weight
cmbuild      cmemit     cmscan    easel       esl-alimap    esl-alipid    esl-compalign   esl-histplot   esl-reformat   esl-seqstat   esl-ssdraw
cmcalibrate  cmfetch    cmsearch  esl-afetch  esl-alimask   esl-alirev    esl-compstruct  esl-mask       esl-selectn    esl-sfetch    esl-translate

Meanwhile, I'll try what @ybdong919 has suggested.

Thanks for the responses.

nawrockie commented 4 years ago

@reymonera : Two more questions: 1) Are you running on linux or mac OS? 2) Can you please send me the output of 'ls -ltr' on your vadr_install_dir/ncbi-blast/bin? Thanks.

reymonera commented 4 years ago

Hi @nawrockie

1: Linux. Ubuntu 16.04.4 LTS to be more specific. 2: Of course:

total 408980
-rwxr-xr-x 1 marlen marlen    51345 jun 28  2010 legacy_blast.pl
-rwxr-xr-x 1 marlen marlen     4709 jun 24  2019 get_species_taxids.sh
-rwxr-xr-x 1 marlen marlen     6243 ago  7  2019 cleanup-blastdb-volumes.py
-rwxr-xr-x 1 marlen marlen    19229 nov 25  2019 update_blastdb.pl
-rwxr-xr-x 1 marlen marlen 24637636 dic  3  2019 blastp
-rwxr-xr-x 1 marlen marlen 24637492 dic  3  2019 blastn
-rwxr-xr-x 1 marlen marlen 24665536 dic  3  2019 tblastn
-rwxr-xr-x 1 marlen marlen 24637636 dic  3  2019 blastx
-rwxr-xr-x 1 marlen marlen 24637660 dic  3  2019 tblastx
-rwxr-xr-x 1 marlen marlen 24683620 dic  3  2019 psiblast
-rwxr-xr-x 1 marlen marlen 24663748 dic  3  2019 rpstblastn
-rwxr-xr-x 1 marlen marlen 24663900 dic  3  2019 rpsblast
-rwxr-xr-x 1 marlen marlen 16545344 dic  3  2019 segmasker
-rwxr-xr-x 1 marlen marlen 16703712 dic  3  2019 makembindex
-rwxr-xr-x 1 marlen marlen 16000448 dic  3  2019 dustmasker
-rwxr-xr-x 1 marlen marlen 19053148 dic  3  2019 windowmasker
-rwxr-xr-x 1 marlen marlen 17025780 dic  3  2019 makeprofiledb
-rwxr-xr-x 1 marlen marlen 17259068 dic  3  2019 makeblastdb
-rwxr-xr-x 1 marlen marlen 24765336 dic  3  2019 deltablast
-rwxr-xr-x 1 marlen marlen 15970844 dic  3  2019 convert2blastmask
-rwxr-xr-x 1 marlen marlen 24645828 dic  3  2019 blast_formatter
-rwxr-xr-x 1 marlen marlen 21657256 dic  3  2019 blastdbcmd
-rwxr-xr-x 1 marlen marlen 17100600 dic  3  2019 blastdbcheck
-rwxr-xr-x 1 marlen marlen 14698644 dic  3  2019 blastdb_aliastool

Thanks for the responses,

nawrockie commented 4 years ago

Those file sizes match the binary files sizes for ncbi-blast/bin that get installed in my hands when I run sh vadr-install.sh macosx instead of sh vadr-install.sh linux. Can you retry installation with sh vadr-install.sh linux and see if that makes a difference?

reymonera commented 4 years ago

Hi @nawrockie :

I'm recieving the same print error I had before when I try to run the test:

marlen@Diogenes:~/Programas/vadr_install_dir$ $VADRSCRIPTSDIR/testfiles/do-install-tests-local.sh
# v-test.pl :: test VADR scripts [TEST SCRIPT]
# VADR 1.1.1 (July 2020)
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# date:    Sat Aug 29 02:20:19 2020
#
# test file:                                                         /home/marlen/Programas/vadr_install_dir/vadr/testfiles/noro.r10.local.testin
# output directory:                                                  vt-n10-local
# forcing directory overwrite:                                       yes [-f]
# if output files listed in testin file already exist, remove them:  yes [--rmout]
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# Parsing test file                                  ... done. [    0.0 seconds]
##teamcity[testStarted name='annotate-noro-10-local' captureStandardOutput='true']
# Running command  1 [annotate-noro-10-local]        ... /home/marlen/Programas/vadr_install_dir/infernal/binaries/esl-seqstat: 1: /home/marlen/Programas/vadr_install_dir/infernal/binaries/esl-seqstat: Syntax error: "(" unexpected

ERROR in utl_RunCommand(), the following command failed:
/home/marlen/Programas/vadr_install_dir/infernal/binaries/esl-seqstat --dna -a va-noro.r10/va-noro.r10.vadr.in.fa > va-noro.r10/va-noro.r10.vadr.seqstat

done. [    3.2 seconds]
#   checking va-noro.r10/va-noro.r10.vadr.pass.tbl                                                                ... FAIL [output file does not exist]
#   checking va-noro.r10/va-noro.r10.vadr.fail.tbl                                                                ... FAIL [output file does not exist]
#   checking va-noro.r10/va-noro.r10.vadr.sqa                                                                     ... FAIL [output file does not exist]
#   checking va-noro.r10/va-noro.r10.vadr.sqc                                                                     ... FAIL [output file does not exist]
#   checking va-noro.r10/va-noro.r10.vadr.ftr                                                                     ... FAIL [output file does not exist]
#   checking va-noro.r10/va-noro.r10.vadr.sgm                                                                     ... FAIL [output file does not exist]
#   checking va-noro.r10/va-noro.r10.vadr.mdl                                                                     ... FAIL [output file does not exist]
#   checking va-noro.r10/va-noro.r10.vadr.alt                                                                     ... FAIL [output file does not exist]
#   checking va-noro.r10/va-noro.r10.vadr.alc                                                                     ... FAIL [output file does not exist]
##teamcity[testFailed name='annotate-noro-10-local' message='v-test.pl failure']
##teamcity[testFinished name='annotate-noro-10-local']
#
#
# FAIL: 9 of 9 files were not created correctly.
#
# Output printed to screen saved in:                   vt-n10-local.vadr.log
# List of executed commands saved in:                  vt-n10-local.vadr.cmd
# List and description of all output files saved in:   vt-n10-local.vadr.list
#
# All output files created in directory ./vt-n10-local/
#
# Elapsed time:  00:00:03.18
#                hh:mm:ss
# 
[FAIL]

ERROR, at least one test FAILed

# v-test.pl :: test VADR scripts [TEST SCRIPT]
# VADR 1.1.1 (July 2020)
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# date:    Sat Aug 29 02:20:23 2020
#
# test file:                                                         /home/marlen/Programas/vadr_install_dir/vadr/testfiles/dengue.r5.local.testin
# output directory:                                                  vt-d5-local
# forcing directory overwrite:                                       yes [-f]
# if output files listed in testin file already exist, remove them:  yes [--rmout]
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# Parsing test file                                  ... done. [    0.0 seconds]
##teamcity[testStarted name='annotate-dengue-5-local' captureStandardOutput='true']
# Running command  1 [annotate-dengue-5-local]       ... /home/marlen/Programas/vadr_install_dir/infernal/binaries/esl-seqstat: 1: /home/marlen/Programas/vadr_install_dir/infernal/binaries/esl-seqstat: Syntax error: "(" unexpected

ERROR in utl_RunCommand(), the following command failed:
/home/marlen/Programas/vadr_install_dir/infernal/binaries/esl-seqstat --dna -a va-dengue.r5/va-dengue.r5.vadr.in.fa > va-dengue.r5/va-dengue.r5.vadr.seqstat

done. [    0.7 seconds]
#   checking va-dengue.r5/va-dengue.r5.vadr.pass.tbl                                                              ... FAIL [output file does not exist]
#   checking va-dengue.r5/va-dengue.r5.vadr.fail.tbl                                                              ... FAIL [output file does not exist]
#   checking va-dengue.r5/va-dengue.r5.vadr.sqa                                                                   ... FAIL [output file does not exist]
#   checking va-dengue.r5/va-dengue.r5.vadr.sqc                                                                   ... FAIL [output file does not exist]
#   checking va-dengue.r5/va-dengue.r5.vadr.ftr                                                                   ... FAIL [output file does not exist]
#   checking va-dengue.r5/va-dengue.r5.vadr.sgm                                                                   ... FAIL [output file does not exist]
#   checking va-dengue.r5/va-dengue.r5.vadr.mdl                                                                   ... FAIL [output file does not exist]
#   checking va-dengue.r5/va-dengue.r5.vadr.alt                                                                   ... FAIL [output file does not exist]
#   checking va-dengue.r5/va-dengue.r5.vadr.alc                                                                   ... FAIL [output file does not exist]
##teamcity[testFailed name='annotate-dengue-5-local' message='v-test.pl failure']
##teamcity[testFinished name='annotate-dengue-5-local']
#
#
# FAIL: 9 of 9 files were not created correctly.
#
# Output printed to screen saved in:                   vt-d5-local.vadr.log
# List of executed commands saved in:                  vt-d5-local.vadr.cmd
# List and description of all output files saved in:   vt-d5-local.vadr.list
#
# All output files created in directory ./vt-d5-local/
#
# Elapsed time:  00:00:00.73
#                hh:mm:ss
# 
[FAIL]

ERROR, at least one test FAILed

FAIL: at least one test failed

Also, this is the result of executing ls -ltr in the bin directory:

marlen@Diogenes:~/Programas/vadr_install_dir/ncbi-blast/bin$ ls -ltr
total 408980
-rwxr-xr-x 1 marlen marlen    51345 jun 28  2010 legacy_blast.pl
-rwxr-xr-x 1 marlen marlen     4709 jun 24  2019 get_species_taxids.sh
-rwxr-xr-x 1 marlen marlen     6243 ago  7  2019 cleanup-blastdb-volumes.py
-rwxr-xr-x 1 marlen marlen    19229 nov 25  2019 update_blastdb.pl
-rwxr-xr-x 1 marlen marlen 24637636 dic  3  2019 blastp
-rwxr-xr-x 1 marlen marlen 24637492 dic  3  2019 blastn
-rwxr-xr-x 1 marlen marlen 24665536 dic  3  2019 tblastn
-rwxr-xr-x 1 marlen marlen 24637636 dic  3  2019 blastx
-rwxr-xr-x 1 marlen marlen 24637660 dic  3  2019 tblastx
-rwxr-xr-x 1 marlen marlen 24683620 dic  3  2019 psiblast
-rwxr-xr-x 1 marlen marlen 24663748 dic  3  2019 rpstblastn
-rwxr-xr-x 1 marlen marlen 24663900 dic  3  2019 rpsblast
-rwxr-xr-x 1 marlen marlen 16545344 dic  3  2019 segmasker
-rwxr-xr-x 1 marlen marlen 16703712 dic  3  2019 makembindex
-rwxr-xr-x 1 marlen marlen 16000448 dic  3  2019 dustmasker
-rwxr-xr-x 1 marlen marlen 19053148 dic  3  2019 windowmasker
-rwxr-xr-x 1 marlen marlen 17025780 dic  3  2019 makeprofiledb
-rwxr-xr-x 1 marlen marlen 17259068 dic  3  2019 makeblastdb
-rwxr-xr-x 1 marlen marlen 24765336 dic  3  2019 deltablast
-rwxr-xr-x 1 marlen marlen 15970844 dic  3  2019 convert2blastmask
-rwxr-xr-x 1 marlen marlen 24645828 dic  3  2019 blast_formatter
-rwxr-xr-x 1 marlen marlen 21657256 dic  3  2019 blastdbcmd
-rwxr-xr-x 1 marlen marlen 17100600 dic  3  2019 blastdbcheck
-rwxr-xr-x 1 marlen marlen 14698644 dic  3  2019 blastdb_aliastool

I get the same files with the same size.

Regards,

nawrockie commented 4 years ago

Thanks for doing that test @reymonera . Could you please try one more test for me? Download and execute the script vadr-install-force-linux.sh from here:

https://ftp.ncbi.nlm.nih.gov/pub/nawrocki/vadr-example-files/vadr-install-force-linux.sh

in a new directory and post the results of an ls -ltr on the ncbi-blast/bin directory again? Thanks!

reymonera commented 4 years ago

Hi @nawrockie

Finally, I could test the script you sent me. It didn't work either. For a reason, it couldn't recognize the linux option and insisted in installing the MacOSX files. So I edited it to force the linux installation, basically erasing every line that mentioned the installation of the MacOSX files.

The tests came up pretty good, although I couldn't pass any of the tests in the .sh files. This is a sample (I ran all tests instead of the local option):

##teamcity[testFinished name='annotate-dengue-5-local']
#
#
# PASS: all 9 files were created correctly.
#
# Output printed to screen saved in:                   vt-d5-local.vadr.log
# List of executed commands saved in:                  vt-d5-local.vadr.cmd
# List and description of all output files saved in:   vt-d5-local.vadr.list
#
# All output files created in directory ./vt-d5-local/
#
# Elapsed time:  00:00:34.21
#                hh:mm:ss
# 
[ok]
/home/marlen/Programas/vadr_install_dir/vadr/testfiles/do-install-tests-local.sh: 17: [: 0: unexpected operator
FAIL: at least one test failed

I searched for this error: /home/marlen/Programas/vadr_install_dir/vadr/testfiles/do-install-tests-local.sh: 17: [: 0: unexpected operator

And it seems it has to do with the sh command not supporting the == symbols. For the last test using the files in the testfile directory, I decided to just use one = and it came well. The others, however, still showed that specific error.

But besides that it seems that everything is working well and all tests passed.

Thanks for your help.

nawrockie commented 4 years ago

Thank you @reymonera ! I believe you've found the crucial issue. I will replace all '==' with '=' in shell scripts packaged with VADR in a future release. Thanks again.