search

geraldinepascal / FROGS

FROGS is a galaxy/CLI workflow designed to produce an OTUs count matrix from high depth sequencing amplicon data.
GNU General Public License v3.0
24 stars 22 forks source link

Error in preprocess execution #27

Closed najouamghazli closed 6 years ago

najouamghazli commented 6 years ago

Hello community,

I'm still new to FROGS, and I'm using the command lines version but when I try to run the preprocess command I got an error:

Here is the command:

/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py illumina --min-amplicon-size 260 --max-amplicon-size 340 --five-prim-primer CAGCMGCCGCGGTAA --three-prim-primer GGATTAGATACCCBDGTAGTC --R1-size 250 --R2-size 250 --expected-amplicon-size 300 --input-archive /home/frogs-metagenomics/Desktop/Data.tar.gz --output-dereplicated OUT_FROGS/01-prepro.fasta --output-count OUT_FROGS/01-prepro.tsv --summary OUT_FROGS/01-prepro.html --log-file OUT_FROGS/01-prepro.log --nb-cpus 8

And here is the output error:

Process Process-5:
Process Process-7:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Process Process-8:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 604, in filter_process_multiples_files
Process Process-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 604, in filter_process_multiples_files
    process_sample( R1_files[idx], R2_files[idx], samples_names[idx], out_files[idx], lengths_files[idx], log_files[idx], args )
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 643, in process_sample
    flash_cmd.submit(log_file)
  File "/home/frogs-metagenomics/FROGS-2.0.0/lib/frogsUtils.py", line 135, in submit
    FH_log.write( '# ' + self.description + ' (' + os.path.basename(self.program) + ' version : ' + self.get_version() + ')\n' )
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 81, in get_version
    return Cmd.get_version(self, 'stdout').split()[1].strip()
IndexError: list index out of range
    process_sample( R1_files[idx], R2_files[idx], samples_names[idx], out_files[idx], lengths_files[idx], log_files[idx], args )
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 643, in process_sample
    flash_cmd.submit(log_file)
  File "/home/frogs-metagenomics/FROGS-2.0.0/lib/frogsUtils.py", line 135, in submit
    FH_log.write( '# ' + self.description + ' (' + os.path.basename(self.program) + ' version : ' + self.get_version() + ')\n' )
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 81, in get_version
    return Cmd.get_version(self, 'stdout').split()[1].strip()
IndexError: list index out of range
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 604, in filter_process_multiples_files
    process_sample( R1_files[idx], R2_files[idx], samples_names[idx], out_files[idx], lengths_files[idx], log_files[idx], args )
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 643, in process_sample
    flash_cmd.submit(log_file)
  File "/home/frogs-metagenomics/FROGS-2.0.0/lib/frogsUtils.py", line 135, in submit
    FH_log.write( '# ' + self.description + ' (' + os.path.basename(self.program) + ' version : ' + self.get_version() + ')\n' )
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 81, in get_version
    return Cmd.get_version(self, 'stdout').split()[1].strip()
IndexError: list index out of range
Process Process-3:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 604, in filter_process_multiples_files
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
Process Process-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 604, in filter_process_multiples_files
    process_sample( R1_files[idx], R2_files[idx], samples_names[idx], out_files[idx], lengths_files[idx], log_files[idx], args )
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 643, in process_sample
    flash_cmd.submit(log_file)
  File "/home/frogs-metagenomics/FROGS-2.0.0/lib/frogsUtils.py", line 135, in submit
    FH_log.write( '# ' + self.description + ' (' + os.path.basename(self.program) + ' version : ' + self.get_version() + ')\n' )
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 81, in get_version
    return Cmd.get_version(self, 'stdout').split()[1].strip()
IndexError: list index out of range
    process_sample( R1_files[idx], R2_files[idx], samples_names[idx], out_files[idx], lengths_files[idx], log_files[idx], args )
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 643, in process_sample
    flash_cmd.submit(log_file)
  File "/home/frogs-metagenomics/FROGS-2.0.0/lib/frogsUtils.py", line 135, in submit
    FH_log.write( '# ' + self.description + ' (' + os.path.basename(self.program) + ' version : ' + self.get_version() + ')\n' )
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 81, in get_version
    return Cmd.get_version(self, 'stdout').split()[1].strip()
IndexError: list index out of range
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 604, in filter_process_multiples_files
    process_sample( R1_files[idx], R2_files[idx], samples_names[idx], out_files[idx], lengths_files[idx], log_files[idx], args )
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 643, in process_sample
    flash_cmd.submit(log_file)
  File "/home/frogs-metagenomics/FROGS-2.0.0/lib/frogsUtils.py", line 135, in submit
    FH_log.write( '# ' + self.description + ' (' + os.path.basename(self.program) + ' version : ' + self.get_version() + ')\n' )
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 81, in get_version
    return Cmd.get_version(self, 'stdout').split()[1].strip()
IndexError: list index out of range
    self._target(*self._args, **self._kwargs)
Process Process-6:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 604, in filter_process_multiples_files
    process_sample( R1_files[idx], R2_files[idx], samples_names[idx], out_files[idx], lengths_files[idx], log_files[idx], args )
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 643, in process_sample
    flash_cmd.submit(log_file)
  File "/home/frogs-metagenomics/FROGS-2.0.0/lib/frogsUtils.py", line 135, in submit
    FH_log.write( '# ' + self.description + ' (' + os.path.basename(self.program) + ' version : ' + self.get_version() + ')\n' )
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 81, in get_version
    return Cmd.get_version(self, 'stdout').split()[1].strip()
IndexError: list index out of range
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 604, in filter_process_multiples_files
    process_sample( R1_files[idx], R2_files[idx], samples_names[idx], out_files[idx], lengths_files[idx], log_files[idx], args )
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 643, in process_sample
    flash_cmd.submit(log_file)
  File "/home/frogs-metagenomics/FROGS-2.0.0/lib/frogsUtils.py", line 135, in submit
    FH_log.write( '# ' + self.description + ' (' + os.path.basename(self.program) + ' version : ' + self.get_version() + ')\n' )
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 81, in get_version
    return Cmd.get_version(self, 'stdout').split()[1].strip()
IndexError: list index out of range
Traceback (most recent call last):
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 885, in <module>
    process( args )
  File "/home/frogs-metagenomics/FROGS-2.0.0//app/preprocess.py", line 733, in process
    raise Exception( "Error in sub-process execution." )
Exception: Error in sub-process execution.

I'm using:

I searched around and it seems that python can't multiprocessing, but I don't really know how to solve this !

PS: I'm not using the Galaxy version because my data is heavier than the allocated memory.

Your feedback is appreciated, thank you.

mariabernard commented 6 years ago

Hello,

The error occur when the first command is submitted. It seems that it fail to get the version of flash.

Before using FROGS on you own data, did you test your installation with the script test.sh

 cd <FROGS_PATH>/test
 bash test.sh <FROGS_PATH> <NB_CPU> <JAVA_MEM> <OUT_FOLDER>

All dependancies listed in the Readme file are in your PATH or have a link in the /libexec directory ?

najouamghazli commented 6 years ago

Hello @mariabernard ,

Thank you for your reply.

I did install all the dependencies listed in the Readme file, but when I run the test command I get somehow the same output as before:

gs-metagenomics/FROGS-2.0.0 2 3 /home/frogs-metagenomics/Out_Test Step preprocess Thu Feb 15 05:49:06 EST 2018 Process Process-2: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "/home/frogs-metagenomics/FROGS-2.0.0/app/preprocess.py", line 604, in filter_process_multiples_files process_sample( R1_files[idx], R2_files[idx], samples_names[idx], out_files[idx], lengths_files[idx], log_files[idx], args ) File "/home/frogs-metagenomics/FROGS-2.0.0/app/preprocess.py", line 643, in process_sample flash_cmd.submit(log_file) File "/home/frogs-metagenomics/FROGS-2.0.0/lib/frogsUtils.py", line 135, in submit FH_log.write( '# ' + self.description + ' (' + os.path.basename(self.program) + ' version : ' + self.get_version() + ')\n' ) File "/home/frogs-metagenomics/FROGS-2.0.0/app/preprocess.py", line 81, in get_version return Cmd.get_version(self, 'stdout').split()[1].strip() IndexError: list index out of range Process Process-1: Traceback (most recent call last): File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "/home/frogs-metagenomics/FROGS-2.0.0/app/preprocess.py", line 604, in filter_process_multiples_files process_sample( R1_files[idx], R2_files[idx], samples_names[idx], out_files[idx], lengths_files[idx], log_files[idx], args ) File "/home/frogs-metagenomics/FROGS-2.0.0/app/preprocess.py", line 643, in process_sample flash_cmd.submit(log_file) File "/home/frogs-metagenomics/FROGS-2.0.0/lib/frogsUtils.py", line 135, in submit FH_log.write( '# ' + self.description + ' (' + os.path.basename(self.program) + ' version : ' + self.get_version() + ')\n' ) File "/home/frogs-metagenomics/FROGS-2.0.0/app/preprocess.py", line 81, in get_version return Cmd.get_version(self, 'stdout').split()[1].strip() IndexError: list index out of range Traceback (most recent call last): File "/home/frogs-metagenomics/FROGS-2.0.0/app/preprocess.py", line 885, in <module> process( args ) File "/home/frogs-metagenomics/FROGS-2.0.0/app/preprocess.py", line 733, in process raise Exception( "Error in sub-process execution." ) Exception: Error in sub-process execution.

mariabernard commented 6 years ago

ok.

could you please give me the results of:

# if you linked binary dependancies in <FROGS_PATH>/libexec
export PATH=/home/frogs-metagenomics/FROGS-2.0.0/libexec:$PATH

# then
flash --version
najouamghazli commented 6 years ago

The command

export PATH=/home/frogs-metagenomics/FROGS-2.0.0/libexec:$PATH

Doesn't have an output

flash --version

No command 'flash' found, did you mean: Command 'klash' from package 'klash' (universe) Command 'flasm' from package 'flasm' (universe) flash: command not found

I don't know why, even though I did install flash 1.2.11 !

mariabernard commented 6 years ago

Ok.

Let me know if you encountered other problem after installing Flash.

najouamghazli commented 6 years ago

I just found out that flash wasn't in my $PATH

Here is the output of

flash --version `FLASH v1.2.11 Copyright (C) 2012 Tanja Magoc Copyright (C) 2012, 2013, 2014 Eric Biggers License GPLv3+; GNU GPL version 3 or later http://gnu.org/licenses/gpl.html. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

Report bugs to flash.comment@gmail.com or https://sourceforge.net/p/flashpage/bugs `

But I'm now having problem with removing chimera when running the command

bash test/test.sh 

Step preprocess Thu Feb 15 17:19:45 EST 2018
Step clustering Thu Feb 15 17:21:24 EST 2018
Step remove_chimera Thu Feb 15 17:22:21 EST 2018
Process Process-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/frogs-metagenomics/FROGS-2.0.0/libexec/parallelChimera.py", line 369, in chimera
    chimera_by_sample( sample_names[idx], input_fasta, input_abund, outputs_fasta[idx], outputs_chimera[idx], log_chimera[idx], user_size_separator )
  File "/home/frogs-metagenomics/FROGS-2.0.0/libexec/parallelChimera.py", line 411, in chimera_by_sample
    submit_cmd( ["vsearch", "--uchime_denovo", tmp_fasta, "--nonchimeras", output_fasta, "--uchimeout", tmp_log], tmp_stdout, tmp_stderr )
  File "/home/frogs-metagenomics/FROGS-2.0.0/libexec/parallelChimera.py", line 152, in submit_cmd
    p = Popen(cmd, stdout=PIPE, stderr=PIPE)
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/frogs-metagenomics/FROGS-2.0.0/libexec/parallelChimera.py", line 369, in chimera
    chimera_by_sample( sample_names[idx], input_fasta, input_abund, outputs_fasta[idx], outputs_chimera[idx], log_chimera[idx], user_size_separator )
  File "/home/frogs-metagenomics/FROGS-2.0.0/libexec/parallelChimera.py", line 411, in chimera_by_sample
    submit_cmd( ["vsearch", "--uchime_denovo", tmp_fasta, "--nonchimeras", output_fasta, "--uchimeout", tmp_log], tmp_stdout, tmp_stderr )
  File "/home/frogs-metagenomics/FROGS-2.0.0/libexec/parallelChimera.py", line 152, in submit_cmd
    p = Popen(cmd, stdout=PIPE, stderr=PIPE)
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
Traceback (most recent call last):
  File "/home/frogs-metagenomics/FROGS-2.0.0/app/remove_chimera.py", line 212, in <module>
    ParallelChimera( args.input_fasta, args.input_biom, args.non_chimera, args.out_abundance, tmp_chimera_summary, "biom", args.nb_cpus, tmp_log, size_separator ).submit( args.log_file )
  File "/home/frogs-metagenomics/FROGS-2.0.0/lib/frogsUtils.py", line 140, in submit
    subprocess.check_output( self.get_cmd(), shell=True )
  File "/usr/lib/python2.7/subprocess.py", line 574, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command 'parallelChimera.py --lenient-filter --nb-cpus 2 --sequences /home/frogs-metagenomics/Out_Test/02-clustering.fasta --biom /home/frogs-metagenomics/Out_Test/02-clustering.biom --non-chimera /home/frogs-metagenomics/Out_Test/03-chimera.fasta --out-abundance /home/frogs-metagenomics/Out_Test/03-chimera.biom --summary /home/frogs-metagenomics/Out_Test/1518733342.04_30118_03-chimera.fasta_summary.tsv --log-file /home/frogs-metagenomics/Out_Test/1518733342.04_30118_03-chimera.fasta_tmp.log' returned non-zero exit status 1
Error in remove_chimera
mariabernard commented 6 years ago

Good morning,

Could you :

See you later.

mariabernard commented 6 years ago

feel free to reopen it if you have some troubles