search

bonsai-team / matam

Mapping-Assisted Targeted-Assembly for Metagenomics
GNU Affero General Public License v3.0
19 stars 9 forks source link

Components multiprocessed assembly stalls if sga binary is not found #81

Closed ppericard closed 4 years ago

ppericard commented 4 years ago

If SGA binary is not available, MATAM runs the components assembly with multiprocessing and each process fail with "No valid binary found for sga" but the main matam_assembly.py process does not stop until the user manually kills it using Ctrl-C

2019-07-22 14:56:13,913 - root - INFO - === Contigs assembly ===
2019-07-22 14:56:13,914 - components_assembly - INFO - Save components to fastq files
2019-07-22 14:56:13,914 - components_assembly - DEBUG - Reading read-->component from /media/data/test_matam/matam_assembly/workdir/16sp.art_HS25_pe_100bp_50x.sortmerna_vs_SILVA_128_SSURef_NR95_b10_m10.scr_filt_ge
o_90pct.ovgb_i100_o50.cpts_N1_E1.read_metanode_component.tab
2019-07-22 14:56:13,929 - components_assembly - DEBUG - Storing reads by component from /media/data/test_matam/matam_assembly/workdir/16sp.art_HS25_pe_100bp_50x.sortmerna_vs_SILVA_128_SSURef_NR95_b10_m10.fq
2019-07-22 14:56:14,035 - components_assembly - INFO - Assemble components
2019-07-22 14:56:14,053 - components_assembly - DEBUG - Assembling: /media/data/test_matam/matam_assembly/workdir/components_assembly/component1_reads.fq
2019-07-22 14:56:14,053 - components_assembly - DEBUG - Assembling: /media/data/test_matam/matam_assembly/workdir/components_assembly/component33_reads.fq
2019-07-22 14:56:14,053 - components_assembly - DEBUG - Assembling: /media/data/test_matam/matam_assembly/workdir/components_assembly/component10_reads.fq
2019-07-22 14:56:14,053 - components_assembly - DEBUG - Assembling: /media/data/test_matam/matam_assembly/workdir/components_assembly/component36_reads.fq
2019-07-22 14:56:14,054 - components_assembly - DEBUG - Assembling: /media/data/test_matam/matam_assembly/workdir/components_assembly/component8_reads.fq
2019-07-22 14:56:14,054 - components_assembly - DEBUG - Assembling: /media/data/test_matam/matam_assembly/workdir/components_assembly/component21_reads.fq
No valid binary found for sga
No valid binary found for sga
No valid binary found for sga
No valid binary found for sga
No valid binary found for sga
No valid binary found for sga
2019-07-22 14:56:14,155 - components_assembly - DEBUG - Assembling: /media/data/test_matam/matam_assembly/workdir/components_assembly/component22_reads.fq
No valid binary found for sga
2019-07-22 14:56:14,157 - components_assembly - DEBUG - Assembling: /media/data/test_matam/matam_assembly/workdir/components_assembly/component12_reads.fq
No valid binary found for sga
2019-07-22 14:56:14,159 - components_assembly - DEBUG - Assembling: /media/data/test_matam/matam_assembly/workdir/components_assembly/component48_reads.fq
No valid binary found for sga
2019-07-22 14:56:14,162 - components_assembly - DEBUG - Assembling: /media/data/test_matam/matam_assembly/workdir/components_assembly/component54_reads.fq
2019-07-22 14:56:14,163 - components_assembly - DEBUG - Assembling: /media/data/test_matam/matam_assembly/workdir/components_assembly/component56_reads.fq
No valid binary found for sga
No valid binary found for sga
2019-07-22 14:56:14,166 - components_assembly - DEBUG - Assembling: /media/data/test_matam/matam_assembly/workdir/components_assembly/component60_reads.fq
No valid binary found for sga
2019-07-22 14:56:14,267 - components_assembly - DEBUG - Assembling: /media/data/test_matam/matam_assembly/workdir/components_assembly/component70_reads.fq
No valid binary found for sga
2019-07-22 14:56:14,269 - components_assembly - DEBUG - Assembling: /media/data/test_matam/matam_assembly/workdir/components_assembly/component103_reads.fq
No valid binary found for sga
2019-07-22 14:56:14,271 - components_assembly - DEBUG - Assembling: /media/data/test_matam/matam_assembly/workdir/components_assembly/component109_reads.fq
No valid binary found for sga
2019-07-22 14:56:14,273 - components_assembly - DEBUG - Assembling: /media/data/test_matam/matam_assembly/workdir/components_assembly/component88_reads.fq
No valid binary found for sga
2019-07-22 14:56:14,275 - components_assembly - DEBUG - Assembling: /media/data/test_matam/matam_assembly/workdir/components_assembly/component82_reads.fq
No valid binary found for sga
2019-07-22 14:56:14,277 - components_assembly - DEBUG - Assembling: /media/data/test_matam/matam_assembly/workdir/components_assembly/component86_reads.fq
No valid binary found for sga
2019-07-22 14:56:14,379 - components_assembly - DEBUG - Assembling: /media/data/test_matam/matam_assembly/workdir/components_assembly/component77_reads.fq
No valid binary found for sga
2019-07-22 14:56:14,381 - components_assembly - DEBUG - Assembling: /media/data/test_matam/matam_assembly/workdir/components_assembly/component111_reads.fq
No valid binary found for sga
2019-07-22 14:56:14,383 - components_assembly - DEBUG - Assembling: /media/data/test_matam/matam_assembly/workdir/components_assembly/component118_reads.fq
No valid binary found for sga
2019-07-22 14:56:14,385 - components_assembly - DEBUG - Assembling: /media/data/test_matam/matam_assembly/workdir/components_assembly/component125_reads.fq
No valid binary found for sga
2019-07-22 14:56:14,387 - components_assembly - DEBUG - Assembling: /media/data/test_matam/matam_assembly/workdir/components_assembly/component131_reads.fq
No valid binary found for sga
Process ForkPoolWorker-28:
Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/ubuntu/miniconda3/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/miniconda3/lib/python3.7/multiprocessing/pool.py", line 110, in worker
    task = get()
  File "/home/ubuntu/miniconda3/lib/python3.7/multiprocessing/queues.py", line 351, in get
    with self._rlock:
  File "/home/ubuntu/miniconda3/lib/python3.7/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
Process ForkPoolWorker-29:
Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/ubuntu/miniconda3/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/miniconda3/lib/python3.7/multiprocessing/pool.py", line 110, in worker
    task = get()
  File "/home/ubuntu/miniconda3/lib/python3.7/multiprocessing/queues.py", line 351, in get
    with self._rlock:
  File "/home/ubuntu/miniconda3/lib/python3.7/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
Process ForkPoolWorker-26:
Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/ubuntu/miniconda3/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/miniconda3/lib/python3.7/multiprocessing/pool.py", line 110, in worker
    task = get()
  File "/home/ubuntu/miniconda3/lib/python3.7/multiprocessing/queues.py", line 351, in get
    with self._rlock:
  File "/home/ubuntu/miniconda3/lib/python3.7/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
Process ForkPoolWorker-25:
Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/ubuntu/miniconda3/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/miniconda3/lib/python3.7/multiprocessing/pool.py", line 110, in worker
    task = get()
  File "/home/ubuntu/miniconda3/lib/python3.7/multiprocessing/queues.py", line 351, in get
    with self._rlock:
  File "/home/ubuntu/miniconda3/lib/python3.7/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
KeyboardInterrupt
...
ppericard commented 4 years ago

In any case, any error encountered by the multiprocess threads should be raised and stop the main process