search

cschin / Peregrine

Peregrine: Fast Genome Assembler Using SHIMMER Index
Other
99 stars 9 forks source link

Pypeflow issues #10

Closed kautto closed 5 years ago

kautto commented 5 years ago

We're running the docker image using singularity, but I don't think the issue I'm running into is related to that. It works fine when we used some simulated reads for it. So I tested it on the test FASTA file (K12MG1655.fa), and it ends up crashing in step 4 (4-cns):

INFO:pypeflow.simple_pwatcher_bridge:(slept for another 25.20000000000001s -- another 8 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 36.000000000000014s -- another 9 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 49.499999999999986s -- another 10 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 65.99999999999994s -- another 11 loop iterations)
ERROR:pypeflow.simple_pwatcher_bridge:Task Node(output/4-cns/map-27) failed with exit-code=1
ERROR:pypeflow.simple_pwatcher_bridge:Some tasks are recently_done but not satisfied: {Node(output/4-cns/map-27)}
ERROR:pypeflow.simple_pwatcher_bridge:ready: set()
        submitted: {Node(output/4-cns/map-01), Node(output/4-cns/map-24), Node(output/4-cns/map-11), Node(output/4-cns/map-16), Node(output/4-cns/map-06), Node(output/4-cns/map-22), Node(output/4-cns/map-05), Node(output/4-cns/map-29), Node(output/4-cns/map-03), Node(output/4-cns/map-32), Node(output/4-cns/map-17), Node(output/4-cns/map-13), Node(output/4-cns/map-08), Node(output/4-cns/map-15), Node(output/4-cns/map-30), Node(output/4-cns/map-23), Node(output/4-cns/map-12), Node(output/4-cns/map-07), Node(output/4-cns/map-31), Node(output/4-cns/map-04), Node(output/4-cns/map-25), Node(output/4-cns/map-14), Node(output/4-cns/map-21), Node(output/4-cns/map-20), Node(output/4-cns/map-18), Node(output/4-cns/map-09), Node(output/4-cns/map-02), Node(output/4-cns/map-10), Node(output/4-cns/map-26), Node(output/4-cns/map-28), Node(output/4-cns/map-19)}
ERROR:pwatcher.blocking:Noop. We cannot kill blocked threads. Hopefully, everything will die on SIGTERM.
Traceback (most recent call last):
  File "/opt/conda/envs/peregrine/bin/pg_run.py", line 4, in <module>
    __import__('pkg_resources').run_script('peregrine==0+unknown', 'pg_run.py')
  File "/opt/conda/envs/peregrine/lib/python3.7/site-packages/pkg_resources/__init__.py", line 666, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/opt/conda/envs/peregrine/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1453, in run_script
    exec(code, namespace, namespace)
  File "/opt/conda/envs/peregrine/lib/python3.7/site-packages/peregrine-0+unknown-py3.7-linux-x86_64.egg/EGG-INFO/scripts/pg_run.py", line 651, in <module>
    main(args)
  File "/opt/conda/envs/peregrine/lib/python3.7/site-packages/peregrine-0+unknown-py3.7-linux-x86_64.egg/EGG-INFO/scripts/pg_run.py", line 595, in main
    ctg_out)
  File "/opt/conda/envs/peregrine/lib/python3.7/site-packages/peregrine-0+unknown-py3.7-linux-x86_64.egg/EGG-INFO/scripts/pg_run.py", line 467, in run_cns
    wf.refreshTargets()
  File "/opt/conda/envs/peregrine/lib/python3.7/site-packages/pypeflow-2.1.1+git.d63b0e79f5a7b2d370b7de84a890f88271afa476-py3.7.egg/pypeflow/simple_pwatcher_bridge.py", line 278, in refreshTargets
    self._refreshTargets(updateFreq, exitOnFailure)
  File "/opt/conda/envs/peregrine/lib/python3.7/site-packages/pypeflow-2.1.1+git.d63b0e79f5a7b2d370b7de84a890f88271afa476-py3.7.egg/pypeflow/simple_pwatcher_bridge.py", line 362, in _refreshTargets
    raise Exception(msg)
Exception: Some tasks are recently_done but not satisfied: {Node(output/4-cns/map-27)}

Looking at the stderr file from that folder, the error appears to be:

shmr_map: shmr_map.c:88: process_map: Assertion `ref_mmers->n > 0' failed.
Command terminated by signal 6
0.01user 0.03system 4:20.39elapsed 0%CPU (0avgtext+0avgdata 3032maxresident)k
0inputs+4880outputs (0major+803minor)pagefaults 0swaps
2019-06-11 16:29:50,792 - root - WARNING - Call '/bin/bash user_script.sh' returned 34304.
2019-06-11 16:29:50,792 - root - WARNING - CD: '/igm/projects/KAUTTO_Esko/test/peregrine2/output/4-cns/map-27' -> '/igm/projects/KAUTTO_Esko/test/peregrine2/output/4-cns/map-27'
2019-06-11 16:29:51,161 - root - WARNING - CD: '/igm/projects/KAUTTO_Esko/test/peregrine2/output/4-cns/map-27' -> '/igm/projects/KAUTTO_Esko/test/peregrine2/output/4-cns/map-27'
2019-06-11 16:29:51,161 - root - CRITICAL - Error in /opt/conda/envs/peregrine/lib/python3.7/site-packages/pypeflow-2.1.1+git.d63b0e79f5a7b2d370b7de84a890f88271afa476-py3.7.egg/pypeflow/do_task.py with args="{'json_fn': '/igm/projects/KAUTTO_Esko/test/peregrine2/output/4-cns/map-27/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
Traceback (most recent call last):
  File "/opt/conda/envs/peregrine/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/envs/peregrine/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/peregrine/lib/python3.7/site-packages/pypeflow-2.1.1+git.d63b0e79f5a7b2d370b7de84a890f88271afa476-py3.7.egg/pypeflow/do_task.py", line 267, in <module>
    main()
  File "/opt/conda/envs/peregrine/lib/python3.7/site-packages/pypeflow-2.1.1+git.d63b0e79f5a7b2d370b7de84a890f88271afa476-py3.7.egg/pypeflow/do_task.py", line 259, in main
    run(**vars(parsed_args))
  File "/opt/conda/envs/peregrine/lib/python3.7/site-packages/pypeflow-2.1.1+git.d63b0e79f5a7b2d370b7de84a890f88271afa476-py3.7.egg/pypeflow/do_task.py", line 253, in run
    run_cfg_in_tmpdir(cfg, tmpdir, '.')
  File "/opt/conda/envs/peregrine/lib/python3.7/site-packages/pypeflow-2.1.1+git.d63b0e79f5a7b2d370b7de84a890f88271afa476-py3.7.egg/pypeflow/do_task.py", line 228, in run_cfg_in_tmpdir
    run_bash(bash_template, myinputs, myoutputs, parameters)
  File "/opt/conda/envs/peregrine/lib/python3.7/site-packages/pypeflow-2.1.1+git.d63b0e79f5a7b2d370b7de84a890f88271afa476-py3.7.egg/pypeflow/do_task.py", line 187, in run_bash
    util.system(cmd)
  File "/opt/conda/envs/peregrine/lib/python3.7/site-packages/pypeflow-2.1.1+git.d63b0e79f5a7b2d370b7de84a890f88271afa476-py3.7.egg/pypeflow/io.py", line 29, in syscall
    raise Exception(msg)
Exception: Call '/bin/bash user_script.sh' returned 34304.

Since all the files are inside the docker image, it's kind of hard to fish out what could be causing it.. would you have any quick insights into how to fix the issue?

kautto commented 5 years ago

Never mind - realized that's a complete assembly, not a set of test reads.

smoretti commented 4 years ago

I got the same error on my side with Peregrine version 0.1.5.3 and the command pg_run.py asm reads_for_tmp.lst 12 12 12 12 12 12 12 12 12 --with-consensus --shimmer-r 3 --best_n_ovlp 8 --output ref_r4

+ /usr/bin/time shmr_map -r /stn4/ref_r4/4-cns/p_ctg_index/p_ctg -m /stn4/ref_r4/4-cns/p_ctg_index/p_ctg-L2 -p /stn4/ref_r4/0-seqdb/seq_dataset -l /stn4/ref_r4/1-index/shmr-L2 -t 12 -c 10
using ref index file: /stn4/ref_r4/4-cns/p_ctg_index/p_ctg.idx
using ref seqdb file: /stn4/ref_r4/4-cns/p_ctg_index/p_ctg.seqdb
using ref shimmer data file: /stn4/ref_r4/4-cns/p_ctg_index/p_ctg-L2-01-of-01.dat
number of shimmers load: 0
using index file: /stn4/ref_r4/0-seqdb/seq_dataset.idx
using seqdb file: /stn4/ref_r4/0-seqdb/seq_dataset.seqdb
using shimmer data file: /stn4/ref_r4/1-index/shmr-L2-01-of-12.dat
number of shimmers load: 10658981
using shimmer data file: /stn4/ref_r4/1-index/shmr-L2-02-of-12.dat
number of shimmers load: 10728569
using shimmer data file: /stn4/ref_r4/1-index/shmr-L2-03-of-12.dat
number of shimmers load: 10729637
using shimmer data file: /stn4/ref_r4/1-index/shmr-L2-04-of-12.dat
number of shimmers load: 10716296
using shimmer data file: /stn4/ref_r4/1-index/shmr-L2-05-of-12.dat
number of shimmers load: 10753230
using shimmer data file: /stn4/ref_r4/1-index/shmr-L2-06-of-12.dat
number of shimmers load: 10777522
using shimmer data file: /stn4/ref_r4/1-index/shmr-L2-07-of-12.dat
number of shimmers load: 10730557
using shimmer data file: /stn4/ref_r4/1-index/shmr-L2-08-of-12.dat
number of shimmers load: 10793331
using shimmer data file: /stn4/ref_r4/1-index/shmr-L2-09-of-12.dat
number of shimmers load: 10666638
using shimmer data file: /stn4/ref_r4/1-index/shmr-L2-10-of-12.dat
number of shimmers load: 10723513
using shimmer data file: /stn4/ref_r4/1-index/shmr-L2-11-of-12.dat
number of shimmers load: 10688769
using shimmer data file: /stn4/ref_r4/1-index/shmr-L2-12-of-12.dat
number of shimmers load: 10772298
using shimmer count file: /stn4/ref_r4/1-index/shmr-L2-MC-01-of-12.dat
using shimmer count file: /stn4/ref_r4/1-index/shmr-L2-MC-02-of-12.dat
using shimmer count file: /stn4/ref_r4/1-index/shmr-L2-MC-03-of-12.dat
using shimmer count file: /stn4/ref_r4/1-index/shmr-L2-MC-04-of-12.dat
using shimmer count file: /stn4/ref_r4/1-index/shmr-L2-MC-05-of-12.dat
using shimmer count file: /stn4/ref_r4/1-index/shmr-L2-MC-06-of-12.dat
using shimmer count file: /stn4/ref_r4/1-index/shmr-L2-MC-07-of-12.dat
using shimmer count file: /stn4/ref_r4/1-index/shmr-L2-MC-08-of-12.dat
using shimmer count file: /stn4/ref_r4/1-index/shmr-L2-MC-09-of-12.dat
using shimmer count file: /stn4/ref_r4/1-index/shmr-L2-MC-10-of-12.dat
using shimmer count file: /stn4/ref_r4/1-index/shmr-L2-MC-11-of-12.dat
using shimmer count file: /stn4/ref_r4/1-index/shmr-L2-MC-12-of-12.dat
shmr_map: shmr_map.c:88: process_map: Assertion `ref_mmers->n > 0' failed.
Command terminated by signal 6
37.98user 6.54system 1:45.28elapsed 42%CPU (0avgtext+0avgdata 4394756maxresident)k
0inputs+0outputs (0major+1634644minor)pagefaults 0swaps
2019-10-04 12:15:42,239 - root - WARNING - Call '/bin/bash user_script.sh' returned 34304.
2019-10-04 12:15:42,239 - root - WARNING - CD: '/stn4/ref_r4/4-cns/map-10' -> '/stn4/ref_r4/4-cns/map-10'
2019-10-04 12:15:42,239 - root - WARNING - CD: '/stn4/ref_r4/4-cns/map-10' -> '/stn4/ref_r4/4-cns/map-10'
2019-10-04 12:15:42,240 - root - CRITICAL - Error in /usr/local/lib/python3.6/dist-packages/pypeflow-2.1.1+git.d63b0e79f5a7b2d370b7de84a890f88271afa476-py3.6.egg/pypeflow/do_task.py with args="{'json_fn': '/stn4/ref_r4/4-cns/map-10/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.6/dist-packages/pypeflow-2.1.1+git.d63b0e79f5a7b2d370b7de84a890f88271afa476-py3.6.egg/pypeflow/do_task.py", line 267, in <module>
    main()
  File "/usr/local/lib/python3.6/dist-packages/pypeflow-2.1.1+git.d63b0e79f5a7b2d370b7de84a890f88271afa476-py3.6.egg/pypeflow/do_task.py", line 259, in main
    run(**vars(parsed_args))
  File "/usr/local/lib/python3.6/dist-packages/pypeflow-2.1.1+git.d63b0e79f5a7b2d370b7de84a890f88271afa476-py3.6.egg/pypeflow/do_task.py", line 253, in run
    run_cfg_in_tmpdir(cfg, tmpdir, '.')
  File "/usr/local/lib/python3.6/dist-packages/pypeflow-2.1.1+git.d63b0e79f5a7b2d370b7de84a890f88271afa476-py3.6.egg/pypeflow/do_task.py", line 228, in run_cfg_in_tmpdir
    run_bash(bash_template, myinputs, myoutputs, parameters)
  File "/usr/local/lib/python3.6/dist-packages/pypeflow-2.1.1+git.d63b0e79f5a7b2d370b7de84a890f88271afa476-py3.6.egg/pypeflow/do_task.py", line 187, in run_bash
    util.system(cmd)
  File "/usr/local/lib/python3.6/dist-packages/pypeflow-2.1.1+git.d63b0e79f5a7b2d370b7de84a890f88271afa476-py3.6.egg/pypeflow/io.py", line 29, in syscall
    raise Exception(msg)
Exception: Call '/bin/bash user_script.sh' returned 34304.
+++ pwd
++ echo 'FAILURE. Running top in /stn4/ref_r4/4-cns/map-10 (If you see -terminal database is inaccessible- you are using the python bin-wrapper, so you will not get diagnostic info. No big deal. This process is crashing anyway.)'
cschin commented 4 years ago

This line shmr_map: shmr_map.c:88: process_map: Assertionref_mmers->n > 0' failed.` It seem to me your assembly generates empty contigs for consensus step. What is your input?

mictadlo commented 4 years ago

Hi, I have the similar error:

INFO:pypeflow.simple_pwatcher_bridge:(slept for another 0.0s -- another 1 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 0.30000000000000004s -- another 2 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 1.2000000000000002s -- another 3 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 2.9999999999999996s -- another 4 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 6.0s -- another 5 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 10.500000000000002s -- another 6 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 16.800000000000004s -- another 7 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 25.20000000000001s -- another 8 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 36.000000000000014s -- another 9 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 49.499999999999986s -- another 10 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 65.99999999999994s -- another 11 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 85.7999999999999s -- another 12 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 109.19999999999983s -- another 13 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 135.5999999999998s -- another 14 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 150.0s -- another 15 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 160.0s -- another 16 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 170.0s -- another 17 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 180.0s -- another 18 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 190.0s -- another 19 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 200.0s -- another 20 loop iterations)
ERROR:pypeflow.simple_pwatcher_bridge:Task Node(lab-asm-wc-default/4-cns/map-24) failed with exit-code=1
ERROR:pypeflow.simple_pwatcher_bridge:Some tasks are recently_done but not satisfied: {Node(lab-asm-wc-default/4-cns/map-24)}
ERROR:pypeflow.simple_pwatcher_bridge:ready: set()
        submitted: {Node(lab-asm-wc-default/4-cns/map-06), Node(lab-asm-wc-default/4-cns/map-19), Node(lab-asm-wc-default/4-cns/map-12), Node(lab-asm-wc-default/4-cns/map-14), Node(lab-asm-wc-default/4-cns/map-07), Node(lab-asm-wc-default/4-cns/map-16), Node(lab-asm-wc-default/4-cns/map-22), Node(lab-asm-wc-default/4-cns/map-20), Node(lab-asm-wc-default/4-cns/map-21), Node(lab-asm-wc-default/4-cns/map-02), Node(lab-asm-wc-default/4-cns/map-11), Node(lab-asm-wc-default/4-cns/map-18), Node(lab-asm-wc-default/4-cns/map-08), Node(lab-asm-wc-default/4-cns/map-03), Node(lab-asm-wc-default/4-cns/map-05), Node(lab-asm-wc-default/4-cns/map-17), Node(lab-asm-wc-default/4-cns/map-04), Node(lab-asm-wc-default/4-cns/map-10), Node(lab-asm-wc-default/4-cns/map-13), Node(lab-asm-wc-default/4-cns/map-01), Node(lab-asm-wc-default/4-cns/map-09), Node(lab-asm-wc-default/4-cns/map-15), Node(lab-asm-wc-default/4-cns/map-23)}
ERROR:pwatcher.blocking:Noop. We cannot kill blocked threads. Hopefully, everything will die on SIGTERM.
Traceback (most recent call last):
  File "/work/waterhouse_team/miniconda2/envs/peregrine/bin/pg_run.py", line 4, in <module>
    __import__('pkg_resources').run_script('peregrine==0.1.5.5', 'pg_run.py')
  File "/work/waterhouse_team/miniconda2/envs/peregrine/lib/python3.7/site-packages/pkg_resources/__init__.py", line 666, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/work/waterhouse_team/miniconda2/envs/peregrine/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1462, in run_script
    exec(code, namespace, namespace)
  File "/lustre/work-lustre/waterhouse_team/miniconda2/envs/peregrine/lib/python3.7/site-packages/peregrine-0.1.5.5-py3.7-linux-x86_64.egg/EGG-INFO/scripts/pg_run.py", line 658, in <module>
    main(args)
  File "/lustre/work-lustre/waterhouse_team/miniconda2/envs/peregrine/lib/python3.7/site-packages/peregrine-0.1.5.5-py3.7-linux-x86_64.egg/EGG-INFO/scripts/pg_run.py", line 602, in main
    ctg_out)
  File "/lustre/work-lustre/waterhouse_team/miniconda2/envs/peregrine/lib/python3.7/site-packages/peregrine-0.1.5.5-py3.7-linux-x86_64.egg/EGG-INFO/scripts/pg_run.py", line 474, in run_cns
    wf.refreshTargets()
  File "/work/waterhouse_team/miniconda2/envs/peregrine/lib/python3.7/site-packages/pypeflow-2.1.1+git.d63b0e79f5a7b2d370b7de84a890f88271afa476-py3.7.egg/pypeflow/simple_pwatcher_bridge.py", line 278, in refreshTargets
    self._refreshTargets(updateFreq, exitOnFailure)
  File "/work/waterhouse_team/miniconda2/envs/peregrine/lib/python3.7/site-packages/pypeflow-2.1.1+git.d63b0e79f5a7b2d370b7de84a890f88271afa476-py3.7.egg/pypeflow/simple_pwatcher_bridge.py", line 362, in _refreshTargets
    raise Exception(msg)
Exception: Some tasks are recently_done but not satisfied: {Node(lab-asm-wc-default/4-cns/map-24)}