search

PacificBiosciences / FALCON

FALCON: experimental PacBio diploid assembler -- Out-of-date -- Please use a binary release: https://github.com/PacificBiosciences/FALCON_unzip/wiki/Binaries
https://github.com/PacificBiosciences/FALCON_unzip/wiki/Binaries
Other
204 stars 103 forks source link

Task Node(0-rawreads/tan-runs/tan_005) failed with exit-code=1 #677

Closed amnag closed 5 years ago

amnag commented 5 years ago

I am using the following fc_run.cfg file for fc_run on two fasta files containing subreads where each fasta file is ~ 5.5 G.

[General]

input_fofn = input.fofn

input_type = raw

\# The length cutoff used for seed reads used for initial mapping
length_cutoff = 1000

\# The length cutoff used for seed reads usef for pre-assembly
length_cutoff_pr = 1000

pa_HPCdaligner_option =  -v -B128 -M32
pa_daligner_option =  -e.70 -l1000 -s1000
ovlp_HPCdaligner_option = -v -B128 -M32
ovlp_daligner_option = -h60 -e.96 -l500 -s1000

pa_DBsplit_option = -x500 -s400
ovlp_DBsplit_option = -x500 -s400

\# error correction consensus options
falcon_sense_option = --output_multi --min_idt 0.70 --min_cov 4 --max_n_read 200 --n_core 24

\# overlap filtering options
overlap_filtering_setting = --max_diff 100 --max_cov 100 --min_cov 1 --bestn 10 --n_core 24

[job.defaults]
job_type = local
\#use_tmpdir = /scratch
pwatcher_type = blocking
job_type = string
submit = bash -C ${CMD} >| ${STDOUT_FILE} 2>| ${STDERR_FILE}
\#job_queue = bash -C ${CMD}
\# By dropping STD*_FILE, we see all output on the console.
\# That helps debugging in TravisCI/Bamboo.
\#submit = qsub -S /bin/bash -sync y -V  \
\#  -q ${JOB_QUEUE}     \
\#  -N ${JOB_NAME}        \
\#  -o "${JOB_STDOUT}" \
\#  -e "${JOB_STDERR}" \
\#  -pe smp ${NPROC}    \
\#  "${JOB_SCRIPT}"
\#JOB_QUEUE = bigmem

NPROC=4
\#njobs=1
[job.step.da]
[job.step.pda]
[job.step.la]
\#NPROC=1
[job.step.pla]
\#NPROC=1
[job.step.cns]
\#njobs=1
NPROC=24
[job.step.asm]
NPROC=32

I am copying the lines in the all.log file where the error first shows up:

2018-11-27 17:46:52,434 - pypeflow.simple_pwatcher_bridge:94 - ERROR - Task Node(0-rawreads/tan-runs/tan_005) failed with exit-code=1
2018-11-27 17:46:52,434 - pypeflow.simple_pwatcher_bridge:342 - DEBUG - recently_done: [(Node(0-rawreads/tan-runs/tan_005), False)]
2018-11-27 17:46:52,434 - pypeflow.simple_pwatcher_bridge:343 - DEBUG - Num done in this iteration: 1
2018-11-27 17:46:52,434 - pypeflow.simple_pwatcher_bridge:357 - ERROR - Some tasks are recently_done but not satisfied: set([Node(0-rawreads/tan-runs/tan_005)])
2018-11-27 17:46:52,434 - pypeflow.simple_pwatcher_bridge:358 - ERROR - ready: set([])
        submitted: set([Node(0-rawreads/tan-runs/tan_004), Node(0-rawreads/tan-runs/tan_003), Node(0-rawreads/tan-runs/tan_001), Node(0-rawreads/tan-runs/tan_000), Node(0-rawreads/tan-runs/tan_002)])
2018-11-27 17:46:52,434 - pwatcher.blocking:467 - DEBUG - delete(which='known', jobids=<5>)
2018-11-27 17:46:52,434 - pwatcher.blocking:431 - ERROR - Noop. We cannot kill blocked threads. Hopefully, everything will die on SIGTERM.
2018-11-27 17:46:52,434 - pypeflow.simple_pwatcher_bridge:189 - DEBUG - In notifyTerminate(), result of delete:None
2018-11-27 17:48:20,053 - pwatcher.blocking:235 - DEBUG - rc: 1
2018-11-27 17:48:20,053 - pwatcher.blocking:87 - DEBUG - Thread notify_exited(P750f3fae5c2a17->1).
2018-11-27 17:48:26,188 - pwatcher.blocking:235 - DEBUG - rc: 1
2018-11-27 17:48:26,188 - pwatcher.blocking:87 - DEBUG - Thread notify_exited(Pf1b3782f8a486b->1).
2018-11-27 17:49:09,055 - pwatcher.blocking:235 - DEBUG - rc: 1
2018-11-27 17:49:09,055 - pwatcher.blocking:87 - DEBUG - Thread notify_exited(P43e27df6c80071->1).
2018-11-27 17:49:17,746 - pwatcher.blocking:235 - DEBUG - rc: 1
2018-11-27 17:49:17,746 - pwatcher.blocking:87 - DEBUG - Thread notify_exited(P451b5aaeac19e6->1).
2018-11-27 17:49:22,504 - pwatcher.blocking:235 - DEBUG - rc: 1
2018-11-27 17:49:22,504 - pwatcher.blocking:87 - DEBUG - Thread notify_exited(Pfd3977c98f2cb8->1).

Since the first error message mentions 0-rawreads/tan-runs/tan_005, I have examined the stderr file in this directory and I am copying lines from this file below where the exception occurs:

datander -v -e0.7 -s1000 -P. raw_reads.21 raw_reads.22
+ datander -v -e0.7 -s1000 -P. raw_reads.21 raw_reads.22
LAcheck raw_reads TAN.raw_reads.21.las TAN.raw_reads.22.las
+ LAcheck raw_reads TAN.raw_reads.21.las TAN.raw_reads.22.las
[WARNING]Call 'bash -vex run_datander.sh' returned 256.
Traceback (most recent call last):
  File "/rufous/anag/.conda-envs/denovo_asm/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/rufous/anag/.conda-envs/denovo_asm/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/rufous/anag/.conda-envs/denovo_asm/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 1367, in <module>
    main()
  File "/rufous/anag/.conda-envs/denovo_asm/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 1363, in main
    args.func(args)
  File "/rufous/anag/.conda-envs/denovo_asm/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 933, in cmd_tan_apply
    tan_apply(args.db_fn, args.script_fn, args.job_done_fn)
  File "/rufous/anag/.conda-envs/denovo_asm/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", line 328, in tan_apply
    io.syscall('bash -vex {}'.format(os.path.basename(script_fn)))
  File "/rufous/anag/.conda-envs/denovo_asm/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
    raise Exception(msg)
Exception: Call 'bash -vex run_datander.sh' returned 256.
WARNING:root:Call '/bin/bash user_script.sh' returned 256.
WARNING:root:CD: 'uow-00' -> '/rufous/anag/pacbio/FALCON-examples/run/eric_data/0-rawreads/tan-runs/tan_005'
Traceback (most recent call last):
  File "/rufous/anag/.conda-envs/denovo_asm/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/rufous/anag/.conda-envs/denovo_asm/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/rufous/anag/.conda-envs/denovo_asm/lib/python2.7/site-packages/falcon_kit/mains/generic_run_units_of_work.py", line 116, in <module>
    main()
  File "/rufous/anag/.conda-envs/denovo_asm/lib/python2.7/site-packages/falcon_kit/mains/generic_run_units_of_work.py", line 112, in main
    run(**vars(args))
  File "/rufous/anag/.conda-envs/denovo_asm/lib/python2.7/site-packages/falcon_kit/mains/generic_run_units_of_work.py", line 65, in run
    pypeflow.do_task.run_bash(script, inputs, outputs, params)
  File "/rufous/anag/.conda-envs/denovo_asm/lib/python2.7/site-packages/pypeflow/do_task.py", line 187, in run_bash
    util.system(cmd)
  File "/rufous/anag/.conda-envs/denovo_asm/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
    raise Exception(msg)
Exception: Call '/bin/bash user_script.sh' returned 256.
2018-11-27 10:46:49,538 - root - WARNING - Call '/bin/bash user_script.sh' returned 256.
2018-11-27 10:46:49,538 - root - WARNING - CD: '/rufous/anag/pacbio/FALCON-examples/run/eric_data/0-rawreads/tan-runs/tan_005' -> '/rufous/anag/pacbio/FALCON-examples/run/eric_data/0-rawreads/tan-runs/tan_005'
2018-11-27 10:46:49,538 - root - WARNING - CD: '/rufous/anag/pacbio/FALCON-examples/run/eric_data/0-rawreads/tan-runs/tan_005' -> '/rufous/anag/pacbio/FALCON-examples/run/eric_data/0-rawreads/tan-runs/tan_005'
2018-11-27 10:46:49,539 - root - CRITICAL - Error in /rufous/anag/.conda-envs/denovo_asm/lib/python2.7/site-packages/pypeflow/do_task.py with args="{'json_fn': '/rufous/anag/pacbio/FALCON-examples/run/eric_data/0-rawreads/tan-runs/tan_005/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
Traceback (most recent call last):
  File "/rufous/anag/.conda-envs/denovo_asm/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/rufous/anag/.conda-envs/denovo_asm/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/rufous/anag/.conda-envs/denovo_asm/lib/python2.7/site-packages/pypeflow/do_task.py", line 267, in <module>
    main()
  File "/rufous/anag/.conda-envs/denovo_asm/lib/python2.7/site-packages/pypeflow/do_task.py", line 259, in main
    run(**vars(parsed_args))
  File "/rufous/anag/.conda-envs/denovo_asm/lib/python2.7/site-packages/pypeflow/do_task.py", line 253, in run
    run_cfg_in_tmpdir(cfg, tmpdir, '.')
  File "/rufous/anag/.conda-envs/denovo_asm/lib/python2.7/site-packages/pypeflow/do_task.py", line 228, in run_cfg_in_tmpdir
    run_bash(bash_template, myinputs, myoutputs, parameters)
  File "/rufous/anag/.conda-envs/denovo_asm/lib/python2.7/site-packages/pypeflow/do_task.py", line 187, in run_bash
    util.system(cmd)
  File "/rufous/anag/.conda-envs/denovo_asm/lib/python2.7/site-packages/pypeflow/io.py", line 29, in syscall
    raise Exception(msg)
Exception: Call '/bin/bash user_script.sh' returned 256.
+++ pwd
++ echo 'FAILURE. Running top in /rufous/anag/pacbio/FALCON-examples/run/eric_data/0-rawreads/tan-runs/tan_005 (If you see -terminal database is inaccessible- you are using the python bin-wrapper, so you will not get diagnostic info. No big deal. This process is crashing anyway.)'
++ rm -f top.txt
++ which python
++ which top
++ env -u LD_LIBRARY_PATH top -b -n 1
++ env -u LD_LIBRARY_PATH top -b -n 1
++ pstree -apl

real    1m50.341s
user    6m8.745s
sys     0m32.202s
+ finish
+ echo 'finish code: 1'

Can you kindly pointing out what I am doing incorrectly and how to overcome this error ? Your help is much appreciated. Thanks in advance.

amnag commented 5 years ago

Hi @pb-cdunn, I would really appreciate your help with this issue.

pb-cdunn commented 5 years ago

You were only looking at the top-level message. Errors are reported per-task in the task-run directories, e.g. 0-rawreads/tan-runs/tan_003.