Exception: Timed out waiting for hasm_done

bgbrink commented 6 years ago

I succesfully completed a FALCON assembly and now I tried running FALCON-unzip, where I get the following error message:

ERROR]Task Node(4-quiver/track_reads) failed with exit-code=256
[INFO]recently_satisfied: set([])
[INFO]Num satisfied in this iteration: 0
[INFO]Num still unsatisfied: 4
[ERROR]Some tasks are recently_done but not satisfied: set([Node(4-quiver/track_reads)])
[ERROR]ready: set([])
        submitted: set([])
[INFO]!scancel -n Pf1c38725d9ec59
Traceback (most recent call last):
  File "/work/project/ladsie_002/FALCON-Unzip/fc_env_180605/bin/fc_quiver.py", line 11, in <module>
    load_entry_point('falcon-unzip==1.0.0', 'console_scripts', 'fc_quiver.py')()
  File "/scratch/cdunn/repo/fork/.git/LOCAL4/lib/python2.7/site-packages/falcon_unzip/mains/start_quiver.py", line 29, in main
  File "/scratch/cdunn/repo/fork/.git/LOCAL4/lib/python2.7/site-packages/falcon_unzip/quiver.py", line 92, in run
  File "/scratch/cdunn/repo/fork/.git/LOCAL4/lib/python2.7/site-packages/falcon_unzip/tasks/quiver.py", line 201, in run_workflow
  File "/scratch/cdunn/repo/fork/.git/LOCAL4/lib/python2.7/site-packages/falcon_unzip/tasks/pype.py", line 47, in gen_parallel_tasks
  File "/work/project/ladsie_002/FALCON-Unzip/fc_env_180605/lib/python2.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 284, in refreshTargets
    self._refreshTargets(updateFreq, exitOnFailure)
  File "/work/project/ladsie_002/FALCON-Unzip/fc_env_180605/lib/python2.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 350, in _refreshTargets
    raise Exception(msg)
Exception: Some tasks are recently_done but not satisfied: set([Node(4-quiver/track_reads)])

Looking into the stderr file of the failed job, the reason seems to be a hardcoded timeout:

2018-06-19 23:37:58,846 - root - CRITICAL - Error in /work/project/ladsie_002/FALCON-Unzip/fc_env_180605/lib/python2.7/site-packages/pypeflow/do_task.py with args="{'json_fn': '/work/project/ladsie_002/analyses/2018-04-18_PacBio_remap/falcon-assembly-funghi/4-quiver/track_reads/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/work/project/ladsie_002/FALCON-Unzip/fc_env_180605/lib/python2.7/site-packages/pypeflow/do_task.py", line 250, in <module>
    main()
  File "/work/project/ladsie_002/FALCON-Unzip/fc_env_180605/lib/python2.7/site-packages/pypeflow/do_task.py", line 242, in main
    run(**vars(parsed_args))
  File "/work/project/ladsie_002/FALCON-Unzip/fc_env_180605/lib/python2.7/site-packages/pypeflow/do_task.py", line 236, in run
    run_cfg_in_tmpdir(cfg, tmpdir)
  File "/work/project/ladsie_002/FALCON-Unzip/fc_env_180605/lib/python2.7/site-packages/pypeflow/do_task.py", line 182, in run_cfg_in_tmpdir
    wait_for(fn)
  File "/work/project/ladsie_002/FALCON-Unzip/fc_env_180605/lib/python2.7/site-packages/pypeflow/do_task.py", line 73, in wait_for
    raise Exception('Timed out waiting for {!r}'.format(fn))
Exception: Timed out waiting for u'/work/project/ladsie_002/analyses/2018-04-18_PacBio_remap/falcon-assembly-funghi/3-unzip/1-hasm/hasm_done'

real    0m30.311s
user    0m0.074s
sys 0m0.067s
 returned: 256

Do you have any idea how I could fix this? Any help is much appreciated.

pb-cdunn commented 6 years ago

How did you install Unzip? (The Github version is too old.)

To debug this, look in 3-unzip/1-hasm and 4-quiver/track_reads. Look for a file called something like *.stderr.

Also, you need to post your .cfg so I can see your pypeflow settings. (If you use the fs_based pwatcher, then stderr will be in the pwatcher.dir. With the blocking pwatcher, it's in the run-dir.)

Also, you should always always always run on a small test-case first. We have a pretty quick one called greg200k-sv2 available via FALCON-examples: https://github.com/pb-cdunn/FALCON-examples

cd FALCON-examples/run/greg200k-sv2
git-sym .
# edit *.cfg
make falcon
make unzip

But you need the Unzip tarball to be at all up-to-date: https://github.com/PacificBiosciences/FALCON_unzip/wiki/Binaries

And in general, you need to provide a lot more information. I am not allowed much time to help via GitHub.

xiaoaozqd commented 6 years ago

@pb-cdunn I have the same problem！ I installed the FALCON_unzip according to https://pb-falcon.readthedocs.io/en/latest/quick_start.html with the tar ball 3/12/2018. The tree of the output dir as

|-- 0-rawreads
|   |-- build_rdb.sh
|   |-- cns-chunks
|   |-- cns-gather
|   |-- cns-runs
|   |-- cns-split
|   |-- daligner-chunks
|   |-- daligner-gathered
|   |-- daligner-intermediate-gathered-las
|   |-- daligner-runs
|   |-- daligner-split
|   |-- las-gather
|   |-- las-merge-chunks
|   |-- las-merge-gathered
|   |-- las-merge-runs
|   |-- las-merge-split
|   |-- length_cutoff
|   |-- my.input.fofn
|   |-- preads
|   |-- pwatcher.dir
|   |-- raw_reads.db
|   |-- rdb_build_done
|   |-- report
|   |-- run_jobs.sh
|   |-- run.sh
|   |-- run.sh.done
|   |-- task.json
|   |-- task.sh
|   |-- template.sh
|   `-- user_script.sh
|-- 1-preads_ovl
|   |-- build_pdb.sh
|   |-- daligner-chunks
|   |-- daligner-gathered
|   |-- daligner-intermediate-gathered-las
|   |-- daligner-runs
|   |-- daligner-split
|   |-- db2falcon
|   |-- las-gather
|   |-- las-merge-chunks
|   |-- las-merge-gathered
|   |-- las-merge-runs
|   |-- las-merge-split
|   |-- pdb_build_done
|   |-- preads.db
|   |-- preads.fofn
|   |-- pwatcher.dir
|   |-- run_jobs.sh
|   |-- run.sh
|   |-- run.sh.done
|   |-- task.json
|   |-- task.sh
|   |-- template.sh
|   `-- user_script.sh
|-- 2-asm-falcon
|   |-- a_ctg_all.fa
|   |-- a_ctg_base.fa
|   |-- a_ctg_base_tiling_path
|   |-- a_ctg.fa
|   |-- a_ctg_tiling_path
|   |-- asm.gfa
|   |-- chimers_nodes
|   |-- c_path
|   |-- ctg_paths
|   |-- falcon_asm_done
|   |-- fc_ovlp_to_graph.log
|   |-- p_ctg.fa
|   |-- p_ctg_tiling_path
|   |-- preads4falcon.fasta
|   |-- preads.ovl
|   |-- pwatcher.dir
|   |-- run.sh
|   |-- run.sh.done
|   |-- sg_edges_list
|   |-- sg.gfa
|   |-- task.json
|   |-- task.sh
|   |-- template.sh
|   |-- user_script.sh
|   `-- utg_data
|-- 3-unzip
|   `-- reads
|-- 4-quiver
|   `-- track_reads
|-- all.log
|-- assembly.sh
|-- assembly.sh.e1059657
|-- assembly.sh.e1059658
|-- assembly.sh.o1059657
|-- assembly.sh.o1059658
|-- bam
|   |-- m54168_180413_073102.subreads.bam
|   `-- m54168_180413_073102.subreadset.xml
|-- config.json
|-- falcon_unzip.e
|-- falcon_unzip.o
|-- fc_run_fungal.cfg
|-- fc_run.log
|-- fc_unzip.cfg
|-- fc_unzip.log
|-- fc_unzip_quiver.log
|-- foo.snake
|-- input_bam.fofn
|-- input.fofn
|-- mypwatcher
|   |-- exits
|   |-- heartbeats
|   |-- jobs
|   `-- wrappers
`-- rhc_falcon_unzip.sh

the fc_unzip.cfg

[General]
#job_type = SGE
job_type = local
#job_queue = default

[Unzip]
input_fofn= input.fofn
input_bam_fofn= input_bams.fofn

#path to bin directory containing samtools, blasr, and various GenomicConsensus utilities
smrt_bin=/home/zengqd/software/geno_ass/FALCON/fc_env/bin /home/zengqd/software/geno_ass/FALCON/GenomicConsensus/bin
#smrt_bin=/share/nas3/zqd/wheat-trans/soft/samtools-1.9/samtools /share/nas3/zqd/wheat-trans/soft/gen_ass/FALCON/fc_env/bin/blasr /share/nas3/zqd/wheat-trans/soft/gen_ass/FALCON/fc_env/bin/ /share/nas3/zqd/wheat-trans/soft/gen_ass/GenomicConsensus-2.3.2/bin
#smrt_bin=/path/to/smrtcmds/bin/

#sge_phasing= -pe smp 12
#sge_quiver= -pe smp 24
#sge_track_reads= -pe smp 12
#sge_blasr_aln=  -pe smp 24
#sge_hasm=  -pe smp 48
#unzip_blasr_concurrent_jobs = 80
unzip_phasing_concurrent_jobs =24
quiver_concurrent_jobs = 24

The stderr and stdout as attachment

pb-cdunn commented 5 years ago

You would need to check stderr for the failing task: ERROR - Task Node(4-quiver/track_reads) failed with exit-code=256

PacificBiosciences / FALCON_unzip

Exception: Timed out waiting for hasm_done #125