green-genome
commented
6 years ago After posting the following Q, I learned that there is no "reads.fa" in any of 3-unzip/reads/*F directories; each of these directories include only one file named "ref.fa" (which I assume is an assembled config). All raw reads somehow made their way into the "unassigned" folder and not mapped to any of the assembled reads. On the other hand, inspection of read_to_contig_map (under get_read_ctg_map) suggests that mapping procedure went well:
$ head read_to_contig_map
000153545 000425518 m160820_062313_42137_c101060742550000001823251101061765_s1_p0/148377/5437_44266 000011F
000277273 000707616 m160911_192112_42137_c101102542550000001823258304261737_s1_p0/30510/801_45196 000014F
000098706 000290649 m160621_204529_42137_c100996862550000001823225908031697_s1_p0/99150/0_17041 000178F
000044304 000138141 m160614_024833_00124_c100994142550000001823228808031641_s1_p0/30168/0_31605 000029F
000264014 000679467 m160910_180420_42137_c101102542550000001823258304261733_s1_p0/72669/0_22013 000198F
000066160 000198705 m160617_202057_00124_c100994362550000001823228808031665_s1_p0/109277/0_42103 000080F
000077259 000231621 m160618_085907_00124_c100994362550000001823228808031667_s1_p0/11623/0_34171 000062F
000322623 000804202 m160820_000351_42137_c101060742550000001823251101061764_s1_p0/82133/0_27225 000062F
000315975 000788042 m160910_180420_42137_c101102542550000001823258304261733_s1_p0/122789/24697_63328 000007F
000119358 000346545 m160715_213729_00124_c101009842550000001823230710211660_s1_p0/161146/0_58239 000054FAny thought on what may have caused the problem? (perhaps seq name having "/" character?)
It seems we are having the same problem. I was also able to run FALCON successfully but stumbled upon errors during FALCON unzip (e.g. CRITICAL - Error in /array1/ekim1/fc_env_180618/lib/python2.7/site-packages/pypeflow/do_task.py with args="{'json_fn': '/array1/ekim1/falcon/mantamonad/3-unzip/0-phasing/000255F/task.json',\n 'timeout': 30,\n 'tmpdir': None}" Here is portion of one of the stderr files:
2018-06-26 13:28:50,375 - root - DEBUG - Running "/array1/ekim1/fc_env_180618/lib/python2.7/site-packages/pypeflow/do_task.py /array1/ekim1/falcon/mantamonad/3-unzip/0-phasing/000255F/task.json"
2018-06-26 13:28:50,378 - root - DEBUG - Checking existence of '/array1/ekim1/falcon/mantamonad/3-unzip/0-phasing/000255F/task.json' with timeout=30
2018-06-26 13:28:50,378 - root - DEBUG - Loading JSON from '/array1/ekim1/falcon/mantamonad/3-unzip/0-phasing/000255F/task.json'
2018-06-26 13:28:50,379 - root - DEBUG - {u'bash_template_fn': u'template.sh',
u'inputs': {u'read_fasta': u'/array1/ekim1/falcon/mantamonad/3-unzip/reads/000255F/reads.fa',
u'ref_fasta': u'/array1/ekim1/falcon/mantamonad/3-unzip/reads/000255F/ref.fa'},
u'outputs': {u'rid_to_phase_out': u'rid_to_phase'},
u'parameters': {u'ctg_id': u'000255F'}}
2018-06-26 13:28:50,379 - root - WARNING - CD: '/array1/ekim1/falcon/mantamonad/3-unzip/0-phasing/000255F' <- '/array1/ekim1/falcon/mantamonad/3-unzip/0-phasing/000255F'
2018-06-26 13:28:50,379 - root - DEBUG - Checking existence of u'/array1/ekim1/falcon/mantamonad/3-unzip/reads/000255F/reads.fa' with timeout=30
2018-06-26 13:29:20,410 - root - WARNING - CD: '/array1/ekim1/falcon/mantamonad/3-unzip/0-phasing/000255F' -> '/array1/ekim1/falcon/mantamonad/3-unzip/0-phasing/000255F'
2018-06-26 13:29:20,411 - root - CRITICAL - Error in /array1/ekim1/fc_env_180618/lib/python2.7/site-packages/pypeflow/do_task.py with args="{'json_fn': '/array1/ekim1/falcon/mantamonad/3-unzip/0-phasing/000255F/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/array1/ekim1/fc_env_180618/lib/python2.7/site-packages/pypeflow/do_task.py", line 250, in <module>
main()
File "/array1/ekim1/fc_env_180618/lib/python2.7/site-packages/pypeflow/do_task.py", line 242, in main
run(**vars(parsed_args))
File "/array1/ekim1/fc_env_180618/lib/python2.7/site-packages/pypeflow/do_task.py", line 236, in run
run_cfg_in_tmpdir(cfg, tmpdir)
File "/array1/ekim1/fc_env_180618/lib/python2.7/site-packages/pypeflow/do_task.py", line 182, in run_cfg_in_tmpdir
wait_for(fn)
File "/array1/ekim1/fc_env_180618/lib/python2.7/site-packages/pypeflow/do_task.py", line 73, in wait_for
raise Exception('Timed out waiting for {!r}'.format(fn))
Exception: Timed out waiting for u'/array1/ekim1/falcon/mantamonad/3-unzip/reads/000255F/reads.fa'
real 0m30.148s
user 0m0.099s
sys 0m0.022s
+ finish
+ echo 'finish code: 1'FYI, here is the content of cfg file:
[General]
job_type = PBS
pwatcher_type = blocking
job_type = string
job_queue = bash -C ${CMD} > ${STDOUT_FILE} 2> ${STDERR_FILE}
#job_type = local
#job_queue = default
[Unzip]
input_fofn= input.fofn
input_bam_fofn= input_bams.fofn
smrt_bin= /array1/ekim1/apps/bin
sge_phasing = -l select=1:ncpus=12:mem=128gb
sge_quiver = -l select=1:ncpus=12:mem=128gb
sge_track_reads = -l select=1:ncpus=12:mem=128gb
sge_blasr_aln = -l select=1:ncpus=12:mem=128gb
sge_hasm = -l select=1:ncpus=12:mem=128gb
unzip_blasr_concurrent_jobs = 14
unzip_phasing_concurrent_jobs = 14
quiver_concurrent_jobs = 14Here is sh file:
#!/bin/bash
#PBS -V
#PBS -q batch
#PBS -S /bin/bash
#PBS -N falcon_unzip1_mantamonad
#PBS -l select=1:ncpus=14
#PBS -l walltime=80:00:00
cd $PBS_O_WORKDIR
source /array1/ekim1/fc_env/bin/activate
#load dependencies
module load gcc/4.9.2
#run it!
#fc_run.py fc_run_fungal.cfg
fc_unzip.py fc_unzip.cfg
fc_quiver.py fc_unzip.cfgThanks in advance for your help with this.
pb-cdunn
ViriatoII
I also have the same exception: timed out waiting for hasm_done.
I have successfully completed FALCON and FALCON-unzip on ecoli test dataset from https://pb-falcon.readthedocs.io/en/latest/tutorial.html#tutorial
I've also completed FALCON assembly on this human dataset. I have the following error when running FALCON-unzip:
2018-06-23 01:47:11,513 - pypeflow.simple_pwatcher_bridge - ERROR - Task Node(4-quiver/track_reads) failed with exit-code=256 2018-06-23 01:47:11,513 - pypeflow.simple_pwatcher_bridge - DEBUG - recently_done: [(Node(4-quiver/track_reads), False)] 2018-06-23 01:47:11,514 - pypeflow.simple_pwatcher_bridge - DEBUG - Num done in this iteration: 1 2018-06-23 01:47:11,514 - pypeflow.simple_pwatcher_bridge - INFO - recently_satisfied: set([]) 2018-06-23 01:47:11,514 - pypeflow.simple_pwatcher_bridge - INFO - Num satisfied in this iteration: 0 2018-06-23 01:47:11,514 - pypeflow.simple_pwatcher_bridge - INFO - Num still unsatisfied: 4 2018-06-23 01:47:11,514 - pypeflow.simple_pwatcher_bridge - ERROR - Some tasks are recently_done but not satisfied: set([Node(4-quiver/track_reads)]) 2018-06-23 01:47:11,515 - pypeflow.simple_pwatcher_bridge - ERROR - ready: set([]) submitted: set([]) 2018-06-23 01:47:11,515 - pwatcher.fs_based - DEBUG - delete(which='known', jobids=<0>) 2018-06-23 01:47:11,515 - pwatcher.fs_based - DEBUG - Deleting jobs for jobids from known ([]) 2018-06-23 01:47:11,515 - pwatcher.fs_based - INFO - !scancel -n P5a9de1ed13eb9d 2018-06-23 01:47:11,535 - pwatcher.fs_based - DEBUG - Cannot remove heartbeat: OSError(2, 'No such file or directory') 2018-06-23 01:47:11,536 - pypeflow.simple_pwatcher_bridge - DEBUG - In notifyTerminate(), result of delete:Nonemy config file
and stderr of job P5a9de1ed13eb9d
I installed falcon-unzip from https://pb-falcon.readthedocs.io/en/latest/quick_start.html#quick-start. I believe the version of the tarball is this:
UNZIP_TARBALL=falcon-2018.03.12-04.00-py2.7-ucs4.tar.gzAlso, from what I can tell, hasm_done does not exist in 3-unzip/1-hasm/.
Thanks in advance for your help.