fc_quiver.py does not get past quiver_scatter

[kevlim83]

Hi,

I am polishing my assembly after unzip using fc_quiver.py (latest version). However, it seem to stop after quiver_scatter step and I have no clue how to debug or proceed. The last few lines in my sge log file is:

[INFO]sleep 6.2s
[INFO]sleep 6.3s
[INFO]sleep 6.4s
[INFO]sleep 6.5s
[INFO]sleep 6.6s
[INFO]sleep 6.7s
[INFO]recently_satisfied: set([Node(4-quiver/quiver_scatter)])
[INFO]Num satisfied in this iteration: 1
[INFO]Num still unsatisfied: 0

The last few lines in the all.log file is:

2017-01-20 12:56:09,937 - pypeflow.simple_pwatcher_bridge - DEBUG - recently_done: [(Node(4-quiver/quiver_scatter), True)]
2017-01-20 12:56:09,937 - pypeflow.simple_pwatcher_bridge - DEBUG - Num done in this iteration: 1
2017-01-20 12:56:09,937 - pypeflow.simple_pwatcher_bridge - INFO - recently_satisfied: set([Node(4-quiver/quiver_scatter)])
2017-01-20 12:56:09,937 - pypeflow.simple_pwatcher_bridge - INFO - Num satisfied in this iteration: 1
2017-01-20 12:56:09,937 - pypeflow.simple_pwatcher_bridge - INFO - Num still unsatisfied: 0

Why is it stalling though? Any ideas?

Thanks.

[pb-cdunn]

It's "stalling"? It says "Num still unsatisfied: 0` but does not exit? If you can't attach a debugger, just hit Ctrl-C and post the stack-trace. I am very curious where it is blocked. Can you post the whole log somewhere?

[Timothy-WANG]

Ｉ also come across a same problem with the above one. How to debug?

[Timothy-WANG]

The err log is as follow: 2017-02-20 05:33:43,526 - pypeflow.simple_pwatcher_bridge - DEBUG - Result of watcher.run()={'submitted': ['Pcb1c61f3096ee8']} 2017-02-20 05:33:43,526 - pypeflow.simple_pwatcher_bridge - DEBUG - N in queue: 1 (max_jobs=32) 2017-02-20 05:33:43,526 - pwatcher.fs_based - DEBUG - query(which='list', jobids=<1>) 2017-02-20 05:33:43,528 - pwatcher.fs_based - DEBUG - Status RUNNING for heartbeat:heartbeat-Pcb1c61f3096ee8 2017-02-20 05:33:43,528 - pypeflow.simple_pwatcher_bridge - INFO - sleep 0.1s 2017-02-20 05:33:43,628 - pypeflow.simple_pwatcher_bridge - DEBUG - N in queue: 1 (max_jobs=32) 2017-02-20 05:33:43,629 - pwatcher.fs_based - DEBUG - query(which='list', jobids=<1>) 2017-02-20 05:33:43,630 - pwatcher.fs_based - DEBUG - Status RUNNING for heartbeat:heartbeat-Pcb1c61f3096ee8 2017-02-20 05:33:43,630 - pypeflow.simple_pwatcher_bridge - INFO - sleep 0.2s 2017-02-20 05:33:43,831 - pypeflow.simple_pwatcher_bridge - DEBUG - N in queue: 1 (max_jobs=32) 2017-02-20 05:33:43,831 - pwatcher.fs_based - DEBUG - query(which='list', jobids=<1>) 2017-02-20 05:33:43,833 - pwatcher.fs_based - DEBUG - Status RUNNING for heartbeat:heartbeat-Pcb1c61f3096ee8 2017-02-20 05:33:43,833 - pypeflow.simple_pwatcher_bridge - INFO - sleep 0.3s 2017-02-20 05:33:44,149 - pypeflow.simple_pwatcher_bridge - DEBUG - N in queue: 1 (max_jobs=32) 2017-02-20 05:33:44,150 - pwatcher.fs_based - DEBUG - query(which='list', jobids=<1>) 2017-02-20 05:33:44,151 - pwatcher.fs_based - DEBUG - Status RUNNING for heartbeat:heartbeat-Pcb1c61f3096ee8 2017-02-20 05:33:44,151 - pypeflow.simple_pwatcher_bridge - INFO - sleep 0.4s 2017-02-20 05:33:44,552 - pypeflow.simple_pwatcher_bridge - DEBUG - N in queue: 1 (max_jobs=32) 2017-02-20 05:33:44,553 - pwatcher.fs_based - DEBUG - query(which='list', jobids=<1>) 2017-02-20 05:33:44,554 - pwatcher.fs_based - DEBUG - Status RUNNING for heartbeat:heartbeat-Pcb1c61f3096ee8 2017-02-20 05:33:44,554 - pypeflow.simple_pwatcher_bridge - INFO - sleep 0.5s 2017-02-20 05:33:45,055 - pypeflow.simple_pwatcher_bridge - DEBUG - N in queue: 1 (max_jobs=32) 2017-02-20 05:33:45,055 - pwatcher.fs_based - DEBUG - query(which='list', jobids=<1>) 2017-02-20 05:33:45,056 - pwatcher.fs_based - DEBUG - Status RUNNING for heartbeat:heartbeat-Pcb1c61f3096ee8 2017-02-20 05:33:45,056 - pypeflow.simple_pwatcher_bridge - INFO - sleep 0.6s 2017-02-20 05:33:45,657 - pypeflow.simple_pwatcher_bridge - DEBUG - N in queue: 1 (max_jobs=32) 2017-02-20 05:33:45,658 - pwatcher.fs_based - DEBUG - query(which='list', jobids=<1>) 2017-02-20 05:33:45,659 - pwatcher.fs_based - DEBUG - Status RUNNING for heartbeat:heartbeat-Pcb1c61f3096ee8 2017-02-20 05:33:45,659 - pypeflow.simple_pwatcher_bridge - INFO - sleep 0.7s 2017-02-20 05:33:46,360 - pypeflow.simple_pwatcher_bridge - DEBUG - N in queue: 1 (max_jobs=32) 2017-02-20 05:33:46,361 - pwatcher.fs_based - DEBUG - query(which='list', jobids=<1>) 2017-02-20 05:33:46,362 - pwatcher.fs_based - DEBUG - Status RUNNING for heartbeat:heartbeat-Pcb1c61f3096ee8 2017-02-20 05:33:46,362 - pypeflow.simple_pwatcher_bridge - INFO - sleep 0.8s 2017-02-20 05:33:47,163 - pypeflow.simple_pwatcher_bridge - DEBUG - N in queue: 1 (max_jobs=32) 2017-02-20 05:33:47,164 - pwatcher.fs_based - DEBUG - query(which='list', jobids=<1>) 2017-02-20 05:33:47,165 - pwatcher.fs_based - DEBUG - Status RUNNING for heartbeat:heartbeat-Pcb1c61f3096ee8 2017-02-20 05:33:47,165 - pypeflow.simple_pwatcher_bridge - INFO - sleep 0.9s 2017-02-20 05:33:48,068 - pypeflow.simple_pwatcher_bridge - DEBUG - N in queue: 1 (max_jobs=32) 2017-02-20 05:33:48,068 - pwatcher.fs_based - DEBUG - query(which='list', jobids=<1>) 2017-02-20 05:33:48,069 - pwatcher.fs_based - DEBUG - Status RUNNING for heartbeat:heartbeat-Pcb1c61f3096ee8 2017-02-20 05:33:48,070 - pypeflow.simple_pwatcher_bridge - INFO - sleep 1.0s 2017-02-20 05:33:49,071 - pypeflow.simple_pwatcher_bridge - DEBUG - N in queue: 1 (max_jobs=32) 2017-02-20 05:33:49,071 - pwatcher.fs_based - DEBUG - query(which='list', jobids=<1>) 2017-02-20 05:33:49,073 - pwatcher.fs_based - DEBUG - Status RUNNING for heartbeat:heartbeat-Pcb1c61f3096ee8 2017-02-20 05:33:49,073 - pypeflow.simple_pwatcher_bridge - INFO - sleep 1.1s 2017-02-20 05:33:50,195 - pypeflow.simple_pwatcher_bridge - DEBUG - N in queue: 1 (max_jobs=32) 2017-02-20 05:33:50,195 - pwatcher.fs_based - DEBUG - query(which='list', jobids=<1>) 2017-02-20 05:33:50,196 - pwatcher.fs_based - DEBUG - Status RUNNING for heartbeat:heartbeat-Pcb1c61f3096ee8 2017-02-20 05:33:50,197 - pypeflow.simple_pwatcher_bridge - INFO - sleep 1.2s 2017-02-20 05:33:51,398 - pypeflow.simple_pwatcher_bridge - DEBUG - N in queue: 1 (max_jobs=32) 2017-02-20 05:33:51,399 - pwatcher.fs_based - DEBUG - query(which='list', jobids=<1>) 2017-02-20 05:33:51,400 - pwatcher.fs_based - DEBUG - Status RUNNING for heartbeat:heartbeat-Pcb1c61f3096ee8 2017-02-20 05:33:51,400 - pypeflow.simple_pwatcher_bridge - INFO - sleep 1.3s 2017-02-20 05:33:52,702 - pypeflow.simple_pwatcher_bridge - DEBUG - N in queue: 1 (max_jobs=32) 2017-02-20 05:33:52,702 - pwatcher.fs_based - DEBUG - query(which='list', jobids=<1>) 2017-02-20 05:33:52,704 - pwatcher.fs_based - DEBUG - Unable to remove heartbeat '/exeh/exe5/mqwang/dust/04_reassemble/falcon/mypwatcher/heartbeats/heartbeat-Pcb1c61f3096ee8' when sentinal was found in exit-sentinels listdir. Traceback (most recent call last): File "/exeh/exe5/mqwang/software/falcon_1/FALCON-integrate/pypeFLOW/pwatcher/fs_based.py", line 563, in get_status os.remove(heartbeat_path) OSError: [Errno 2] No such file or directory: '/exeh/exe5/mqwang/dust/04_reassemble/falcon/mypwatcher/heartbeats/heartbeat-Pcb1c61f3096ee8'

2017-02-20 05:33:52,704 - pwatcher.fs_based - DEBUG - Status EXIT 0 for heartbeat:heartbeat-Pcb1c61f3096ee8 2017-02-20 05:33:52,704 - pypeflow.simple_pwatcher_bridge - DEBUG - recently_done: [(Node(2-asm-falcon), True)] 2017-02-20 05:33:52,704 - pypeflow.simple_pwatcher_bridge - DEBUG - Num done in this iteration: 1 2017-02-20 05:33:52,704 - pypeflow.simple_pwatcher_bridge - INFO - recently_satisfied: set([Node(2-asm-falcon)]) 2017-02-20 05:33:52,705 - pypeflow.simple_pwatcher_bridge - INFO - Num satisfied in this iteration: 1 2017-02-20 05:33:52,705 - pypeflow.simple_pwatcher_bridge - INFO - Num still unsatisfied: 0

[kevlim83]

@CHEKNP I wasnt able to get fc_quiver.py to work. However, I finished polishing my draft assembly by running the steps manually using the files from 03-unzip folder. First, was your data from the RSII or sequel? This is important because you might need to know how to convert your files to the correct format. i.e. h5 to bam format.

You will also need to go through and understand pacbio's internal workings yourself, I'll list a few points here which has led me to finish the polish step: 1\ pbalign can produce xml files, this is important if you want to reduce IO cost 2\ use dataset to partition your alignments if you have a large cluster for your quiver jobs 3\ finally quiver all the small partitions before merging the final consensus.

Also, it looks like @CHEKNP, you are stalling at a different step: 2-asm-falcon.

[Timothy-WANG]

@kevlim83 Thanks for your suggestions. I have not yet a deeper understand of the process. My data is from SEQUEL in bam format. Does this make a difference ? And how to polish the draft by steps, can you inform this in detail?

[kevlim83]

@CHEKNP I think I have detailed the steps in my previous comment. Since you already have the data in bam format, it is quite straight forward. There are 3 steps, where you have to use pbalign to align your bam files to your draft assembly, then use dataset to partition your alignments into smaller parts and quiver each of the parts directly, lastly you can just concatenate the final results using cat. The fine grain details (e.g. how much to split data for quiver) depend on your computing environment and is hard for me to tell you precisely what to do. Let me know if you have specific questions.

[pb-cdunn]

Use FALCON-integrate to install Falcon and examples. (Falcon-unzip and smrttools can be installed separately.) Then, try the example in FALCON-examples/run/greg200k-sv2. I use that example regularly, and the config there should avoid your problems with "heartbeats" since it will run locally.

[conte1]

@kevlim83 and @CHEKNP I just wanted to chime in as I was running into this issue as well. I think one of the problems for me was that I was using a previous version of FALCON-integrate to generate assemblies and a newer version of FALCON_unzip for the unzip and quiver steps. Once I re-ran the assembly from scratch using the same, new version of FALCON-integrate things mostly worked.

I did run into an issue of having some P5-C3 data which was causing a hidden error right after the quiver_scatter step since the following line is hard coded in _falcon_unzip/runquiver.py: ({variantCaller} --algorithm=arrow -x 5 -X 120 -q 20 -j 24 -r {ref_fasta} aln-{ctg_id}.bam\ Once I changed that line to: ({variantCaller} --algorithm=quiver -x 5 -X 120 -q 20 -j 24 -r {ref_fasta} aln-{ctg_id}.bam\ My error went away and the entire workflow went to completion. You mentioned you're working with SEQUEL data so this last part probably isn't an issue for you though.

[pb-cdunn]

@conte1, That's interesting. How did you build variantCaller? Many folks have trouble getting the latest stuff. I don't think the latest smrttools tarball is publicly available for download, but you can build everything using PacificBiosciences/pitchfork.

[conte1]

@pb-cdunn, Yeah, I was able to get variantCaller built using pitchfork.