search

PacificBiosciences / FALCON_unzip

Making diploid assembly becomes common practice for genomic study
BSD 3-Clause Clear License
30 stars 18 forks source link

error at 4-polish step with "failed multiprocessing" #159

Closed y842739756 closed 4 years ago

y842739756 commented 4 years ago

I tried to run the example data set "greg200k-sv2". and the pipeline exits with the info

[ERROR]Task Node(4-polish/quiver-run/000000F) failed with exit-code=1

[ERROR]Task Node(4-polish/quiver-run/000000F_001) failed with exit-code=1 [ERROR]Some tasks are recently_done but not satisfied: {Node(4-polish/quiver-run/000000F), Node(4-polish/quiver-run/000000F_001)} [ERROR]ready: set() submitted: set() [ERROR]Noop. We cannot kill blocked threads. Hopefully, everything will die on SIGTERM. [ERROR]Error for run(target=clr, config_fn=fc_unzip.cfg, logging_config_fn=None) Traceback (most recent call last): File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_unzip/unzip.py", line 195, in run run_logged(target, config_fn) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_unzip/unzip.py", line 184, in run_logged unzip_all(config, unzip_config_fn) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_unzip/unzip.py", line 27, in unzip_all tasks_unzip.run_workflow(wf, config, unzip_config_fn) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_unzip/tasks/unzip.py", line 739, in run_workflow job_dict=config['job.step.unzip.quiver'], File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_kit/pype.py", line 195, in gen_parallel_tasks wf.refreshTargets() File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 278, in refreshTargets self._refreshTargets(updateFreq, exitOnFailure) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 362, in _refreshTargets raise Exception(msg) Exception: Some tasks are recently_done but not satisfied: {Node(4-polish/quiver-run/000000F), Node(4-polish/quiver-run/000000F_001)} Traceback (most recent call last): File "/home/yanxu2016/anaconda2/envs/pb-assembly/bin/fc_unzip.py", line 11, in load_entry_point('falcon-unzip==1.3.7', 'console_scripts', 'fc_unzip.py')() File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_unzip/mains/start_unzip.py", line 33, in main unzip.run(**vars(args)) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_unzip/unzip.py", line 195, in run run_logged(target, config_fn) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_unzip/unzip.py", line 184, in run_logged unzip_all(config, unzip_config_fn) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_unzip/unzip.py", line 27, in unzip_all tasks_unzip.run_workflow(wf, config, unzip_config_fn) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_unzip/tasks/unzip.py", line 739, in run_workflow job_dict=config['job.step.unzip.quiver'], File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_kit/pype.py", line 195, in gen_parallel_tasks wf.refreshTargets() File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 278, in refreshTargets self._refreshTargets(updateFreq, exitOnFailure) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 362, in _refreshTargets raise Exception(msg) Exception: Some tasks are recently_done but not satisfied: {Node(4-polish/quiver-run/000000F), Node(4-polish/quiver-run/000000F_001)}

` And The stderr show that :

`set +e

  • set +e gcpp --algorithm=arrow -x 5 -X 120 -q 0 -j $nproc -r ../../../quiver-split/refs/000000F_001/ref.fasta aln-000000F_001.bam -o ${fasta_fn},${fastq_fn},cns.vcf
  • gcpp --algorithm=arrow -x 5 -X 120 -q 0 -j 1 -r ../../../quiver-split/refs/000000F_001/ref.fasta aln-000000F_001.bam -o cns.fasta,cns.fastq,cns.vcf rc=$?
  • rc=1 if [[ $rc != 0 ]]; then if [[ $VC_IGNORE_ERROR != 1 ]]; then echo ERROR gcpp failed. Maybe no reads for this block? exit 1 else echo WARNING gcpp failed. Maybe no reads for this block.

    We expect gcpp to write files even on error, so we do not need to "touch" them.

    fi fi

  • [[ 1 != 0 ]]
  • [[ 0 != 1 ]]
  • echo ERROR gcpp failed. Maybe no reads for this 'block?'
  • exit 1 touch quiver_done.exit
  • touch quiver_done.exit WARNING:root:Call '/bin/bash user_script.sh' returned 256. INFO:root:CD: 'uow-00' -> '/home/yanxu2016/Documents/workfile/assembly_test/FALCON-examples/run/greg200k-sv2/4-polish/quiver-run/000000F_001' ERROR:root:failed multiprocessing multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_kit/util/io.py", line 68, in run_func ret = func(*args) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_kit/mains/generic_run_units_of_work.py", line 45, in run_uow pypeflow.do_task.run_bash(script, inputs, outputs, params) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/pypeflow/do_task.py", line 200, in run_bash util.system(cmd) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/pypeflow/io.py", line 27, in syscall raise Exception(msg) Exception: Call '/bin/bash user_script.sh' returned 256.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/multiprocessing/pool.py", line 121, in worker result = (True, func(*args, *kwds)) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_kit/util/io.py", line 73, in run_func raise Exception(traceback.format_exc()) Exception: Traceback (most recent call last): File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_kit/util/io.py", line 68, in run_func ret = func(args) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_kit/mains/generic_run_units_of_work.py", line 45, in run_uow pypeflow.do_task.run_bash(script, inputs, outputs, params) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/pypeflow/do_task.py", line 200, in run_bash util.system(cmd) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/pypeflow/io.py", line 27, in syscall raise Exception(msg) Exception: Call '/bin/bash user_script.sh' returned 256.

"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_kit/mains/generic_run_units_of_work.py", line 103, in run for res in exe_pool.imap(run_func, inputs): File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/multiprocessing/pool.py", line 748, in next raise value Exception: Traceback (most recent call last): File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_kit/util/io.py", line 68, in run_func ret = func(*args) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_kit/mains/generic_run_units_of_work.py", line 45, in run_uow pypeflow.do_task.run_bash(script, inputs, outputs, params) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/pypeflow/do_task.py", line 200, in run_bash util.system(cmd) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/pypeflow/io.py", line 27, in syscall raise Exception(msg) Exception: Call '/bin/bash user_script.sh' returned 256.

multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_kit/util/io.py", line 68, in run_func ret = func(*args) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_kit/mains/generic_run_units_of_work.py", line 45, in run_uow pypeflow.do_task.run_bash(script, inputs, outputs, params) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/pypeflow/do_task.py", line 200, in run_bash util.system(cmd) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/pypeflow/io.py", line 27, in syscall raise Exception(msg) Exception: Call '/bin/bash user_script.sh' returned 256.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/multiprocessing/pool.py", line 121, in worker result = (True, func(*args, *kwds)) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_kit/util/io.py", line 73, in run_func raise Exception(traceback.format_exc()) Exception: Traceback (most recent call last): File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_kit/util/io.py", line 68, in run_func ret = func(args) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_kit/mains/generic_run_units_of_work.py", line 45, in run_uow pypeflow.do_task.run_bash(script, inputs, outputs, params) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/pypeflow/do_task.py", line 200, in run_bash util.system(cmd) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/pypeflow/io.py", line 27, in syscall raise Exception(msg) Exception: Call '/bin/bash user_script.sh' returned 256.

"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_kit/mains/generic_run_units_of_work.py", line 152, in main() File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_kit/mains/generic_run_units_of_work.py", line 148, in main run(*vars(args)) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_kit/mains/generic_run_units_of_work.py", line 103, in run for res in exe_pool.imap(run_func, inputs): File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/multiprocessing/pool.py", line 748, in next raise value Exception: Traceback (most recent call last): File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_kit/util/io.py", line 68, in run_func ret = func(args) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/falcon_kit/mains/generic_run_units_of_work.py", line 45, in run_uow pypeflow.do_task.run_bash(script, inputs, outputs, params) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/pypeflow/do_task.py", line 200, in run_bash util.system(cmd) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/pypeflow/io.py", line 27, in syscall raise Exception(msg) Exception: Call '/bin/bash user_script.sh' returned 256.

2020-02-18 22:37:48,531 - root - WARNING - Call '/bin/bash user_script.sh' returned 256. 2020-02-18 22:37:48,531 - root - INFO - CD: '/home/yanxu2016/Documents/workfile/assembly_test/FALCON-examples/run/greg200k-sv2/4-polish/quiver-run/000000F_001' -> '/home/yanxu2016/Documents/workfile/assembly_test/FALCON-examples/run/greg200k-sv2/4-polish/quiver-run/000000F_001' 2020-02-18 22:37:48,531 - root - INFO - CD: '/home/yanxu2016/Documents/workfile/assembly_test/FALCON-examples/run/greg200k-sv2/4-polish/quiver-run/000000F_001' -> '/home/yanxu2016/Documents/workfile/assembly_test/FALCON-examples/run/greg200k-sv2/4-polish/quiver-run/000000F_001' 2020-02-18 22:37:48,531 - root - CRITICAL - Error in /home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/pypeflow/do_task.py with args="{'json_fn': '/home/yanxu2016/Documents/workfile/assembly_test/FALCON-examples/run/greg200k-sv2/4-polish/quiver-run/000000F_001/task.json',\n 'timeout': 30,\n 'tmpdir': None}" Traceback (most recent call last): File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/pypeflow/do_task.py", line 280, in main() File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/pypeflow/do_task.py", line 272, in main run(**vars(parsed_args)) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/pypeflow/do_task.py", line 266, in run run_cfg_in_tmpdir(cfg, tmpdir, '.') File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/pypeflow/do_task.py", line 241, in run_cfg_in_tmpdir run_bash(bash_template, myinputs, myoutputs, parameters) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/pypeflow/do_task.py", line 200, in run_bash util.system(cmd) File "/home/yanxu2016/anaconda2/envs/pb-assembly/lib/python3.7/site-packages/pypeflow/io.py", line 27, in syscall raise Exception(msg) Exception: Call '/bin/bash user_script.sh' returned 256. +++ pwd ++ echo 'FAILURE. Running top in /home/yanxu2016/Documents/workfile/assembly_test/FALCON-examples/run/greg200k-sv2/4-polish/quiver-run/000000F_001 (If you see -terminal database is inaccessible- you are using the python bin-wrapper, so you will not get diagnostic info. No big deal. This process is crashing anyway.)' ++ rm -f top.txt ++ which python ++ which top ++ env -u LD_LIBRARY_PATH top -b -n 1 ++ env -u LD_LIBRARY_PATH top -b -n 1 ++ pstree -apl

real 0m2.089s user 0m3.262s sys 0m0.550s

  • finish
  • echo 'finish code: 1'

` I can not fix that problem ,help me please ~ Thanks in advance and best regards Xu

RAWWiberg commented 4 years ago

Hi, I'm having the same error. Have you figured out what is going wrong here?

EricTulowetzke commented 4 years ago

Hi, I'm having the same error as well. Have you figured out what is going wrong here?

mroach-awri commented 4 years ago

Not a fix but a workaround: If it's just the quiver stage you're having issues with you can map the reads to the unpolished assembly with blasr and manually run quiver or arrow.

villegar commented 4 years ago

Hi, after coming across this same error message I started to test different things. The one that worked for me was installing SMRTLink 7.0.1 (https://www.pacb.com/support/software-downloads/) as the error seemed to be related with some unsupported stuff on newer versions, I found the following message in my output unsupported sequencing chemistry combination. After installing SMRTLink, I executed the following command:

export PATH=/path/to/SMRTLink/7.0.1/smrtlink/current/bundles/smrttools/smrtcmds/bin:$PATH

Hope that helps if you come across a similar error message.

Reference: https://github.com/PacificBiosciences/pbbioconda/issues/211

pb-cdunn commented 4 years ago

We've updated the testcase in falcon-integrate to use a new "chemistry". It should be available in the pacbio cloud. I hope that fixes it.

AG-Run commented 3 years ago

I got the same error,

ERROR:root:failed multiprocessing multiprocessing.pool.RemoteTraceback:

In my case I used pb-assembly

Thanks

pb-cdunn commented 3 years ago

You've got to step into the sub-dir in which the error is happening. If you can't figure it out from stderr there, you can re-run the specific command and try to debug it.

IPA works great. The old pb-assembly stuff is hard to maintain because nearly all the questions submitted by users relate to installation and integration. It's just too time-consuming. If you find an actual bug, we might be able to update the software and release a new version with a fix.

It is very important to run the entire workflow on a tiny example first, locally rather than in a cluster. Then in a cluster. And finally in a cluster on your own data. The fact that people refuse to do that drives me absolutely bonkers.

The simple testdata is here:

If you have trouble with FALCON-integrate or git-sym, you can download directly by looking at:

I think greg200k-sv2.3will work with the latest chemistry. (I have to regenerate the test dataset whenever PB updates the chemistry, which takes a bit of effort.)