search

cschin / Peregrine

Peregrine: Fast Genome Assembler Using SHIMMER Index
Other
101 stars 9 forks source link

Exception: Some tasks are recently_done but not satisfied #33

Open chhylp123 opened 4 years ago

chhylp123 commented 4 years ago

Hello,

I am using Peregrine with the following commands:

docker run -i -t cschin/peregrine:0.1.6.1 asm /hlilab/chhy/Peregrine/chm13-seqdata.lst 32 32 32 32 32 32 32 32 32 --with-consensus --shimmer-r 3 --best_n_ovlp 8 --output /hlilab/chhy/Peregrine/HG002-34X-chr11-asm

But it crashed. Here is the log:

Peregrine Assembler (0+unknown) has been started with the following option:
 {'--aln_bw': '100',
 '--best_n_ovlp': '8',
 '--help': False,
 '--mc_lower': '2',
 '--mc_upper': '240',
 '--min_idt': '96',
 '--min_len': '4000',
 '--output': '/hlilab/chhy/Peregrine/HG002-34X-chr11-asm',
 '--ovlp_upper': '120',
 '--shimmer-k': '16',
 '--shimmer-l': '2',
 '--shimmer-r': '3',
 '--shimmer-w': '80',
 '--verison': False,
 '--with-L0-index': False,
 '--with-alt': False,
 '--with-consensus': True,
 '<cns_nchunk>': '32',
 '<cns_nproc>': '32',
 '<index_nchunk>': '32',
 '<index_nproc>': '32',
 '<mapping_nchunk>': '32',
 '<mapping_nproc>': '32',
 '<ovlp_nchunk>': '32',
 '<ovlp_nproc>': '32',
 '<reads.lst>': '/hlilab/chhy/Peregrine/chm13-seqdata.lst',
 '<sort_nproc>': '32',
 'asm': True}
INFO:pypeflow.simple_pwatcher_bridge:In simple_pwatcher_bridge, pwatcher_impl=<module 'pwatcher.blocking' from '/opt/conda/envs/peregrine/lib/python3.7/site-packages/pypeflow-2.1.1+git.d63b0e79f5a7b2d370b7de84a890f88271afa476-py3.7.egg/pwatcher/blocking.py'>
INFO:pypeflow.simple_pwatcher_bridge:job_type='local', (default)job_defaults={'njobs': 1, 'NPROC': 1, 'MB': 24000, 'submit': 'bash -C ${CMD} >| ${STDOUT_FILE} 2>| ${STDERR_FILE}', 'job_type': 'local', 'pwatcher_type': 'blocking'}, use_tmpdir=None, squash=False, job_name_style=0
INFO:pypeflow.simple_pwatcher_bridge:Setting max_jobs to 1; was None
INFO:pypeflow.simple_pwatcher_bridge:Num unsatisfied: 1, graph: 1
INFO:pypeflow.simple_pwatcher_bridge:About to submit: Node(../../hlilab/chhy/Peregrine/HG002-34X-chr11-asm/0-seqdb)
INFO:pwatcher.blocking:Popen: '/bin/bash -C /opt/conda/envs/peregrine/lib/python3.7/site-packages/pypeflow-2.1.1+git.d63b0e79f5a7b2d370b7de84a890f88271afa476-py3.7.egg/pwatcher/mains/job_start.sh >| /hlilab/chhy/Peregrine/HG002-34X-chr11-asm/0-seqdb/run-P473d95153387b9.bash.stdout 2>| /hlilab/chhy/Peregrine/HG002-34X-chr11-asm/0-seqdb/run-P473d95153387b9.bash.stderr'
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 0.0s -- another 1 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 0.30000000000000004s -- another 2 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 1.2000000000000002s -- another 3 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 2.9999999999999996s -- another 4 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 6.0s -- another 5 loop iterations)
INFO:pypeflow.simple_pwatcher_bridge:(slept for another 10.500000000000002s -- another 6 loop iterations)
ERROR:pypeflow.simple_pwatcher_bridge:Task Node(../../hlilab/chhy/Peregrine/HG002-34X-chr11-asm/0-seqdb) failed with exit-code=1
ERROR:pypeflow.simple_pwatcher_bridge:Some tasks are recently_done but not satisfied: {Node(../../hlilab/chhy/Peregrine/HG002-34X-chr11-asm/0-seqdb)}
ERROR:pypeflow.simple_pwatcher_bridge:ready: set()
        submitted: set()
ERROR:pwatcher.blocking:Noop. We cannot kill blocked threads. Hopefully, everything will die on SIGTERM.
Traceback (most recent call last):
  File "/opt/conda/envs/peregrine/bin/pg_run.py", line 4, in <module>
    __import__('pkg_resources').run_script('peregrine==0+unknown', 'pg_run.py')
  File "/opt/conda/envs/peregrine/lib/python3.7/site-packages/pkg_resources/__init__.py", line 666, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/opt/conda/envs/peregrine/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1462, in run_script
    exec(code, namespace, namespace)
  File "/opt/conda/envs/peregrine/lib/python3.7/site-packages/peregrine-0+unknown-py3.7-linux-x86_64.egg/EGG-INFO/scripts/pg_run.py", line 689, in <module>
    main(args)
  File "/opt/conda/envs/peregrine/lib/python3.7/site-packages/peregrine-0+unknown-py3.7-linux-x86_64.egg/EGG-INFO/scripts/pg_run.py", line 587, in main
    read_db_abs_prefix, read_db = run_build_db(wf, args, seq_dataset_lst)
  File "/opt/conda/envs/peregrine/lib/python3.7/site-packages/peregrine-0+unknown-py3.7-linux-x86_64.egg/EGG-INFO/scripts/pg_run.py", line 222, in run_build_db
    wf.refreshTargets()
  File "/opt/conda/envs/peregrine/lib/python3.7/site-packages/pypeflow-2.1.1+git.d63b0e79f5a7b2d370b7de84a890f88271afa476-py3.7.egg/pypeflow/simple_pwatcher_bridge.py", line 278, in refreshTargets
    self._refreshTargets(updateFreq, exitOnFailure)
  File "/opt/conda/envs/peregrine/lib/python3.7/site-packages/pypeflow-2.1.1+git.d63b0e79f5a7b2d370b7de84a890f88271afa476-py3.7.egg/pypeflow/simple_pwatcher_bridge.py", line 362, in _refreshTargets
    raise Exception(msg)
Exception: Some tasks are recently_done but not satisfied: {Node(../../hlilab/chhy/Peregrine/HG002-34X-chr11-asm/0-seqdb)}

Could you help me to solve this problem? Thank you so much.

cschin commented 4 years ago

This is likely caused by that the process inside the docker image can not find your data. You will need to use "-v" option when you run docker. A Docker container is like a lightweight virtual machine, you will need to let the container knows which directory that you want to see inside the container by "mounting" the host directory in the container by using "-v" option.

If you like to know more, check https://docs.docker.com/engine/reference/commandline/run/ and find "Bind mount a volume".

cschin commented 4 years ago

one thing to check is to ensure you use the full paths (as seen by the docker container) to specify the input files and the output directory.

chhylp123 commented 4 years ago

Thank you so much! BTW, I have a quick question for '--with-alt': If I have the assemblies generated without '--with-alt', how can I get 'a_ctg_cns.fa' without running the whole pipeline with '--with-alt'? It seems the 'p_ctg_cns.fa' without '--with-alt' is the same as the 'p_ctg_cns.fa' with '--with-alt'.

woodoo46 commented 3 years ago

Hi, there,

I am having the same error message, I am using the pg_run.py command instead of docker. Any idea of it? The command is this

pg_run.py asm \ /home/run/reads.ls 6 6 6 6 6 6 6 6 6 \ --with-consensus --shimmer-r 3 \ --best_n_ovlp 8 \ --output /data/CCS/Sample_CCS/Peregrine/out"

Where reads.ls is

/data/CCS/Sample_CCS/reads/read1.fastq /data/CCS/Sample_CCS/reads/read2.fastq

Many thanks!

cschin commented 3 years ago

@woodoo46 can you find the part that shows error in the log? check the paths in your input files too. Thanks.

woodoo46 commented 3 years ago

Yes. It says this

/usr/bin/time: cannot run ovlp_to_graph.py: No such file or directory

Since the program was installed by the Admin, I am not quite sure where this is installed. I looked at the directory where "pg_run.py " was installed, there was no file named "ovlp_to_graph.py"; but I found it in the original source code directory. Does this mean our installation is not complete? Or we have to copy the file manually?

Thanks!