search

PacificBiosciences / FALCON

FALCON: experimental PacBio diploid assembler -- Out-of-date -- Please use a binary release: https://github.com/PacificBiosciences/FALCON_unzip/wiki/Binaries
https://github.com/PacificBiosciences/FALCON_unzip/wiki/Binaries
Other
204 stars 103 forks source link

Exception: Call '/bin/bash user_script.sh' returned 256. #687

Closed llengcn closed 5 years ago

llengcn commented 5 years ago

Dear falcon developers,

Thank you advance for your help.

Here's my linux version information:

cat /proc/version 
Linux version 2.6.32-358.el6.x86_64 (mockbuild@x86-022.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC) ) #1 SMP Tue Jan 29 11:47:41 EST 2013

I have installed falcon via conda, as suggested in this page (https://github.com/PacificBiosciences/pb-assembly#availability):

conda create -n denovo_asm
conda activate denovo_asm
conda install pb-assembly

here's the output of conda list

/# packages in environment at /gpfs01/lengliang/miniconda2/envs/denovo_asm:
/#
/# Name                    Version                   Build  Channel
bedtools                  2.28.0               hdf88d34_0    bioconda
blas                      1.0                         mkl  
blasr                     5.3.3                h707fff8_0    bioconda
blasr_libcpp              5.3.3                h707fff8_0    bioconda
bwa                       0.7.17               h84994c4_5    bioconda
bzip2                     1.0.6                h14c3975_5  
ca-certificates           2019.1.23                     0  
certifi                   2019.3.9                 py37_0  
curl                      7.64.1               hbc83047_0  
decorator                 4.4.0                    py37_1  
falcon-kit                1.4.1                    pypi_0    pypi
falcon-phase              1.1.0                    pypi_0    pypi
falcon-unzip              1.3.0                    pypi_0    pypi
future                    0.17.1                   py37_0  
hdf5                      1.10.4               hb1b8bf9_0  
htslib                    1.9                  ha228f0b_7    bioconda
intel-openmp              2019.3                      199  
krb5                      1.16.1               h173b8e3_7  
libcurl                   7.64.1               h20c2e04_0  
libdeflate                1.0                  h14c3975_1    bioconda
libedit                   3.1.20181209         hc058e9b_0  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 8.2.0                hdf63c60_1  
libgfortran-ng            7.3.0                hdf63c60_0  
libssh2                   1.8.2                h1ba5d50_0  
libstdcxx-ng              8.2.0                hdf63c60_1  
minimap2                  2.17                 h84994c4_0    bioconda
mkl                       2019.3                      199    <unknown>
mkl_fft                   1.0.12           py37ha843d7b_0  
mkl_random                1.0.2            py37hd81dba3_0  
mummer4                   4.0.0beta2      pl526hf484d3e_4    bioconda
ncurses                   6.1                  he6710b0_1  
networkx                  2.3                        py_0  
nim-falcon                0.0.0                         0    bioconda
numpy                     1.16.3           py37h7e9f1db_0  
numpy-base                1.16.3           py37hde5b4d6_0  
openssl                   1.1.1b               h7b6447c_1  
pb-assembly               0.0.6                    py37_5    bioconda
pb-dazzler                0.0.0                h470a237_0    bioconda
pb-falcon                 0.3.0            py37hc0aa232_1    bioconda
pbbam                     0.23.0               h88345a9_0    bioconda
pbcopper                  0.4.2                h3f2a3d2_1    bioconda
pbgcpp                    0.0.1                         0    bioconda
pbmm2                     1.0.0                ha888412_0    bioconda
perl                      5.26.2               h14c3975_0  
pip                       19.1                     py37_0  
python                    3.7.3                h0371630_0  
python-edlib              1.2.4            py37h6bb024c_0    bioconda
python-intervaltree       3.0.2                      py_0    bioconda
python-msgpack            0.6.1            py37h6bb024c_0    bioconda
python-sortedcontainers   2.1.0                      py_0    bioconda
racon                     1.3.2                he860b03_1    bioconda
readline                  7.0                  h7b6447c_5  
samtools                  1.9                 h8571acd_11    bioconda
setuptools                41.0.1                   py37_0  
sqlite                    3.28.0               h7b6447c_0  
tk                        8.6.8                hbc83047_0  
wheel                     0.33.1                   py37_0  
xz                        5.2.4                h14c3975_4  
zlib                      1.2.11               h7b6447c_3

Then I downloaded example data set, i.e. greg200k-sv2

(denovo_asm) [lengliang@mgt03 run]$ ls
arab  ecoli  ecoli2  greg200k-sv2  lambda  lambda-hgap  lambda-smb14  synth0

I went to greg200k-sv2 folder and run falcon as:

fc_run fc_run.cfg

My fc_run.log changed just one option: 

use_tmpdir = /gpfs02/lengliang/tmp1

and here's this fc_run.cfg:

[General]
#sge_option_da =
#sge_option_la =
#sge_option_pda =
#sge_option_pla =
#sge_option_cns =
#sge_option_fc =
#pwatcher_type = blocking
#job_type = string
#job_queue = bash -C ${CMD} >| ${STDOUT_FILE} 2>| ${STDERR_FILE}
/# list of files of the initial bas.h5 files
input_fofn = input.fofn
#input_fofn = preads.fofn
input_type = raw
#input_type = preads
#openending = True
/# The length cutoff used for seed reads used for initial mapping
length_cutoff = 1000
genome_size = 200000
#seed_coverage = 60
/# The length cutoff used for seed reads usef for pre-assembly
length_cutoff_pr = 1
#pa_REPmask_code = 1,300;0,300;0,300
pa_daligner_option   = -e.8 -l3200 -k18 -h480  -w8 -s100
ovlp_daligner_option = -e.9 -l2500 -k24 -h1024 -w6 -s100
pa_HPCdaligner_option   =  -v -B128 -M24
ovlp_HPCdaligner_option =  -v -B128 -M24
#pa_HPCTANmask_option = -k18 -h480 -w8 -e.8 -s100
#pa_HPCREPmask_option = -k18 -h480 -w8 -e.8 -s100
pa_DBsplit_option = -a -x500 -s4
ovlp_DBsplit_option = -s5
falcon_sense_option = --output-multi --min-cov-aln 4 --min-idt 0.70 --min-cov 4 --max-n-read 200 --
n-core 8
falcon_sense_skip_contained = False
overlap_filtering_setting = --max-diff 120 --max-cov 120 --min-cov 2 --n-core 8
#dazcon = 1
[job.defaults]
job_type = local
use_tmpdir = /gpfs02/lengliang/tmp1
pwatcher_type = blocking
job_type = string
submit = bash -C ${CMD} >| ${STDOUT_FILE} 2>| ${STDERR_FILE}
#job_queue = bash -C ${CMD}
/# By dropping STD*_FILE, we see all output on the console.
/# That helps debugging in TravisCI/Bamboo.
NPROC=4
#njobs=1
[job.step.da]
[job.step.pda]
[job.step.la]
#NPROC=1
[job.step.pla]
#NPROC=1
[job.step.cns]
#njobs=1
NPROC=8
[job.step.asm]
NPROC=32

Unfortunately, falcon stoped with such information:

[INFO]Popen: 'bash -C /gpfs01/lengliang/miniconda2/envs/denovo_asm/lib/python3.7/site-packages/pwat
cher/mains/job_start.sh >| /gpfs02/lengl/project/tmp/falcon/test_data/FALCON-examples/run/greg200k-
sv2/2-asm-falcon/run-P759a6a113f94bf.bash.stdout 2>| /gpfs02/lengl/project/tmp/falcon/test_data/FAL
CON-examples/run/greg200k-sv2/2-asm-falcon/run-P759a6a113f94bf.bash.stderr'
[INFO](slept for another 0.9s -- another 4 loop iterations)
[INFO](slept for another 1.5s -- another 5 loop iterations)
[INFO](slept for another 5.1s -- another 6 loop iterations)
[ERROR]Task Node(2-asm-falcon) failed with exit-code=1
[ERROR]Some tasks are recently_done but not satisfied: {Node(2-asm-falcon)}
[ERROR]ready: set()
        submitted: set()
[ERROR]Noop. We cannot kill blocked threads. Hopefully, everything will die on SIGTERM.
Traceback (most recent call last):
  File "/gpfs01/lengliang/miniconda2/envs/denovo_asm/bin/fc_run", line 11, in <module>
    load_entry_point('falcon-kit==1.4.1', 'console_scripts', 'fc_run')()
  File "/gpfs01/lengliang/miniconda2/envs/denovo_asm/lib/python3.7/site-packages/falcon_kit/mains/r
un1.py", line 706, in main
    main1(argv[0], args.config, args.logger)
  File "/gpfs01/lengliang/miniconda2/envs/denovo_asm/lib/python3.7/site-packages/falcon_kit/mains/run1.py", line 73, in main1
    input_fofn_fn=input_fofn_fn,
  File "/gpfs01/lengliang/miniconda2/envs/denovo_asm/lib/python3.7/site-packages/falcon_kit/mains/run1.py", line 460, in run
    wf.refreshTargets()
  File "/gpfs01/lengliang/miniconda2/envs/denovo_asm/lib/python3.7/site-packages/pypeflow/simple_pw
atcher_bridge.py", line 278, in refreshTargets
    self._refreshTargets(updateFreq, exitOnFailure)
  File "/gpfs01/lengliang/miniconda2/envs/denovo_asm/lib/python3.7/site-packages/pypeflow/simple_pw
atcher_bridge.py", line 362, in _refreshTargets
    raise Exception(msg)
Exception: Some tasks are recently_done but not satisfied: {Node(2-asm-falcon)}

And the last lines of 2-asm-falcon/run-P759a6a113f94bf.bash.stderr is as follows:

/# Given preads.ovl,
/# write sg_edges_list, c_path, utg_data, ctg_paths.
time python3 -m falcon_kit.mains.ovlp_to_graph  --min-len 1 --overlap-file preads.ovl >| fc_ovlp_to_graph.log + python3 -m falcon_kit.mains.ovlp_to_graph --min-len 1 --overlap-file preads.ovl
falcon-kit 1.4.1
Traceback (most recent call last):
  File "/gpfs01/lengliang/miniconda2/envs/denovo_asm/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/gpfs01/lengliang/miniconda2/envs/denovo_asm/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/gpfs01/lengliang/miniconda2/envs/denovo_asm/lib/python3.7/site-packages/falcon_kit/mains/ovlp_to_graph.py", line 1600, in <module>
    main(sys.argv)
  File "/gpfs01/lengliang/miniconda2/envs/denovo_asm/lib/python3.7/site-packages/falcon_kit/mains/ovlp_to_graph.py", line 1596, in main
    ovlp_to_graph(args)
  File "/gpfs01/lengliang/miniconda2/envs/denovo_asm/lib/python3.7/site-packages/falcon_kit/mains/ovlp_to_graph.py", line 1418, in ovlp_to_graph
    ug2 = identify_spurs(ug, u_edge_data, 50000)
  File "/gpfs01/lengliang/miniconda2/envs/denovo_asm/lib/python3.7/site-packages/falcon_kit/mains/ovlp_to_graph.py", line 1207, in identify_spurs
    for s, t, v in ug2.out_edges(v1, keys=True):
  File "/gpfs01/lengliang/miniconda2/envs/denovo_asm/lib/python3.7/site-packages/networkx/classes/reportviews.py", line 806, in <genexpr>
    for nbr, kd in nbrs.items() for k, dd in kd.items())
RuntimeError: dictionary changed size during iteration
2019-05-16 16:55:57,495 - root - WARNING - Call '/bin/bash user_script.sh' returned 256.
2019-05-16 16:55:57,495 - root - INFO - CD: '/gpfs02/lengliang/tmp1/lengliang/pypetmp//gpfs02/lengl
/project/tmp/falcon/test_data/FALCON-examples/run/greg200k-sv2/2-asm-falcon' -> '/gpfs02/lengl/project/tmp/falcon/test_data/FALCON-examples/run/greg200k-sv2/2-asm-falcon'
2019-05-16 16:55:57,495 - root - INFO - CD: '/gpfs02/lengl/project/tmp/falcon/test_data/FALCON-exam
ples/run/greg200k-sv2/2-asm-falcon' -> '/gpfs02/lengl/project/tmp/falcon/test_data/FALCON-examples/
run/greg200k-sv2/2-asm-falcon'
2019-05-16 16:55:57,496 - root - CRITICAL - Error in /gpfs01/lengliang/miniconda2/envs/denovo_asm/l
ib/python3.7/site-packages/pypeflow/do_task.py with args="{'json_fn': '/gpfs02/lengl/project/tmp/fa
lcon/test_data/FALCON-examples/run/greg200k-sv2/2-asm-falcon/task.json',\n 'timeout': 30,\n 'tmpdir': '/gpfs02/lengliang/tmp1'}"
Traceback (most recent call last):
  File "/gpfs01/lengliang/miniconda2/envs/denovo_asm/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/gpfs01/lengliang/miniconda2/envs/denovo_asm/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/gpfs01/lengliang/miniconda2/envs/denovo_asm/lib/python3.7/site-packages/pypeflow/do_task.py", line 280, in <module>
    main()
  File "/gpfs01/lengliang/miniconda2/envs/denovo_asm/lib/python3.7/site-packages/pypeflow/do_task.py", line 272, in main
    run(**vars(parsed_args))
  File "/gpfs01/lengliang/miniconda2/envs/denovo_asm/lib/python3.7/site-packages/pypeflow/do_task.py", line 266, in run
    run_cfg_in_tmpdir(cfg, tmpdir, '.')
  File "/gpfs01/lengliang/miniconda2/envs/denovo_asm/lib/python3.7/site-packages/pypeflow/do_task.py", line 241, in run_cfg_in_tmpdir
    run_bash(bash_template, myinputs, myoutputs, parameters)
  File "/gpfs01/lengliang/miniconda2/envs/denovo_asm/lib/python3.7/site-packages/pypeflow/do_task.py", line 200, in run_bash
    util.system(cmd)
  File "/gpfs01/lengliang/miniconda2/envs/denovo_asm/lib/python3.7/site-packages/pypeflow/io.py", line 27, in syscall
    raise Exception(msg)
Exception: Call '/bin/bash user_script.sh' returned 256.
pwd) (If you see -terminal database is inaccessible- you are using the python bin-wrapper, so you will not get diagnostic info. No big deal. This process is crashing anyway.)"
pwd) (If you see -terminal database is inaccessible- you are using the python bin-wrapper, so you will not get diagnostic info. No big deal. This process is crashing anyway.)
pwd
+++ pwd
++ echo 'FAILURE. Running top in /gpfs02/lengl/project/tmp/falcon/test_data/FALCON-examples/run/gre
g200k-sv2/2-asm-falcon (If you see -terminal database is inaccessible- you are using the python bin
-wrapper, so you will not get diagnostic info. No big deal. This process is crashing anyway.)'
++ rm -f top.txt
++ which python
++ which top
++ env -u LD_LIBRARY_PATH top -b -n 1
++ env -u LD_LIBRARY_PATH top -b -n 1
++ pstree -apl
real    0m11.439s
user    0m2.066s
sys     0m0.628s
+ finish
+ echo 'finish code: 1'

So, what's wrong in this process? Thank you very much, Liang

pb-cdunn commented 5 years ago 
RuntimeError: dictionary changed size during iteration

I think networkx changed after a release. We've fixed our code for the latest networkx in all the spots where we saw this error. I think it will work for you now.

conda update --all
llengcn commented 5 years ago

Thank you very much for your reply!