search

PacificBiosciences / FALCON

FALCON: experimental PacBio diploid assembler -- Out-of-date -- Please use a binary release: https://github.com/PacificBiosciences/FALCON_unzip/wiki/Binaries
https://github.com/PacificBiosciences/FALCON_unzip/wiki/Binaries
Other
205 stars 102 forks source link

Falcon running [ERROR]Task Fred{'URL': 'task://localhost/d_0003_raw_reads'} failed with exit-code=256 #445

Open yuyutangTW opened 8 years ago

yuyutangTW commented 8 years ago

Python 2.7.9 pip 8.1.2 from /usr/local/python-2.7/lib/python2.7/site-packages (python 2.7) setuptools 27.3.0 from /usr/local/python-2.7/lib/python2.7/site-packages (Python 2.7)

I have set parameters

[General]
input_fofn = input.fofn
input_type = raw
length_cutoff = 1000
length_cutoff_pr = 1000

job_type = local
sge_option_da = -pe smp 8
sge_option_la = -pe smp 2
sge_option_pda = -pe smp 8
sge_option_pla = -pe smp 2
sge_option_fc = -pe smp 24
sge_option_cns = -pe smp 8

pa_concurrent_jobs = 32
cns_concurrent_jobs = 32
ovlp_concurrent_jobs = 32

pa_HPCdaligner_option =  -v -dal4 -t16 -e.70 -l1000 -s1000
ovlp_HPCdaligner_option = -v -dal4 -t32 -h60 -e.96 -l500 -s1000

pa_DBsplit_option = -x500 -s50
ovlp_DBsplit_option = -x500 -s50
overlap_filtering_setting = --max_diff 100 --max_cov 100 --min_cov 20 --bestn 10
falcon_sense_option = --output_multi --min_idt 0.70 --min_cov 4 --max_n_read 200 --n_core 4

i have run error 

[INFO]Success ('done'). Joining 'task://localhost/d_001a_raw_reads'...
[INFO]Success ('done'). Joining 'task://localhost/d_0012_raw_reads'...
[INFO]Success ('done'). Joining 'task://localhost/d_000b_raw_reads'...
[INFO]Success ('done'). Joining 'task://localhost/d_001c_raw_reads'...
[INFO]Success ('done'). Joining 'task://localhost/d_000c_raw_reads'...
[INFO]Success ('done'). Joining 'task://localhost/d_001d_raw_reads'...
[INFO]Success ('done'). Joining 'task://localhost/d_0018_raw_reads'...
# [ERROR]Task Fred{'URL': 'task://localhost/d_0003_raw_reads'} failed with exit-code=256
# [INFO]Failure ('fail'). Joining 'task://localhost/d_0003_raw_reads'...
[INFO]Success ('done'). Joining 'task://localhost/d_0004_raw_reads'...
[INFO]Success ('done'). Joining 'task://localhost/d_000d_raw_reads'...
[INFO]Success ('done'). Joining 'task://localhost/d_0008_raw_reads'...
[INFO]Success ('done'). Joining 'task://localhost/d_0000_raw_reads'...
[INFO]Success ('done'). Joining 'task://localhost/d_0016_raw_reads'...
[INFO]Success ('done'). Joining 'task://localhost/d_000e_raw_reads'...
# [INFO]_refreshTargets() finished with no thread running and no new job to submit
[CRITICAL]Any exception caught in RefreshTargets() indicates an unrecoverable error. Shutting down...
/bip7_disk/yuyu105/FALCON-integrate/pypeFLOW/pypeflow/controller.py:537: UserWarning:
            "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
            "! Please wait for all threads / processes to terminate !"
            "! Also, maybe use 'ps' or 'qstat' to check all threads,!"
            "! processes and/or jobs are terminated cleanly.        !"
            "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
  warnings.warn(shutdown_msg)
[WARNING]#tasks=32, #alive=0
Traceback (most recent call last):
  File "./fc_run.py", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/bip7_disk/yuyu105/FALCON-integrate/FALCON/src/py_scripts/fc_run.py", line 5, in <module>
    main(sys.argv)
  File "/bip7_disk/yuyu105/FALCON-integrate/FALCON/falcon_kit/mains/run1.py", line 576, in main
    main1(argv[0], args.config, args.logger)
  File "/bip7_disk/yuyu105/FALCON-integrate/FALCON/falcon_kit/mains/run1.py", line 353, in main1
    setNumThreadAllowed=PypeProcWatcherWorkflow.setNumThreadAllowed)
  File "/bip7_disk/yuyu105/FALCON-integrate/FALCON/falcon_kit/mains/run1.py", line 431, in run
    wf.refreshTargets(exitOnFailure=exitOnFailure)
  File "/bip7_disk/yuyu105/FALCON-integrate/pypeFLOW/pypeflow/controller.py", line 548, in refreshTargets
    raise Exception('Caused by:\n' + tb)
Exception: Caused by:
Traceback (most recent call last):
  File "/bip7_disk/yuyu105/FALCON-integrate/pypeFLOW/pypeflow/controller.py", line 523, in refreshTargets
    rtn = self._refreshTargets(task2thread, objs = objs, callback = callback, updateFreq = updateFreq, exitOnFailure = exitOnFailure)
  File "/bip7_disk/yuyu105/FALCON-integrate/pypeFLOW/pypeflow/controller.py", line 750, in [_refreshTargets](url)
    failedJobCount, succeededJobCount))
LateTaskFailureError: Counted a total of 1 failure(s) and 31 success(es)
..

my /0-rawreads/pwatcher.dir file is information

> `+ python2.7 -m pwatcher.mains.fs_heartbeat --directory=/bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads --heartbeat-file=/bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/mypwatcher/heartbeats/heartbeat-Jd1e63a2c0205f4bf15f25e7e21b2c0e2942132e7fbd30a9ec9384f11734409b7 --exit-file=/bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/mypwatcher/exits/exit-Jd1e63a2c0205f4bf15f25e7e21b2c0e2942132e7fbd30a9ec9384f11734409b7 --rate=10.0 /bin/bash prepare_rdb.sh
> Namespace(command=['/bin/bash', 'prepare_rdb.sh'], directory='/bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads', exit_file='/bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/mypwatcher/exits/exit-Jd1e63a2c0205f4bf15f25e7e21b2c0e2942132e7fbd30a9ec9384f11734409b7', heartbeat_file='/bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/mypwatcher/heartbeats/heartbeat-Jd1e63a2c0205f4bf15f25e7e21b2c0e2942132e7fbd30a9ec9384f11734409b7', rate=10.0)
> 
> cwd:'/bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads'
> hostname=bip7
> heartbeat_fn='/bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/mypwatcher/heartbeats/heartbeat-Jd1e63a2c0205f4bf15f25e7e21b2c0e2942132e7fbd30a9ec9384f11734409b7'
> exit_fn='/bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/mypwatcher/exits/exit-Jd1e63a2c0205f4bf15f25e7e21b2c0e2942132e7fbd30a9ec9384f11734409b7'
> sleep_s=10.0
> before setpgid: pid=23188 pgid=23180
>  after setpgid: pid=23188 pgid=23188
> In cwd: /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads, Blocking call: '/bin/bash prepare_rdb.sh'
> cd /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads
> + cd /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads
> trap 'touch /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads/rdb_build_done.exit' EXIT
> + trap 'touch /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads/rdb_build_done.exit' EXIT
> ls -il prepare_rdb.sub.sh
> + ls -il prepare_rdb.sub.sh
> hostname
> + hostname
> ls -il prepare_rdb.sub.sh
> + ls -il prepare_rdb.sub.sh
> time /bin/bash ./prepare_rdb.sub.sh
> + /bin/bash ./prepare_rdb.sub.sh
> #fc_fasta2fasta < /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads/input.fofn >| fc.fofn
> while read fn; do fasta2DB -v raw_reads $fn; done < /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads/input.fofn
> + read fn
> + fasta2DB -v raw_reads /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/m160311_074813_42180_c100906132550000001823204104301695_s1_p0.1f.subreads.fasta
> Adding 'm160311_074813_42180_c100906132550000001823204104301695_s1_p0.1f.subreads.fasta' ...
> + read fn
> + fasta2DB -v raw_reads /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/m160311_074813_42180_c100906132550000001823204104301695_s1_p0.2f.subreads.fasta
> Adding 'm160311_074813_42180_c100906132550000001823204104301695_s1_p0.2f.subreads.fasta' ...
> + read fn
> + fasta2DB -v raw_reads /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/m160311_074813_42180_c100906132550000001823204104301695_s1_p0.3f.subreads.fasta
> Adding 'm160311_074813_42180_c100906132550000001823204104301695_s1_p0.3f.subreads.fasta' ...
> + read fn
> #cat fc.fofn | xargs rm -f
> DBsplit -x500 -s50 raw_reads
> + DBsplit -x500 -s50 raw_reads
> #DBdust -w128 -t2.5 -m20 raw_reads
> LB=$(cat raw_reads.db | LD_LIBRARY_PATH= awk '$1 == "blocks" {print $3}')
> ++ cat raw_reads.db
> ++ LD_LIBRARY_PATH=
> ++ awk '$1 == "blocks" {print $3}'
> + LB=14
> rm -f run_jobs.sh
> + rm -f run_jobs.sh
> CUTOFF=1000
> + CUTOFF=1000
> echo -n $CUTOFF >| length_cutoff
> + echo -n 1000
> HPC.daligner -v -B4 -t16 -e.70 -l1000 -s1000  -H$CUTOFF raw_reads 1-$LB >| run_jobs.sh
> + HPC.daligner -v -B4 -t16 -e.70 -l1000 -s1000 -H1000 raw_reads 1-14
> 
> real    0m7.196s
> user    0m6.380s
> sys     0m0.802s
> touch /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads/rdb_build_done
> + touch /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads/rdb_build_done
> touch /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads/rdb_build_done.exit
> + touch /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads/rdb_build_done.exit
>  returned: 0`

my job_0003/pwatcher.dir file is information

 + python2.7 -m pwatcher.mains.fs_heartbeat --directory=/bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads/job_0003 --heartbeat-file=/bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/mypwatcher/heartbeats/heartbeat-J825015fb8607d08aea7eebe3022cd2020cbcee49a0e9a57526317e3c080cd8a5 --exit-file=/bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/mypwatcher/exits/exit-J825015fb8607d08aea7eebe3022cd2020cbcee49a0e9a57526317e3c080cd8a5 --rate=10.0 /bin/bash rj_0003.sh
Namespace(command=['/bin/bash', 'rj_0003.sh'], directory='/bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads/job_0003', exit_file='/bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/mypwatcher/exits/exit-J825015fb8607d08aea7eebe3022cd2020cbcee49a0e9a57526317e3c080cd8a5', heartbeat_file='/bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/mypwatcher/heartbeats/heartbeat-J825015fb8607d08aea7eebe3022cd2020cbcee49a0e9a57526317e3c080cd8a5', rate=10.0)

cwd:'/bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads/job_0003'
hostname=bip7
heartbeat_fn='/bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/mypwatcher/heartbeats/heartbeat-J825015fb8607d08aea7eebe3022cd2020cbcee49a0e9a57526317e3c080cd8a5'
exit_fn='/bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/mypwatcher/exits/exit-J825015fb8607d08aea7eebe3022cd2020cbcee49a0e9a57526317e3c080cd8a5'
sleep_s=10.0
before setpgid: pid=23315 pgid=23180
 after setpgid: pid=23315 pgid=23315
In cwd: /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads/job_0003, Blocking call: '/bin/bash rj_0003.sh'
cd /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads/job_0003
  + cd /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads/job_0003
trap 'touch /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads/job_0003/job_0003_done.exit' EXIT
  + trap 'touch /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads/job_0003/job_0003_done.exit' EXIT
ls -il rj_0003.sub.sh
  + ls -il rj_0003.sub.sh
hostname
+ hostname
ls -il rj_0003.sub.sh
+ ls -il rj_0003.sub.sh
time /bin/bash ./rj_0003.sub.sh
+ /bin/bash ./rj_0003.sub.sh

db_dir=/bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads
 + db_dir=/bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads
ln -sf ${db_dir}/.raw_reads.bps .
 + ln -sf /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads/.raw_reads.bps .
ln -sf ${db_dir}/.raw_reads.idx .
 + ln -sf /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads/.raw_reads.idx .
ln -sf ${db_dir}/raw_reads.db .
 + ln -sf /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads/raw_reads.db .
ln -sf ${db_dir}/.raw_reads.dust.anno .
  + ln -sf /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads/.raw_reads.dust.anno .
ln -sf ${db_dir}/.raw_reads.dust.data .
  + ln -sf /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads/.raw_reads.dust.data .
daligner -v -t16 -H1000 -e0.7 -s1000 raw_reads.11 raw_reads.8 raw_reads.9 raw_reads.10 raw_reads.11
  + daligner -v -t16 -H1000 -e0.7 -s1000 raw_reads.11 raw_reads.8 raw_reads.9 raw_reads.10 raw_reads.11
LAcheck -v raw_reads *.las
  + LAcheck -v raw_reads raw_reads.10.raw_reads.11.C0.las raw_reads.10.raw_reads.11.C1.las raw_reads.10.raw_reads.11.C2.las raw_reads.10.raw_reads.11.C3.las raw_reads.10.raw_reads.11.N0.las raw_reads.10.raw_reads.11.N1.las raw_reads.10.raw_reads.11.N2.las raw_reads.10.raw_reads.11.N3.las raw_reads.11.raw_reads.10.C0.las raw_reads.11.raw_reads.10.C1.las raw_reads.11.raw_reads.10.C2.las raw_reads.11.raw_reads.10.C3.las raw_reads.11.raw_reads.10.N0.las raw_reads.11.raw_reads.10.N1.las raw_reads.11.raw_reads.10.N2.las raw_reads.11.raw_reads.10.N3.las raw_reads.11.raw_reads.11.C0.las raw_reads.11.raw_reads.11.C1.las raw_reads.11.raw_reads.11.C2.las raw_reads.11.raw_reads.11.C3.las raw_reads.11.raw_reads.11.N0.las raw_reads.11.raw_reads.11.N1.las raw_reads.11.raw_reads.11.N2.las raw_reads.11.raw_reads.11.N3.las raw_reads.11.raw_reads.8.C0.las raw_reads.11.raw_reads.8.C1.las raw_reads.11.raw_reads.8.C2.las raw_reads.11.raw_reads.8.C3.las raw_reads.11.raw_reads.8.N0.las raw_reads.11.raw_reads.8.N1.las raw_reads.11.raw_reads.8.N2.las raw_reads.11.raw_reads.8.N3.las raw_reads.11.raw_reads.9.C0.las raw_reads.11.raw_reads.9.C1.las raw_reads.11.raw_reads.9.C2.las raw_reads.11.raw_reads.9.C3.las raw_reads.11.raw_reads.9.N0.las raw_reads.11.raw_reads.9.N1.las raw_reads.11.raw_reads.9.N2.las raw_reads.11.raw_reads.9.N3.las raw_reads.8.raw_reads.11.C0.las raw_reads.8.raw_reads.11.C1.las raw_reads.8.raw_reads.11.C2.las raw_reads.8.raw_reads.11.C3.las raw_reads.8.raw_reads.11.N0.las raw_reads.8.raw_reads.11.N1.las raw_reads.8.raw_reads.11.N2.las raw_reads.8.raw_reads.11.N3.las raw_reads.9.raw_reads.11.C0.las raw_reads.9.raw_reads.11.C1.las raw_reads.9.raw_reads.11.C2.las raw_reads.9.raw_reads.11.C3.las raw_reads.9.raw_reads.11.N0.las raw_reads.9.raw_reads.11.N1.las raw_reads.9.raw_reads.11.N2.las raw_reads.9.raw_reads.11.N3.las
  raw_reads.10.raw_reads.11.C0: 12,086 all OK
  raw_reads.10.raw_reads.11.C1: 12,290 all OK
  raw_reads.10.raw_reads.11.C2: 12,036 all OK
  raw_reads.10.raw_reads.11.C3: 12,671 all OK
  raw_reads.10.raw_reads.11.N0: 12,450 all OK
  raw_reads.10.raw_reads.11.N1: 12,338 all OK
  raw_reads.10.raw_reads.11.N2: 12,879 all OK
  raw_reads.10.raw_reads.11.N3: 12,705 all OK
  raw_reads.11.raw_reads.10.C0: 12,091 all OK
  raw_reads.11.raw_reads.10.C1: 12,287 all OK
  raw_reads.11.raw_reads.10.C2: 12,037 all OK
  raw_reads.11.raw_reads.10.C3: 12,670 all OK
  raw_reads.11.raw_reads.10.N0: 12,450 all OK
  raw_reads.11.raw_reads.10.N1: 12,336 all OK
  raw_reads.11.raw_reads.10.N2: 12,883 all OK
  raw_reads.11.raw_reads.10.N3: 12,703 all OK
  raw_reads.11.raw_reads.11.C0: 12,029 all OK
  raw_reads.11.raw_reads.11.C1: 12,158 all OK
  raw_reads.11.raw_reads.11.C2: 12,592 all OK
  raw_reads.11.raw_reads.11.C3: 12,526 all OK
  raw_reads.11.raw_reads.11.N0: 12,512 all OK
  raw_reads.11.raw_reads.11.N1: 12,690 all OK
  raw_reads.11.raw_reads.11.N2: 13,377 all OK
  raw_reads.11.raw_reads.11.N3: 12,618 all OK
  raw_reads.11.raw_reads.8.C0: 12,285 all OK
  raw_reads.11.raw_reads.8.C1: 12,185 all OK
  raw_reads.11.raw_reads.8.C2: 12,649 all OK
  raw_reads.11.raw_reads.8.C3: 12,369 all OK
  raw_reads.11.raw_reads.8.N0: 12,384 all OK
  raw_reads.11.raw_reads.8.N1: 12,253 all OK
  raw_reads.11.raw_reads.8.N2: 12,566 all OK
  raw_reads.11.raw_reads.8.N3: 12,691 all OK
  raw_reads.11.raw_reads.9.C0: 12,614 all OK
  raw_reads.11.raw_reads.9.C1: 12,469 all OK
  raw_reads.11.raw_reads.9.C2: 12,190 all OK
#   raw_reads.11.raw_reads.9.C3: Duplicate overlap (89109 vs 74581)
  raw_reads.11.raw_reads.9.N0: 13,021 all OK
  raw_reads.11.raw_reads.9.N1: 12,940 all OK
  raw_reads.11.raw_reads.9.N2: 12,827 all OK
  raw_reads.11.raw_reads.9.N3: 12,865 all OK
  raw_reads.8.raw_reads.11.C0: 12,286 all OK
  raw_reads.8.raw_reads.11.C1: 12,189 all OK
  raw_reads.8.raw_reads.11.C2: 12,652 all OK
  raw_reads.8.raw_reads.11.C3: 12,370 all OK
  raw_reads.8.raw_reads.11.N0: 12,391 all OK
  raw_reads.8.raw_reads.11.N1: 12,254 all OK
  raw_reads.8.raw_reads.11.N2: 12,568 all OK
  raw_reads.8.raw_reads.11.N3: 12,693 all OK
  raw_reads.9.raw_reads.11.C0: 12,608 all OK
  raw_reads.9.raw_reads.11.C1: 12,468 all OK
  raw_reads.9.raw_reads.11.C2: 12,192 all OK

#    raw_reads.9.raw_reads.11.C3: Duplicate overlap (74581 vs 89109)

  raw_reads.9.raw_reads.11.N0: 13,012 all OK
  raw_reads.9.raw_reads.11.N1: 12,941 all OK
  raw_reads.9.raw_reads.11.N2: 12,827 all OK
  raw_reads.9.raw_reads.11.N3: 12,866 all OK
touch /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads/job_0003/job_0003_done.exit
+ touch /bip7_disk/yuyu105/FALCON-integrate/fc_env/bin/0-rawreads/job_0003/job_0003_done.exit
 returned: 256

How to solve this problem? Thanks!

lijingjing1 commented 8 years ago

it is a known bug. see related discussion thegenemyers/DALIGNER#43

you can try to use "-s100" to avoid that. It will increase the las file sizes.

pb-cdunn commented 8 years ago

@lijingjing1, in theory that bug as has been fixed already.

@yuyia, check your SHA1 for DALIGNER. Possibly you need to update your submodules in FALCON-integrate and rebuild.