search

PacificBiosciences / FALCON_unzip

Making diploid assembly becomes common practice for genomic study
BSD 3-Clause Clear License
30 stars 18 forks source link

[ERROR]Task Node(1-preads_ovl) failed with exit-code=256 #116

Open AteeqMKhaliq opened 6 years ago

AteeqMKhaliq commented 6 years ago

Hello All,

I am having an error while using falcon, the error is as follows, I am using Human reads. It would be really helpful if you can please help me.

`**[ERROR]**Task Node(1-preads_ovl) failed with exit-code=256
[INFO]recently_satisfied: set([])
[INFO]Num satisfied in this iteration: 0
[INFO]Num still unsatisfied: 3
[ERROR]Some tasks are recently_done but not satisfied: set([Node(1-preads_ovl)])
[ERROR]ready: set([])
    submitted: set([Node(0-rawreads/report)])
Traceback (most recent call last):
  File "/gpfs/software/genomics/falcon/fc_env_180305/bin/fc_run.py", line 11, in <module>
    load_entry_point('falcon-kit==1.0+git.3fb5b0d1d2003805ad0869919ff6722eac176de0', 'console_scripts', 'fc_run.py')()
  File "/gpfs/software/genomics/falcon/fc_env_180305/lib/python2.7/site-packages/falcon_kit/mains/run1.py", line 599, in main
    main1(argv[0], args.config, args.logger)
  File "/gpfs/software/genomics/falcon/fc_env_180305/lib/python2.7/site-packages/falcon_kit/mains/run1.py", line 159, in main1
    input_fofn_plf=input_fofn_plf,
  File "/gpfs/software/genomics/falcon/fc_env_180305/lib/python2.7/site-packages/falcon_kit/mains/run1.py", line 481, in run
    parameters={},
  File "/gpfs/software/genomics/falcon/fc_env_180305/lib/python2.7/site-packages/falcon_kit/pype.py", line 71, in gen_parallel_tasks
    wf.refreshTargets()
  File "/gpfs/software/genomics/falcon/fc_env_180305/lib/python2.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 273, in refreshTargets
    self._refreshTargets(updateFreq, exitOnFailure)
  File "/gpfs/software/genomics/falcon/fc_env_180305/lib/python2.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 339, in _refreshTargets
    raise Exception(msg)
Exception: Some tasks are recently_done but not satisfied: set([Node(1-preads_ovl)])

` My Confif file is as follows:

`[General]
input_fofn = input.fofn

input_type = raw

#length_cutoff = 23000
length_cutoff = 16000

#length_cutoff_pr = 23000
length_cutoff_pr = 6000

genome_size = 3000000000000

#jobqueue = default

job_type = local 

sge_option_da = -n 32 -q jobqueue
sge_option_la = -n 32 -q jobqueue
sge_option_pda = -n 32 -q jobqueue
sge_option_pla = -n 32 -q jobqueue
sge_option_fc = -n 32 -q jobqueue
sge_option_cns = -n 32 -q jobqueue

#sge_option_da = -pe smp 4 -q %(jobqueue)s
#sge_option_la = -pe smp 20 -q %(jobqueue)s
#sge_option_pda = -pe smp 6 -q %(jobqueue)s
#sge_option_pla = -pe smp 16 -q %(jobqueue)s
#sge_option_fc = -pe smp 24 -q %(jobqueue)s
#sge_option_cns = -pe smp 8 -q %(jobqueue)s

skip_ckeck = True

pa_concurrent_jobs = 384
cns_concurrent_jobs = 384
ovlp_concurrent_jobs = 384

pa_HPCdaligner_option =  -v -dal128 -t16 -e0.75 -M24 -l4800 -k18 -h480 -w8 -s100
ovlp_HPCdaligner_option =  -v -dal128  -M24 -k24 -h1024 -e.96 -l2500 -s100 

pa_DBsplit_option = -a -x500 -s400
ovlp_DBsplit_option = -s400

falcon_sense_option = --output_multi --min_idt 0.70 --min_cov 4 --max_n_read 200 --n_core 8 
falcon_sense_skip_contained = True

overlap_filtering_setting = --max_diff 60 --max_cov 60 --min_cov 0 --n_core 12
#use_tmpdir = false

` My StdErr output is as follows

`+ python2.7 -m pwatcher.mains.fs_heartbeat --directory=/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/0-rawreads --heartbeat-file=/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/mypwatcher/heartbeats/heartbeat-Pe07e8148cf7926 --exit-file=/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/mypwatcher/exits/exit-Pe07e8148cf7926 --rate=10.0 /bin/bash run.sh
Namespace(command=['/bin/bash', 'run.sh'], directory='/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/0-rawreads', exit_file='/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/mypwatcher/exits/exit-Pe07e8148cf7926', heartbeat_file='/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/mypwatcher/heartbeats/heartbeat-Pe07e8148cf7926', rate=10.0)

cwd:'/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/0-rawreads'
hostname=hpctestlogin1.research.sidra.local
heartbeat_fn='/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/mypwatcher/heartbeats/heartbeat-Pe07e8148cf7926'
exit_fn='/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/mypwatcher/exits/exit-Pe07e8148cf7926'
sleep_s=10.0
before setpgid: pid=4805 pgid=4784
 after setpgid: pid=4805 pgid=4805
In cwd: /gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/0-rawreads, Blocking call: '/bin/bash run.sh'
cd /gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/0-rawreads
+ cd /gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/0-rawreads
bash task.sh
+ bash task.sh
2018-04-17 12:59:19,279 - root - DEBUG - Running "/gpfs/software/genomics/falcon/fc_env_180305/lib/python2.7/site-packages/pypeflow/do_task.py /gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/0-rawreads/task.json"
2018-04-17 12:59:19,280 - root - DEBUG - Checking existence of '/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/0-rawreads/task.json' with timeout=60
2018-04-17 12:59:19,280 - root - DEBUG - Loading JSON from '/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/0-rawreads/task.json'
2018-04-17 12:59:19,280 - root - DEBUG - {u'bash_template_fn': u'template.sh',
 u'inputs': {u'config': u'/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/config.json',
             u'raw_reads_fofn': u'/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/input.fofn'},
 u'outputs': {u'db_build_done': u'rdb_build_done',
              u'length_cutoff': u'length_cutoff',
              u'raw_reads_db': u'raw_reads.db',
              u'run_jobs': u'run_jobs.sh'},
 u'parameters': {u'topdir': u'..'}}
2018-04-17 12:59:19,280 - root - DEBUG - CD: '/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/0-rawreads' <- '/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/0-rawreads'
2018-04-17 12:59:19,280 - root - DEBUG - Checking existence of u'/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/input.fofn' with timeout=60
2018-04-17 12:59:19,281 - root - DEBUG - Checking existence of u'/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/config.json' with timeout=60
2018-04-17 12:59:19,281 - root - DEBUG - Checking existence of u'template.sh' with timeout=60
2018-04-17 12:59:19,281 - root - DEBUG - CD: '/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/0-rawreads' <- '/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/0-rawreads'
2018-04-17 12:59:19,281 - root - INFO - bash user_script.sh
hostname
+ hostname
pwd
+ pwd
date
+ date
Substitution will be similar to snakemake "shell".
 python -m falcon_kit.mains.build_rdb --input-fofn-fn=/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/input.fofn --config-fn=/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/config.json --run-jobs-fn=run_jobs.sh --length-cutoff-fn=length_cutoff --job-done-fn=rdb_build_done
+ python -m falcon_kit.mains.build_rdb --input-fofn-fn=/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/input.fofn --config-fn=/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/config.json --run-jobs-fn=run_jobs.sh --length-cutoff-fn=length_cutoff --job-done-fn=rdb_build_done
INFO:root:Building rdb from '/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/input.fofn' ('/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/config.json'), to write 'run_jobs.sh'
INFO:root:u"Deserializing from '/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/config.json'"
INFO:root:u"  Opened for read: '/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/config.json'"
INFO:root:u'  Read 0.0MB as json'
INFO:root:u'  Deserialized 52 records'
INFO:root:$(u'bash -vex build_rdb.sh')
module () {  eval `/usr/bin/modulecmd bash $*`
}
#!/bin/bash
set -vex
+ set -vex
echo "PBFALCON_ERRFILE=$PBFALCON_ERRFILE"
+ echo PBFALCON_ERRFILE=
set -o pipefail
+ set -o pipefail
#fc_fasta2fasta < my.input.fofn >| fc.fofn
while read fn; do  cat  $fn | fasta2DB -v raw_reads -i${fn##*/}; done < my.input.fofn
+ read fn
+ cat /gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/reads.fasta
+ fasta2DB -v raw_reads -ireads.fasta
Adding 'reads.fasta.fasta' ...
+ read fn
#cat fc.fofn | xargs rm -f
DBsplit -a -x500 -s400 raw_reads
+ DBsplit -a -x500 -s400 raw_reads
DBdust  raw_reads
+ DBdust raw_reads
LB=$(cat raw_reads.db | LD_LIBRARY_PATH= awk '$1 == "blocks" {print $3}')
++ cat raw_reads.db
++ LD_LIBRARY_PATH=
++ awk '$1 == "blocks" {print $3}'
+ LB=1
rm -f run_jobs.sh
+ rm -f run_jobs.sh
CUTOFF=16000
+ CUTOFF=16000
echo -n $CUTOFF >| length_cutoff
+ echo -n 16000
HPC.daligner -v -B128 -t16 -e0.75 -M24 -l4800 -k18 -h480 -w8 -s100 -mdust -H$CUTOFF raw_reads 1-$LB >| run_jobs.sh
+ HPC.daligner -v -B128 -t16 -e0.75 -M24 -l4800 -k18 -h480 -w8 -s100 -mdust -H16000 raw_reads 1-1

touch rdb_build_done
+ touch rdb_build_done
touch rdb_build_done
+ touch rdb_build_done

date
+ date
2018-04-17 12:59:31,340 - root - DEBUG - CD: '/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/0-rawreads' -> '/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/0-rawreads'
2018-04-17 12:59:31,340 - root - DEBUG - Checking existence of u'length_cutoff' with timeout=60
2018-04-17 12:59:31,340 - root - DEBUG - Checking existence of u'run_jobs.sh' with timeout=60
2018-04-17 12:59:31,340 - root - DEBUG - Checking existence of u'raw_reads.db' with timeout=60
2018-04-17 12:59:31,340 - root - DEBUG - Checking existence of u'rdb_build_done' with timeout=60
2018-04-17 12:59:31,340 - root - DEBUG - CD: '/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/0-rawreads' -> '/gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/0-rawreads'

real    0m12.114s
user    0m11.850s
sys     0m0.374s
touch /gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/0-rawreads/run.sh.done
+ touch /gpfs/projects/sysbio/development/denovo/test/2_assembly_falcon_standalone/0-rawreads/run.sh.done
 returned: 0

Thanks a lot for your help. Regards, Ateeq `

rob123king commented 6 years ago

tmp directory not full and enough resources available memory? I think I was having an error 256 when not enough memory was available