search

PacificBiosciences / FALCON-integrate

Mostly deprecated. See https://github.com/PacificBiosciences/FALCON_unzip/wiki/Binaries
https://github.com/PacificBiosciences/FALCON/wiki/Manual
31 stars 28 forks source link

need new cfg for the latest Falcon (falcon-unzip 20180808) #186

Closed hollandorange closed 6 years ago

hollandorange commented 6 years ago

Hi,

I installed the latest version of falcon (falcon-unzip20180808). When I test it using the example ecoli data and local configure file fc_run_ecoli_local.cfg, it is not working. Could you provide a new configure file which works on running the example ecoli data locally? Moreover, our server has 16 processors and 256G memory, could you give some advise on how to choose the parameters?

Thanks a lot!

$ wget https://downloads.pacbcloud.com/public/data/git-sym/ecoli.m140913_050931_42139_c100713652400000001823152404301535_s1_p0.subreads.tar.gz
$ tar -xvzf ecoli.m140913_050931_42139_c100713652400000001823152404301535_s1_p0.subreads.tar.gz

input.fofn /data/ecoli/ecoli.1.fasta

fc_run_ecoli_local.cfg

[General]
job_type = local

# list of fasta files
input_fofn = input.fofn

# input type, raw or pre-assembled reads (preads, error corrected reads)
input_type = raw
#input_type = preads

# The length cutoff used for seed reads used for error correction.
# "-1" indicates FALCON should calculate the cutoff using
# the user-defined genome length and coverage cut off
# otherwise, user can specify length cut off in bp (e.g. 2000)
length_cutoff = -1
genome_size = 4652500
seed_coverage = 30

# The length cutoff used for overalpping the preassembled reads 
length_cutoff_pr = 12000

## resource usage ##
# need this line even if running local
jobqueue = your_queue
# grid settings for...
# daligner step of raw reads
sge_option_da = -pe smp 5 -q %(jobqueue)s
# las-merging of raw reads
sge_option_la = -pe smp 20 -q %(jobqueue)s
# consensus calling for preads
sge_option_cns = -pe smp 12 -q %(jobqueue)s
# daligner on preads
sge_option_pda = -pe smp 6 -q %(jobqueue)s
# las-merging on preads
sge_option_pla = -pe smp 16 -q %(jobqueue)s
# final overlap/assembly 
sge_option_fc = -pe smp 24 -q %(jobqueue)s

# job concurrency settings for...
# all jobs
default_concurrent_jobs = 30
# preassembly
pa_concurrent_jobs = 30
# consensus calling of preads
cns_concurrent_jobs = 30
# overlap detection
ovlp_concurrent_jobs = 30

# daligner parameter options for...
# https://dazzlerblog.wordpress.com/command-guides/daligner-command-reference-guide/
# initial overlap of raw reads
pa_HPCdaligner_option =  -v -B4 -t16 -e.70 -l1000 -s1000
# overlap of preads
ovlp_HPCdaligner_option = -v -B4 -t32 -h60 -e.96 -l500 -s1000

# parameters for creation of dazzler database of...
# https://dazzlerblog.wordpress.com/command-guides/dazz_db-command-guide/
# raw reads
pa_DBsplit_option = -x500 -s50
# preads
ovlp_DBsplit_option = -x500 -s50

# settings for consensus calling for preads
falcon_sense_option = --output_multi --min_idt 0.70 --min_cov 4 --max_n_read 200 --n_core 6

# setting for filtering of final overlap of preads
overlap_filtering_setting = --max_diff 100 --max_cov 100 --min_cov 20 --bestn 10 --n_core 24

The screen output:

Exception: Missing options.

We now require both "pa_daligner_option" (stage 0) and "ovlp_daligner_option" (stage 1),
which are automatically passed along to
  HPC.daligner
  HPC.TANmask
  HPC.REPmask

These can provide additional flags:
  pa_HPCdaligner_option
  pa_HPCTANmask_option
  ovlp_HPCdaligner_option
  pa_REPmask_code (-g/-c pairs for 3 iterations, e.g. '1,20;5,15;20,10')
pb-cdunn commented 6 years ago

First, please stop using -t for daligner. Use -M64, since you have 64MB available per 4-thread process.

Change/add this part:

pa_daligner_option = -e.70 -l1000 -s1000
ovlp_daligner_option = -e.96 -l500 -s1000 -h60

pa_HPCdaligner_option =  -v -B4 -M64
ovlp_HPCdaligner_option = -v -B4 -M64

The top flags will be passed along to TANmask and REPmask too. (If you want to use REPmask, I can help you configure that if you open a new ticket and post your genome size and coverage.)

For your job configuration, please post the WARNINGS reported by pypeflow. It should become clear which sge/concurrency options should go into which job.step. section. Basically, "concurrency" becomes njobs in each section.

(You can also specify NPROC=5 for the sge options instead of "-pe smp 5", but you are currently using job_type=local anyway. Please try to switch to pwatcher_type=blocking if you want to use job-submission, as the old fs_based process-watcher is very difficult for us to support.)

This can help:

jasmynp commented 6 years ago

@pb-cdunn , thanks. I'll test now.

hollandorange commented 6 years ago

Still, not working. T_T Please help! Is it possible that you post a new tested cfg file which works on the example ecoli data.

my input.fofn file

/home/zhangyanju/data/ecoli/ecoli.1.fasta
/home/zhangyanju/data/ecoli/ecoli.2.fasta
/home/zhangyanju/data/ecoli/ecoli.3.fasta

my cfg file

[General]
input_fofn = input.fofn

input_type = raw

length_cutoff = -1
genome_size = 4652500
seed_coverage = 30

length_cutoff_pr = 12000

pa_daligner_option = -e.70 -l1000 -s1000
ovlp_daligner_option = -e.96 -l500 -s1000 -h60
pa_HPCdaligner_option =  -v -B4 -M64
ovlp_HPCdaligner_option = -v -B4 -M64

pa_DBsplit_option = -x500 -s50
ovlp_DBsplit_option = -x500 -s50

falcon_sense_option = --output_multi --min_idt 0.70 --min_cov 4 --max_n_read 200 --n_core 6
overlap_filtering_setting = --max_diff 100 --max_cov 100 --min_cov 20 --bestn 10 --n_core 24 

[job.defaults]
NPROC = 5
njobs = 10
job_type = local
pwatcher_type = blocking
job_type = string
submit = bash -C ${CMD} >| ${STDOUT_FILE} 2>| ${STDERR_FILE}

[job.step.da]
NPROC=5
[job.step.pda]
NPROC=5
[job.step.la]
NPROC=5
[job.step.pla]
NPROC=5
[job.step.cns]
NPROC=5
[job.step.asm]
NPROC=5

output errors

.......
[INFO]recently_satisfied:
set([Node(0-rawreads/tan-split)])
[INFO]Num satisfied in this iteration: 1
[INFO]Num still unsatisfied: 0
[INFO]Num unsatisfied: 11, graph: 13
[INFO]About to submit: Node(0-rawreads/tan-chunks/tan_002)
[INFO]About to submit: Node(0-rawreads/tan-chunks/tan_003)
[INFO]About to submit: Node(0-rawreads/tan-chunks/tan_004)
[INFO]About to submit: Node(0-rawreads/tan-chunks/tan_000)
[INFO]About to submit: Node(0-rawreads/tan-chunks/tan_001)
[INFO]Popen: '/bin/bash -C /home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/pwatcher/mains/job_start.sh >| /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-chunks/tan_003/run-P29dfb2be63473f.bash.stdout 2>| /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-chunks/tan_003/run-P29dfb2be63473f.bash.stderr'
[INFO]Popen: '/bin/bash -C /home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/pwatcher/mains/job_start.sh >| /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-chunks/tan_002/run-P69669f69a8f11c.bash.stdout 2>| /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-chunks/tan_002/run-P69669f69a8f11c.bash.stderr'
[INFO]Popen: '/bin/bash -C /home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/pwatcher/mains/job_start.sh >| /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-chunks/tan_000/run-P0cf8f616737076.bash.stdout 2>| /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-chunks/tan_000/run-P0cf8f616737076.bash.stderr'
[INFO]Popen: '/bin/bash -C /home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/pwatcher/mains/job_start.sh >| /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-chunks/tan_001/run-P34b4ed909851e7.bash.stdout 2>| /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-chunks/tan_001/run-P34b4ed909851e7.bash.stderr'
[INFO]Popen: '/bin/bash -C /home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/pwatcher/mains/job_start.sh >| /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-chunks/tan_004/run-Pc28de0efbd036f.bash.stdout 2>| /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-chunks/tan_004/run-Pc28de0efbd036f.bash.stderr'
[INFO](slept for another 0.0s -- another 1 loop iterations)
[INFO](slept for another 0.3s -- another 2 loop iterations)
[INFO]recently_satisfied:
set([Node(0-rawreads/tan-chunks/tan_000),
     Node(0-rawreads/tan-chunks/tan_004),
     Node(0-rawreads/tan-chunks/tan_001),
     Node(0-rawreads/tan-chunks/tan_003),
     Node(0-rawreads/tan-chunks/tan_002)])
[INFO]Num satisfied in this iteration: 5
[INFO]Num still unsatisfied: 6
[INFO]About to submit: Node(0-rawreads/tan-runs/tan_004)
[INFO]About to submit: Node(0-rawreads/tan-runs/tan_003)
[INFO]About to submit: Node(0-rawreads/tan-runs/tan_002)
[INFO]About to submit: Node(0-rawreads/tan-runs/tan_001)
[INFO]About to submit: Node(0-rawreads/tan-runs/tan_000)
[INFO]Popen: 'bash -C /home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/pwatcher/mains/job_start.sh >| /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_004/run-Pee9de7ecdde422.bash.stdout 2>| /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_004/run-Pee9de7ecdde422.bash.stderr'
[INFO]Popen: 'bash -C /home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/pwatcher/mains/job_start.sh >| /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_001/run-Pecc00fef80843f.bash.stdout 2>| /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_001/run-Pecc00fef80843f.bash.stderr'
[INFO]Popen: 'bash -C /home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/pwatcher/mains/job_start.sh >| /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_000/run-P0d39a10a4ba714.bash.stdout 2>| /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_000/run-P0d39a10a4ba714.bash.stderr'
[INFO]Popen: 'bash -C /home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/pwatcher/mains/job_start.sh >| /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_002/run-P1a9a741334e194.bash.stdout 2>| /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_002/run-P1a9a741334e194.bash.stderr'
[INFO]Popen: 'bash -C /home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/pwatcher/mains/job_start.sh >| /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_003/run-Pb1c29ff2ff1bd2.bash.stdout 2>| /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_003/run-Pb1c29ff2ff1bd2.bash.stderr'
[INFO](slept for another 0.4s -- another 3 loop iterations)
[INFO](slept for another 1.4s -- another 4 loop iterations)
[INFO](slept for another 4.0s -- another 5 loop iterations)
[ERROR]Task Node(0-rawreads/tan-runs/tan_004) failed with exit-code=1
[ERROR]Some tasks are recently_done but not satisfied: set([Node(0-rawreads/tan-runs/tan_004)])
[ERROR]ready: set([])
    submitted: set([Node(0-rawreads/tan-runs/tan_002), Node(0-rawreads/tan-runs/tan_000), Node(0-rawreads/tan-runs/tan_001), Node(0-rawreads/tan-runs/tan_003)])
[ERROR]Noop. We cannot kill blocked threads. Hopefully, everything will die on SIGTERM.
Traceback (most recent call last):
  File "/home/zhangyanju/software/falcon20180808/fc_env_180826/bin/fc_run", line 11, in <module>
    sys.exit(main())
  File "/home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/falcon_kit/mains/run1.py", line 724, in main
    main1(argv[0], args.config, args.logger)
  File "/home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/falcon_kit/mains/run1.py", line 76, in main1
    input_fofn_fn=input_fofn_fn,
  File "/home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/falcon_kit/mains/run1.py", line 242, in run
    dist=Dist(NPROC=4, MB=4000, job_dict=config['job.step.da']),
  File "/home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/falcon_kit/pype.py", line 192, in gen_parallel_tasks
    wf.refreshTargets()
  File "/home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 277, in refreshTargets
    self._refreshTargets(updateFreq, exitOnFailure)
  File "/home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 361, in _refreshTargets
    raise Exception(msg)
Exception: Some tasks are recently_done but not satisfied: set([Node(0-rawreads/tan-runs/tan_004)])

Error file:0-rawreads/tan-runs/tan_004/run-Pee9de7ecdde422.bash.stderr 

executable=${PYPEFLOW_JOB_START_SCRIPT}
+ executable=/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_004/run-Pee9de7ecdde422.bash
timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default
+ timeout=60

# Wait up to timeout seconds for the executable to become "executable",
# then exec.
#timeleft = int(timeout)
while [[ ! -x "${executable}" ]]; do
    if [[ "${timeout}" == "0" ]]; then
        echo "timed out waiting for (${executable})"
        exit 77
    fi
    echo "not executable: '${executable}', waiting ${timeout}s"
    sleep 1
    timeout=$((timeout-1))
done
+ [[ ! -x /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_004/run-Pee9de7ecdde422.bash ]]

/bin/bash ${executable}
+ /bin/bash /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_004/run-Pee9de7ecdde422.bash
+ '[' '!' -d /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_004 ']'
+ cd /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_004
+ eval '/bin/bash run.sh'
++ /bin/bash run.sh
export PATH=$PATH:/bin
+ export PATH=/home/zhangyanju/software/falcon20180808/fc_env_180826/bin:/home/zhangyanju/miniconda2/bin:/home/zhangyanju
/software/falcon20180808/fc_env_180826/bin:/home/zhangyanju/miniconda2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/
bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/us
r/lib/jvm/java-8-oracle/jre/bin:/bin
+ PATH=/home/zhangyanju/software/falcon20180808/fc_env_180826/bin:/home/zhangyanju/miniconda2/bin:/home/zhangyanju/softwa
re/falcon20180808/fc_env_180826/bin:/home/zhangyanju/miniconda2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sb
in:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/j
vm/java-8-oracle/jre/bin:/bin
cd /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_004
+ cd /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_004
/bin/bash task.sh
+ /bin/bash task.sh
pypeflow 2.0.4+git.005acb16689c18c09cf552b42911e69629ffeceb
2018-08-29 17:57:40,101 - root - DEBUG - Running "/home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/si
te-packages/pypeflow/do_task.py /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_004/task.json
"
2018-08-29 17:57:40,102 - root - DEBUG - Checking existence of '/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-raw
reads/tan-runs/tan_004/task.json' with timeout=30
2018-08-29 17:57:40,103 - root - DEBUG - Loading JSON from '/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawread
s/tan-runs/tan_004/task.json'
2018-08-29 17:57:40,103 - root - DEBUG - {u'bash_template_fn': u'template.sh',
 u'inputs': {u'bash_template': u'/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-split/bash_template.s
h',
             u'units_of_work': u'/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-chunks/tan_004/some-u
nits-of-work.json'},
 u'outputs': {u'results': u'some-done-files.json'},
 u'parameters': {u'pypeflow_mb': 4000, u'pypeflow_nproc': u'5'}}
2018-08-29 17:57:40,103 - root - WARNING - CD: '/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/t
an_004' <- '/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_004'
2018-08-29 17:57:40,103 - root - DEBUG - Checking existence of u'/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-ra
wreads/tan-chunks/tan_004/some-units-of-work.json' with timeout=30
2018-08-29 17:57:40,103 - root - DEBUG - Checking existence of u'/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-ra
wreads/tan-split/bash_template.sh' with timeout=30
2018-08-29 17:57:40,103 - root - DEBUG - Checking existence of u'template.sh' with timeout=30
2018-08-29 17:57:40,103 - root - WARNING - CD: '/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/t
an_004' <- '/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_004'
2018-08-29 17:57:40,104 - root - INFO - $('/bin/bash user_script.sh')
hostname
+ hostname
pwd
+ pwd
date
+ date
# Substitution will be similar to snakemake "shell".
python -m falcon_kit.mains.generic_run_units_of_work --nproc=5 --units-of-work-fn=/HDD1/zhangyanju/sss/ecoli/10/falcon201
80808/output/0-rawreads/tan-chunks/tan_004/some-units-of-work.json --bash-template-fn=/HDD1/zhangyanju/sss/ecoli/10/falco
n20180808/output/0-rawreads/tan-split/bash_template.sh --results-fn=some-done-files.json
+ python -m falcon_kit.mains.generic_run_units_of_work --nproc=5 --units-of-work-fn=/HDD1/zhangyanju/sss/ecoli/10/falcon2
0180808/output/0-rawreads/tan-chunks/tan_004/some-units-of-work.json --bash-template-fn=/HDD1/zhangyanju/sss/ecoli/10/fal
con20180808/output/0-rawreads/tan-split/bash_template.sh --results-fn=some-done-files.json
falcon-kit 1.2.2+git.00e8272b663d32a0962ae92ab92324a3b3eb4b46
pypeflow 2.0.4+git.005acb16689c18c09cf552b42911e69629ffeceb
INFO:root:INPUT:{u'db': u'/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/build/raw_reads.db', u'config': 
u'/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/General_config.json', u'script': u'/HDD1/zhangyanju/sss/ecoli/10/fa
lcon20180808/output/0-rawreads/tan-split/./tan-scripts/tan_004/run_datander.sh'}
INFO:root:OUTPUT:{u'job_done': u'job.done'}
INFO:root:PARAMS:{'pypeflow_nproc': '5'}
INFO:root:$('rm -rf uow-00')
WARNING:root:CD: 'uow-00' <- '/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_004'
INFO:root:$('/bin/bash user_script.sh')
hostname
+ hostname
pwd
+ pwd
date
+ date
python -m falcon_kit.mains.dazzler --config=/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/General_config.json --db=
/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/build/raw_reads.db  tan-apply --script=/HDD1/zhangyanju/ss
s/ecoli/10/falcon20180808/output/0-rawreads/tan-split/./tan-scripts/tan_004/run_datander.sh --job-done=job.done
+ python -m falcon_kit.mains.dazzler --config=/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/General_config.json --d
b=/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/build/raw_reads.db tan-apply --script=/HDD1/zhangyanju/s
ss/ecoli/10/falcon20180808/output/0-rawreads/tan-split/./tan-scripts/tan_004/run_datander.sh --job-done=job.done
falcon-kit 1.2.2+git.00e8272b663d32a0962ae92ab92324a3b3eb4b46
pypeflow 2.0.4+git.005acb16689c18c09cf552b42911e69629ffeceb
[INFO]Log-level: INFO
[INFO]ln -sf ../../../build/raw_reads.db raw_reads.db
[INFO]ln -sf ../../../build/.raw_reads.bps .raw_reads.bps
[INFO]ln -sf ../../../build/.raw_reads.dust.anno .raw_reads.dust.anno
[INFO]ln -sf ../../../build/.raw_reads.dust.data .raw_reads.dust.data
[INFO]ln -sf ../../../build/.raw_reads.idx .raw_reads.idx
[INFO]ln -sf ../../../tan-split/tan-scripts/tan_004/run_datander.sh run_datander.sh
[INFO]$('bash -vex run_datander.sh')
datander -v -e0.7 -s1000 -P. raw_reads.17 raw_reads.18 raw_reads.19 raw_reads.20
+ datander -v -e0.7 -s1000 -P. raw_reads.17 raw_reads.18 raw_reads.19 raw_reads.20
LAcheck raw_reads TAN.raw_reads.17.las TAN.raw_reads.18.las TAN.raw_reads.19.las TAN.raw_reads.20.las
+ LAcheck raw_reads TAN.raw_reads.17.las TAN.raw_reads.18.las TAN.raw_reads.19.las TAN.raw_reads.20.las
[WARNING]Call 'bash -vex run_datander.sh' returned 256.
Traceback (most recent call last):
  File "/home/zhangyanju/miniconda2/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/zhangyanju/miniconda2/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", 
line 1357, in <module>
    main()
  File "/home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", 
line 1353, in main
    args.func(args)
  File "/home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", 
line 923, in cmd_tan_apply
    tan_apply(args.db_fn, args.script_fn, args.job_done_fn)
  File "/home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/falcon_kit/mains/dazzler.py", 
line 326, in tan_apply
    io.syscall('bash -vex {}'.format(os.path.basename(script_fn)))
  File "/home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/pypeflow/io.py", line 29, in s
yscall
    raise Exception(msg)
Exception: Call 'bash -vex run_datander.sh' returned 256.
WARNING:root:Call '/bin/bash user_script.sh' returned 256.
WARNING:root:CD: 'uow-00' -> '/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_004'
Traceback (most recent call last):
  File "/home/zhangyanju/miniconda2/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/zhangyanju/miniconda2/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/falcon_kit/mains/generic_run_u
nits_of_work.py", line 115, in <module>
    main()
  File "/home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/falcon_kit/mains/generic_run_u
nits_of_work.py", line 111, in main
    run(**vars(args))
  File "/home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/falcon_kit/mains/generic_run_u
nits_of_work.py", line 64, in run
    pypeflow.do_task.run_bash(script, inputs, outputs, params)
  File "/home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/pypeflow/do_task.py", line 178
, in run_bash
    util.system(cmd)
  File "/home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/pypeflow/io.py", line 29, in s
yscall
    raise Exception(msg)
Exception: Call '/bin/bash user_script.sh' returned 256.
2018-08-29 17:57:52,505 - root - WARNING - Call '/bin/bash user_script.sh' returned 256.
2018-08-29 17:57:52,505 - root - WARNING - CD: '/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/t
an_004' -> '/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_004'
2018-08-29 17:57:52,506 - root - WARNING - CD: '/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/t
an_004' -> '/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_004'
2018-08-29 17:57:52,506 - root - CRITICAL - Error in /home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7
/site-packages/pypeflow/do_task.py with args="{'json_fn': '/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads
/tan-runs/tan_004/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
Traceback (most recent call last):
  File "/home/zhangyanju/miniconda2/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/home/zhangyanju/miniconda2/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/pypeflow/do_task.py", line 246
, in <module>
    main()
  File "/home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/pypeflow/do_task.py", line 238
, in main
    run(**vars(parsed_args))
  File "/home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/pypeflow/do_task.py", line 232
, in run
    run_cfg_in_tmpdir(cfg, tmpdir)
  File "/home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/pypeflow/do_task.py", line 208
, in run_cfg_in_tmpdir
    run_bash(bash_template, myinputs, myoutputs, parameters)
  File "/home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/pypeflow/do_task.py", line 178
, in run_bash
    util.system(cmd)
  File "/home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7/site-packages/pypeflow/io.py", line 29, in s
yscall
    raise Exception(msg)
Exception: Call '/bin/bash user_script.sh' returned 256.
pwd
+++ pwd
++ echo 'FAILURE. Running top in /HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads/tan-runs/tan_004 (If you 
see -terminal database is inaccessible- you are using the python bin-wrapper, so you will not get diagnostic info. No big
 deal. This process is crashing anyway.)'
++ rm -f top.txt
++ which python
++ which top
++ env -u LD_LIBRARY_PATH top -b -n 1
++ env -u LD_LIBRARY_PATH top -b -n 1
++ pstree -apl

real    0m12.821s
user    0m33.820s
sys 0m1.060s
+ finish
+ echo 'finish code: 1'

when I changed the dataset, I still got this error but in different tan-00x folder.

2018-08-29 17:57:52,506 - root - CRITICAL - Error in /home/zhangyanju/software/falcon20180808/fc_env_180826/lib/python2.7
/site-packages/pypeflow/do_task.py with args="{'json_fn': '/HDD1/zhangyanju/sss/ecoli/10/falcon20180808/output/0-rawreads
/tan-runs/tan_004/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
pb-cdunn commented 6 years ago

Please update to the latest tarball.

Then, try running bash -vex user_script.sh yourself in a failing task-directory. Maybe you can debug it.

Looks to me like LAcheck failed for you, which means a problem with DALIGNER. Maybe you are not using the DALIGNER from the tarball?

Always always always always always always always run synth-5k first (from FALCON-examples/run/synth0). It's very quick, so you can use it to debug your own integration problems.