FALCON-integrate in PBSpro 14.xx

[akaraw]

Hi Everyone,

I am really new to this HPC systems and I would like to get an opinion regarding some of the errors I encountered during a FLACON tutorial. We use PBSpro in our HPC cluster and I tried to run Falcon Ecoli tutorial on it but continuously get the following erros. Would any of you can help please?

### pbs script:

#!/bin/bash
#
#PBS -l select=1:ncpus=4:mem=24GB
#PBS -l walltime=10:00:00
#PBS -N ECOLI
#PBS -A UQ-SCI-SCMB

# load dependencies
module load python
module load falcon

#Source build
cd /sw/RCC/FALCON/FALCON-integrate/
source env.sh

# navigate to job directory, directory containing input.fofn
cd /30days/other/

#run it

#export PATH=/30days/FALCON-integrate/fc_env/bin:$PATH
fc_run /30days/other/fc_run_ecoli2.cfg

cfg file:

job_type = PBS

# list of files of the initial bas.h5 files
input_fofn = input.fofn
#input_fofn = preads.fofn

input_type = raw
#input_type = preads

# The length cutoff used for seed reads used for initial mapping
length_cutoff = 12000

# The length cutoff used for seed reads usef for pre-assembly
length_cutoff_pr = 12000

job_queue = batch
sge_option_da = -l select=1:ppn=4:walltime=10:00:00
sge_option_la = -l select=1:ppn=4:walltime=10:00:00
sge_option_pda = -l select=1:ppn=4:walltime=10:00:00
sge_option_pla = -l select=1:ppn=4:walltime=10:00:00
sge_option_fc = -l select=1:ppn=4:walltime=10:00:00
sge_option_cns = -l select=1:ppn=4:walltime=10:00:00

pa_concurrent_jobs = 26
ovlp_concurrent_jobs = 26

pa_HPCdaligner_option =  -v -B128 -t16 -e.70 -l1000 -s1000
ovlp_HPCdaligner_option = -v -B128 -t32 -h60 -e.96 -l500 -s1000

pa_DBsplit_option = -x500 -s200
ovlp_DBsplit_option = -x500 -s200

falcon_sense_option = --output_multi --min_idt 0.70 --min_cov 4 --max_n_read 200 --n_core 6

overlap_filtering_setting = --max_diff 100 --max_cov 100 --min_cov 20 --bestn 10 --n_core 24

Error:

Unloading compiler-dependent module mkl/2018.2
[118450]$('lfs setstripe -c 12 /gpfs1/scratch/30days/uqakaraw/other')
sh: lfs: command not found
[118450]WARNING: Call 'lfs setstripe -c 12 /gpfs1/scratch/30days/uqakaraw/other' returned 32512.
[INFO]Setup logging from file "None".
[INFO]fc_run started with configuration /30days/uqakaraw/other/fc_run_ecoli2.cfg
/gpfs1/sw1/RCC/FALCON/FALCON-integrate/FALCON/falcon_kit/run_support.py:194: UserWarning: Deprecated setting in config: 'pa_concurrent_jobs' -- Prefer da_concurrent_jobs and la_concurrent_jobs separately
  "Deprecated setting in config: 'pa_concurrent_jobs' -- Prefer da_concurrent_jobs and la_concurrent_jobs separately")
/gpfs1/sw1/RCC/FALCON/FALCON-integrate/FALCON/falcon_kit/run_support.py:199: UserWarning: Deprecated setting in config: 'ovlp_concurrent_jobs' -- Prefer pda_concurrent_jobs and pla_concurrent_jobs separately
  "Deprecated setting in config: 'ovlp_concurrent_jobs' -- Prefer pda_concurrent_jobs and pla_concurrent_jobs separately")
[INFO] No target specified, assuming "assembly" as target
/gpfs1/sw1/RCC/FALCON/FALCON-integrate/FALCON/falcon_kit/run_support.py:456: UserWarning: Unexpected keys in input config: set(['pa_concurrent_jobs', 'ovlp_concurrent_jobs'])
  warnings.warn("Unexpected keys in input config: %s" % repr(unused))
[WARNING]In simple_pwatcher_bridge, pwatcher_impl=<module 'pwatcher.fs_based' from '/gpfs1/sw1/RCC/FALCON/FALCON-integrate/pypeFLOW/pwatcher/fs_based.pyc'>
[INFO]In simple_pwatcher_bridge, pwatcher_impl=<module 'pwatcher.fs_based' from '/gpfs1/sw1/RCC/FALCON/FALCON-integrate/pypeFLOW/pwatcher/fs_based.pyc'>
[INFO]job_type='PBS', job_queue='batch', sge_option='-l select=1:ppn=4:walltime=10:00:00', use_tmpdir=False, squash=False, job_name_style=0
[INFO]Num unsatisfied: 1, graph: 1
[INFO]About to submit: Node(0-rawreads/raw-fofn-abs)
[INFO]starting job Job(jobid='P7bac7a888a35d7', cmd='/bin/bash run.sh', rundir='/gpfs1/scratch/30days/uqakaraw/other/0-rawreads/raw-fofn-abs', options={'job_queue': 'batch', 'sge_option': '-l select=1:ppn=4:walltime=10:00:00', 'job_type': 'PBS'})
[INFO]!qsub -N P7bac7a888a35d7 -q batch -l select=1:ppn=4:walltime=10:00:00 -V -o stdout -e stderr -S /bin/bash /gpfs1/scratch/30days/uqakaraw/other/mypwatcher/wrappers/run-P7bac7a888a35d7.bash
qsub: Illegal attribute or resource value Resource_List.select
[ERROR]In pwatcher.fs_based.cmd_run(), failed to submit background-job:
MetaJobPbs(MetaJob(job=Job(jobid='P7bac7a888a35d7', cmd='/bin/bash run.sh', rundir='/gpfs1/scratch/30days/uqakaraw/other/0-rawreads/raw-fofn-abs', options={'job_queue': 'batch', 'sge_option': '-l select=1:ppn=4:walltime=10:00:00', 'job_type': 'PBS'}), lang_exe='/bin/bash'))
Traceback (most recent call last):
  File "/gpfs1/sw1/RCC/FALCON/FALCON-integrate/pypeFLOW/pwatcher/fs_based.py", line 532, in cmd_run
    state.submit_background(bjob)
  File "/gpfs1/sw1/RCC/FALCON/FALCON-integrate/pypeFLOW/pwatcher/fs_based.py", line 117, in submit_background
    bjob.submit(self, exe, script_fn) # Can raise
  File "/gpfs1/sw1/RCC/FALCON/FALCON-integrate/pypeFLOW/pwatcher/fs_based.py", line 346, in submit
    system(sge_cmd, checked=True) # TODO: Capture q-jobid
  File "/gpfs1/sw1/RCC/FALCON/FALCON-integrate/pypeFLOW/pwatcher/fs_based.py", line 551, in system
    raise Exception('{} <- {!r}'.format(rc, call))
Exception: 42496 <- 'qsub -N P7bac7a888a35d7 -q batch -l select=1:ppn=4:walltime=10:00:00 -V -o stdout -e stderr -S /bin/bash /gpfs1/scratch/30days/uqakaraw/other/mypwatcher/wrappers/run-P7bac7a888a35d7.bash'
[ERROR]Failed to enqueue 1 of 1 jobs: set([Node(0-rawreads/raw-fofn-abs)])
[WARNING]Nothing is happening, and we had 0 failures. Should we quit? Instead, we will just sleep.
[INFO]sleep 0.1s
[INFO]Num unsatisfied: 2, graph: 2
[INFO]About to submit: Node(0-rawreads/raw-fofn-abs)
[INFO]starting job Job(jobid='P7bac7a888a35d7', cmd='/bin/bash run.sh', rundir='/gpfs1/scratch/30days/uqakaraw/other/0-rawreads/raw-fofn-abs', options={'job_queue': 'batch', 'sge_option': '-l select=1:ppn=4:walltime=10:00:00', 'job_type': 'PBS'})
[INFO]!qsub -N P7bac7a888a35d7 -q batch -l select=1:ppn=4:walltime=10:00:00 -V -o stdout -e stderr -S /bin/bash /gpfs1/scratch/30days/uqakaraw/other/mypwatcher/wrappers/run-P7bac7a888a35d7.bash
qsub: Illegal attribute or resource value Resource_List.select
[ERROR]In pwatcher.fs_based.cmd_run(), failed to submit background-job:
MetaJobPbs(MetaJob(job=Job(jobid='P7bac7a888a35d7', cmd='/bin/bash run.sh', rundir='/gpfs1/scratch/30days/uqakaraw/other/0-rawreads/raw-fofn-abs', options={'job_queue': 'batch', 'sge_option': '-l select=1:ppn=4:walltime=10:00:00', 'job_type': 'PBS'}), lang_exe='/bin/bash'))
Traceback (most recent call last):
  File "/gpfs1/sw1/RCC/FALCON/FALCON-integrate/pypeFLOW/pwatcher/fs_based.py", line 532, in cmd_run
    state.submit_background(bjob)
  File "/gpfs1/sw1/RCC/FALCON/FALCON-integrate/pypeFLOW/pwatcher/fs_based.py", line 117, in submit_background
    bjob.submit(self, exe, script_fn) # Can raise
  File "/gpfs1/sw1/RCC/FALCON/FALCON-integrate/pypeFLOW/pwatcher/fs_based.py", line 346, in submit
    system(sge_cmd, checked=True) # TODO: Capture q-jobid
  File "/gpfs1/sw1/RCC/FALCON/FALCON-integrate/pypeFLOW/pwatcher/fs_based.py", line 551, in system
    raise Exception('{} <- {!r}'.format(rc, call))
Exception: 42496 <- 'qsub -N P7bac7a888a35d7 -q batch -l select=1:ppn=4:walltime=10:00:00 -V -o stdout -e stderr -S /bin/bash /gpfs1/scratch/30days/uqakaraw/other/mypwatcher/wrappers/run-P7bac7a888a35d7.bash'
[ERROR]Failed to enqueue 1 of 1 jobs: set([Node(0-rawreads/raw-fofn-abs)])
[WARNING]Nothing is happening, and we had 0 failures. Should we quit? Instead, we will just sleep.
[INFO]sleep 0.1s
[INFO]Num unsatisfied: 3, graph: 3
[INFO]About to submit: Node(0-rawreads/raw-fofn-abs)
[INFO]starting job Job(jobid='P7bac7a888a35d7', cmd='/bin/bash run.sh', rundir='/gpfs1/scratch/30days/uqakaraw/other/0-rawreads/raw-fofn-abs', options={'job_queue': 'batch', 'sge_option': '-l select=1:ppn=4:walltime=10:00:00', 'job_type': 'PBS'})
[INFO]!qsub -N P7bac7a888a35d7 -q batch -l select=1:ppn=4:walltime=10:00:00 -V -o stdout -e stderr -S /bin/bash /gpfs1/scratch/30days/uqakaraw/other/mypwatcher/wrappers/run-P7bac7a888a35d7.bash
qsub: Illegal attribute or resource value Resource_List.select
[ERROR]In pwatcher.fs_based.cmd_run(), failed to submit background-job:
MetaJobPbs(MetaJob(job=Job(jobid='P7bac7a888a35d7', cmd='/bin/bash run.sh', rundir='/gpfs1/scratch/30days/uqakaraw/other/0-rawreads/raw-fofn-abs', options={'job_queue': 'batch', 'sge_option': '-l select=1:ppn=4:walltime=10:00:00', 'job_type': 'PBS'}), lang_exe='/bin/bash'))
Traceback (most recent call last):
  File "/gpfs1/sw1/RCC/FALCON/FALCON-integrate/pypeFLOW/pwatcher/fs_based.py", line 532, in cmd_run
    state.submit_background(bjob)
  File "/gpfs1/sw1/RCC/FALCON/FALCON-integrate/pypeFLOW/pwatcher/fs_based.py", line 117, in submit_background
    bjob.submit(self, exe, script_fn) # Can raise
  File "/gpfs1/sw1/RCC/FALCON/FALCON-integrate/pypeFLOW/pwatcher/fs_based.py", line 346, in submit
    system(sge_cmd, checked=True) # TODO: Capture q-jobid
  File "/gpfs1/sw1/RCC/FALCON/FALCON-integrate/pypeFLOW/pwatcher/fs_based.py", line 551, in system
    raise Exception('{} <- {!r}'.format(rc, call))
Exception: 42496 <- 'qsub -N P7bac7a888a35d7 -q batch -l select=1:ppn=4:walltime=10:00:00 -V -o stdout -e stderr -S /bin/bash /gpfs1/scratch/30days/uqakaraw/other/mypwatcher/wrappers/run-P7bac7a888a35d7.bash'
[ERROR]Failed to enqueue 1 of 1 jobs: set([Node(0-rawreads/raw-fofn-abs)])
[WARNING]Nothing is happening, and we had 0 failures. Should we quit? Instead, we will just sleep.
[INFO]sleep 0.1s
[ERROR]Failed to read JSON from '/gpfs1/scratch/30days/uqakaraw/other/0-rawreads/daligner-scatter/scattered.json'
Traceback (most recent call last):
  File "/gpfs1/sw1/RCC/FALCON/FALCON-integrate/FALCON/falcon_kit/mains/run1.py", line 24, in create_daligner_tasks
    content = json.loads(open(scatter_fn).read())  # array of descriptions
IOError: [Errno 2] No such file or directory: '/gpfs1/scratch/30days/uqakaraw/other/0-rawreads/daligner-scatter/scattered.json'
Traceback (most recent call last):
  File "/sw/RCC/FALCON/FALCON-integrate/fc_env/bin/fc_run", line 11, in <module>
    load_entry_point('falcon-kit', 'console_scripts', 'fc_run')()
  File "/gpfs1/sw1/RCC/FALCON/FALCON-integrate/FALCON/falcon_kit/mains/run1.py", line 504, in main
    main1(argv[0], args.config, args.logger)
  File "/gpfs1/sw1/RCC/FALCON/FALCON-integrate/FALCON/falcon_kit/mains/run1.py", line 150, in main1
    input_fofn_plf=input_fofn_plf,
  File "/gpfs1/sw1/RCC/FALCON/FALCON-integrate/FALCON/falcon_kit/mains/run1.py", line 243, in run
    rawread_dir, scattered_plf)
  File "/gpfs1/sw1/RCC/FALCON/FALCON-integrate/FALCON/falcon_kit/mains/run1.py", line 28, in create_daligner_tasks
    raise Exception(msg)
Exception: Failed to read JSON from '/gpfs1/scratch/30days/uqakaraw/other/0-rawreads/daligner-scatter/scattered.json'

[pb-cdunn]

https://github.com/PacificBiosciences/pypeFLOW/wiki/configuration#pbs

It's very difficult for us to support the fs_based process-watcher everywhere. The "blocking" process-watcher is much simpler. Please use

pwatcher_type = blocking
job_type = string
submit = ...  # including -W block=T