search

PacificBiosciences / FALCON

FALCON: experimental PacBio diploid assembler -- Out-of-date -- Please use a binary release: https://github.com/PacificBiosciences/FALCON_unzip/wiki/Binaries
https://github.com/PacificBiosciences/FALCON_unzip/wiki/Binaries
Other
204 stars 103 forks source link

Assembly with example data on slurm failed: {Node(0-rawreads/build)} #707

Closed sadikmu closed 3 years ago

sadikmu commented 4 years ago

Any feedback please?

I have been struggling for a while to setup Falcon (falcon-kit 1.8.1) on slum to get it run example F1_bull_test data before using Falcon configuration for my Pacbio data.

I have got the F1_bull_test data from here and sadly I have not succeeded it getting it successfully assembly the this F1_bull_test data.

Can you please help me?

Here is the detail below:

pb-assembly suite installed in a separate an environment called 'falcon'

conda create -n falcon source activate falcon conda install pb-assembly

Lists of packages within falcon environment 

# packages in environment at /home/miniconda3/envs/falcon:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                      1_llvm    conda-forge
bedtools                  2.29.2               hc088bd4_0    bioconda
blas                      1.0                         mkl  
blasr                     5.3.3                h707fff8_0    bioconda
blasr_libcpp              5.3.3                h707fff8_0    bioconda
bwa                       0.7.17               hed695b0_7    bioconda
bzip2                     1.0.8                h516909a_2    conda-forge
ca-certificates           2020.1.1                      0    anaconda
certifi                   2020.4.5.1               py37_0    anaconda
curl                      7.69.1               hbc83047_0  
decorator                 4.4.2                      py_0    conda-forge
falcon-kit                1.8.1                    pypi_0    pypi
falcon-phase              1.2.0                    pypi_0    pypi
falcon-unzip              1.3.7                    pypi_0    pypi
future                    0.18.2           py37hc8dfbb8_1    conda-forge
hdf5                      1.10.4          nompi_h3c11f04_1106    conda-forge
htslib                    1.9                  ha228f0b_7    bioconda
intel-openmp              2020.0                      166    anaconda
krb5                      1.17.1               h2fd8d38_0    conda-forge
ld_impl_linux-64          2.34                 h53a641e_0    conda-forge
libblas                   3.8.0                    15_mkl    conda-forge
libcblas                  3.8.0                    15_mkl    conda-forge
libcurl                   7.69.1               h20c2e04_0  
libdeflate                1.0                  h14c3975_1    bioconda
libedit                   3.1.20181209         hc058e9b_0  
libffi                    3.2.1             he1b5a44_1007    conda-forge
libgcc-ng                 9.2.0                h24d8f2e_2    conda-forge
libgfortran-ng            7.3.0                hdf63c60_5    conda-forge
libgomp                   9.2.0                h24d8f2e_2    conda-forge
liblapack                 3.8.0                    15_mkl    conda-forge
libopenblas               0.3.9                h5ec1e0e_0    conda-forge
libssh2                   1.9.0                h1ba5d50_1  
libstdcxx-ng              9.2.0                hdf63c60_2    conda-forge
llvm-openmp               10.0.0               hc9558a2_0    conda-forge
minimap2                  2.17                 h8b12597_1    bioconda
mkl                       2020.0                      166    conda-forge
mkl-service               2.3.0            py37he904b0f_0  
mkl_fft                   1.0.15           py37ha843d7b_0  
mkl_random                1.1.0            py37hd6b4f25_0  
mummer4                   4.0.0beta2      pl526he1b5a44_5    bioconda
ncurses                   6.1               hf484d3e_1002    conda-forge
networkx                  2.4                        py_1    conda-forge
nim-falcon                1.1.0                h2f8a211_0    bioconda
numpy                     1.18.1           py37h8960a57_1    conda-forge
numpy-base                1.18.1           py37hde5b4d6_1  
openssl                   1.1.1                h7b6447c_0    anaconda
pb-assembly               0.0.8                         0    bioconda
pb-dazzler                0.0.1                h516909a_0    bioconda
pb-falcon                 2.2.4            py37hed50d52_0    bioconda
pb-falcon-phase           0.1.0                h3889886_0    bioconda
pbbam                     0.23.0               h88345a9_0    bioconda
pbgcpp                    1.9.0                         0    bioconda
pbmm2                     1.2.1                h56fc30b_0    bioconda
pcre                      8.44                 he1b5a44_0    conda-forge
perl                      5.26.2            h516909a_1006    conda-forge
pip                       20.0.2                     py_2    conda-forge
pysam                     0.15.3           py37hda2845c_1    bioconda
python                    3.7.6                h0371630_2  
python-edlib              1.3.8.post1      py37hc9558a2_0    bioconda
python-intervaltree       3.0.2                      py_0    bioconda
python-msgpack            0.6.1            py37hc9558a2_1    bioconda
python-sortedcontainers   2.1.0                      py_0    bioconda
python_abi                3.7                     1_cp37m    conda-forge
racon                     1.4.12               he513fc3_0    bioconda
readline                  7.0                  h7b6447c_5  
samtools                  1.9                 h10a08f8_12    bioconda
setuptools                46.1.3           py37hc8dfbb8_0    conda-forge
six                       1.14.0                   py37_0  
sqlite                    3.31.1               h7b6447c_0  
tk                        8.6.10               hed695b0_0    conda-forge
wheel                     0.34.2                     py_1    conda-forge
xz                        5.2.5                h516909a_0    conda-forge
zlib                      1.2.11            h516909a_1006    conda-forge

Contents of fc_run.cfg

[job.defaults]
JOB_QUEUE = bigmem
MB = 30000
NPROC = 6
job_type=slurm

submit = srun --wait=0 -p myqueue  \
    -J ${JOB_NAME}           \
    -o ${JOB_STDOUT}         \
    -e ${JOB_STDERR}         \
    --mem-per-cpu=${MB}M     \
    --cpus-per-task=${NPROC} \
    ${JOB_SCRIPT}
    "${CMD}"

[General]
input_type = raw
#input_type = preads

input_fofn = subreads.fasta.fofn

# The length cutoff used for seed reads used for initial mapping
genome_size=20000000
seed_coverage = 40
length_cutoff = -1

# The length cutoff for preads
length_cutoff_pr = 12000

falcon_greedy = False
falcon_sense_greedy=False

# overlapping options for Daligner
pa_daligner_option = -e0.76 -l1200 -k18 -h480 -w8 -s100
ovlp_daligner_option = -k24 -h480 -e.95 -l1800 -s100
pa_HPCdaligner_option = -v -B128 -M24
ovlp_HPCdaligner_option = -v -B128 -M24

pa_HPCTANmask_option = -k18 -h480 -w8 -e.8 -s100
pa_HPCREPmask_option = -k18 -h480 -w8 -e.8 -s100
#pa_REPmask_code=1,20;10,15;50,10

pa_DBsplit_option = -x500 -s200
ovlp_DBsplit_option = -s400

# error correction consensus option
falcon_sense_option = --output_multi --min_idt 0.70 --min_cov 4 --max_n_read 200 --n_core 24

# overlap filtering options
overlap_filtering_setting = --max_diff 120 --max_cov 120 --min_cov 4 --n_core 24

[job.step.da]
NPROC=4
MB=32000
njobs=300
[job.step.la]
NPROC=8
MB=64000
njobs=200
[job.step.cns]
NPROC=8
MB=64000
njobs=200
[job.step.pda]
NPROC=8
MB=64000
njobs=200
[job.step.pla]
NPROC=4
MB=32000
njobs=300
[job.step.asm]
NPROC=24
MB=192000
njobs=1

contents of fc.246025.err

`"falcon_sense_option=--output_multi --min_idt 0.70 --min_cov 4 --max_n_read 200 --n_core 24". Those should be "-", as in
 "falcon_sense_option=--output-multi --min-idt 0.70 --min-cov 4 --max-n-read 200 --n-core 24". Auto-replaced.
[WARNING]Option contains flags with "_":
 "overlap_filtering_setting=--max_diff 120 --max_cov 120 --min_cov 4 --n_core 24". Those should be "-", as in
 "overlap_filtering_setting=--max-diff 120 --max-cov 120 --min-cov 4 --n-core 24". Auto-replaced.
[WARNING]Unexpected keys in input config: {'falcon_greedy'}
[WARNING]Please supply a default for "njobs" (aka concurrency) in section [job.defaults]. For now, we will use 8
[INFO]cfg=
{
  "General": {
    "LA4Falcon_preload": false,
    "avoid_text_file_busy": true,
    "bestn": 12,
    "dazcon": false,
    "falcon_greedy": "False",
    "falcon_sense_greedy": false,
    "falcon_sense_option": "--output-multi --min-idt 0.70 --min-cov 4 --max-n-read 200 --n-core 24",
    "falcon_sense_skip_contained": false,
    "fc_ovlp_to_graph_option": " --min-len 12000",
    "genome_size": "20000000",
    "input_fofn": "subreads.fasta.fofn",
    "input_type": "raw",
    "length_cutoff": "-1",
    "length_cutoff_pr": "12000",
    "overlap_filtering_setting": "--max-diff 120 --max-cov 120 --min-cov 4 --n-core 24",
    "ovlp_DBdust_option": "",
    "ovlp_DBsplit_option": "-s400",
    "ovlp_HPCdaligner_option": "-v -B128 -M24",
    "ovlp_daligner_option": "-k24 -h480 -e.95 -l1800 -s100",
    "pa_DBdust_option": "",
    "pa_DBsplit_option": "-x500 -s200",
    "pa_HPCREPmask_option": "-k18 -h480 -w8 -e.8 -s100",
    "pa_HPCTANmask_option": "-k18 -h480 -w8 -e.8 -s100",
    "pa_HPCdaligner_option": "-v -B128 -M24",
    "pa_REPmask_code": "0,300/0,300/0,300",
    "pa_daligner_option": "-e0.76 -l1200 -k18 -h480 -w8 -s100",
    "pa_dazcon_option": "-j 4 -x -l 500",
    "pa_fasta_filter_option": "streamed-internal-median",
    "pa_subsample_coverage": 0,
    "pa_subsample_random_seed": 12345,
    "pa_subsample_strategy": "random",
    "seed_coverage": "40",
    "skip_checks": false,
    "target": "assembly"
  },
  "job.defaults": {
    "JOB_QUEUE": "bigmem",
    "MB": "30000",
    "NPROC": "6",
    "job_type": "slurm",
    "njobs": 8,
    "pwatcher_type": "fs_based",
    "submit": "srun --wait=0 -p myqueue  \\\n-J ${JOB_NAME}           \\\n-o ${JOB_STDOUT}         \\\n-e ${JOB_STDERR}         \\\n--mem
-per-cpu=${MB}M     \\\n--cpus-per-task=${NPROC} \\\n${JOB_SCRIPT}\n\"${CMD}\"",
    "use_tmpdir": false
  },
  "job.step.asm": {
    "MB": "192000",
    "NPROC": "24",
    "njobs": "1"
  },
  "job.step.cns": {
    "MB": "64000",
    "NPROC": "8",
    "njobs": "200"
  },
},
  "job.step.da": {
    "MB": "32000",
    "NPROC": "4",
    "njobs": "300"
  },
  "job.step.dust": {},
  "job.step.la": {
    "MB": "64000",
    "NPROC": "8",
    "njobs": "200"
  },
  "job.step.pda": {
    "MB": "64000",
    "NPROC": "8",
    "njobs": "200"
  },
  "job.step.pla": {
    "MB": "32000",
    "NPROC": "4",
    "njobs": "300"
  }
}

[INFO]In simple_pwatcher_bridge, pwatcher_impl=<module 'pwatcher.fs_based' from '/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pwatcher/fs_based.py'>
[INFO]job_type='slurm', (default)job_defaults={'JOB_QUEUE': 'bigmem', 'MB': '30000', 'NPROC': '6', 'job_type': 'slurm', 'submit': 'srun --wait=0 -p myqueue  \\\n-J ${JOB_NAME}           \\\n-o ${JOB_STDOUT}         \\\n-e ${JOB_STDERR}         \\\n--mem-per-cpu=${MB}M     \\\n--cpus-per-task=${NPROC} \\\n${JOB_SCRIPT}\n"${CMD}"', 'pwatcher_type': 'fs_based', 'njobs': 8, 'use_tmpdir': False}, use_tmpdir=False, squash=False, job_name_style=0
[INFO]Setting max_jobs to 8; was None
[INFO]Num unsatisfied: 2, graph: 2
[INFO]About to submit: Node(0-rawreads/build)
[INFO] starting job Job(jobid='Pac163ff68e5c9f', cmd='/bin/bash run.sh', rundir='/home/data/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/0-rawreads/build', options={'JOB_QUEUE': 'bigmem', 'MB': 4000, 'NPROC': 1, 'job_type': 'slurm', 'submit': 'srun --wait=0 -p myqueue  \\\n-J ${JOB_NAME}           \\\n-o ${JOB_STDOUT}         \\\n-e ${JOB_STDERR}         \\\n--mem-per-cpu=${MB}M     \\\n--cpus-per-task=${NPROC} \\\n${JOB_SCRIPT}\n"${CMD}"', 'pwatcher_type': 'fs_based', 'njobs': 8, 'use_tmpdir': False}) w/ job_type=SLURM
[INFO]$ srun --wait=0 -p myqueue  \
-J Pac163ff68e5c9f           \
-o stdout         \
-e stderr         \
--mem-per-cpu=4000M     \
--cpus-per-task=1 \
/home/data/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/mypwatcher/wrappers/run-Pac163ff68e5c9f.bash
"/home/data/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/mypwatcher/wrappers/run-Pac163ff68e5c9f.bash" >
[ERROR]In pwatcher.fs_based.cmd_run(), failed to submit background-job:
MetaJobSlurm(MetaJob(job=Job(jobid='Pac163ff68e5c9f', cmd='/bin/bash run.sh', rundir='/home/data/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/0-rawreads/build', options={'JOB_QUEUE': 'bigmem', 'MB': 4000, 'NPROC': 1, 'job_type': 'slurm', 'submit': 'srun --wait=0 -p myqueue  \\\n-J ${JOB_NAME}           \\\n-o ${JOB_STDOUT}         \\\n-e ${JOB_STDERR}         \\\n--mem-per-cpu=${MB}M     \\\n--cpus-per-task=${NPROC} \\\n${JOB_SCRIPT}\n"${CMD}"', 'pwatcher_type': 'fs_based', 'njobs': 8, 'use_tmpdir': False}), lang_exe='/bin/bash'))
Traceback (most recent call last):
  File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pwatcher/fs_based.py", line 474, in cmd_run
    state.submit_background(bjob)
  File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pwatcher/fs_based.py", line 121, in submit_background
    bjob.submit(self, exe, script_fn) # Can raise
  File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pwatcher/fs_based.py", line 325, in submit
    self.submit_capture = capture(sge_cmd)
  File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pypeflow/io.py", line 50, in capture
    raise Exception(msg)
Exception: 126 <- 'srun --wait=0 -p myqueue  \\\n-J Pac163ff68e5c9f           \\\n-o stdout         \\\n-e stderr         \\\n--mem-per-cpu=4000M     \\\n--cpus-per-task=1 \\\n/home/data/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/mypwatcher/wrappers/run-Pac163ff68e5c9f.bash\n"/home/data/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/mypwatcher/wrappers/run-Pac163ff68e5c9f.bash"':
srun: error: cnode84: tasks 0-1: Exited with exit code 13
srun: Terminating job step 246025.0
/bin/sh: line 7: /home/data/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/mypwatcher/wrappers/run-Pac163ff68e5c9f.bash: Permission denied

[ERROR]Failed to enqueue 1 of 1 jobs: {Node(0-rawreads/build)}
[INFO]CALL:
 scancel -n Pac163ff68e5c9f
Traceback (most recent call last):
  File "/home/miniconda3/envs/falcon/bin/fc_run", line 11, in <module>
    load_entry_point('falcon-kit==1.8.1', 'console_scripts', 'fc_run')()
  File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/falcon_kit/mains/run1.py", line 706, in main
    main1(argv[0], args.config, args.logger)
  File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/falcon_kit/mains/run1.py", line 73, in main1
    input_fofn_fn=input_fofn_fn,
  File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/falcon_kit/mains/run1.py", line 235, in run
    dist=Dist(NPROC=4, MB=4000, job_dict=config['job.step.da']),
  File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/falcon_kit/pype.py", line 106, in gen_parallel_tasks
    wf.refreshTargets()
  File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 278, in refreshTargets
    self._refreshTargets(updateFreq, exitOnFailure)
  File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 323, in _refreshTargets
    raise Exception(msg)
Exception: Failed to enqueue 1 of 1 jobs: {Node(0-rawreads/build)}

`

Thank you,

sadikmu commented 4 years ago

Thanks Bas Tolhuis from from PacBio techsupport @PacificBiosciences

I got feedback to check 0-rawreads/build for stderr file and the following error message noticed

2020-04-13 11:31:50,050 - root - ERROR - Was waiting for '../../subreads.fasta.fofn'

Checking my files again I found that subreads.fasta.fofn file was missing.

After creating the subreads.fasta.fofn file containing absolute path to the input tutorial F1_bull_test.subreads.fasta.gz file job now submitted to slurm. Progress will be updated.

sadikmu commented 4 years ago

resubmitted job failed again.

I couldn’t see stderr file in … 0-rawreads/build path.

List of files in in …0-rawreads/build

lrwxrwxrwx 1 sadikm sadikm 108 Apr 20 23:27 pwatcher.dir -> /home/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/mypwatcher/jobs/Pac163ff68e5c9f
-rw-rw-r-- 1 sadikm sadikm 269 Apr 20 23:27 run.sh
-rw-rw-r-- 1 sadikm sadikm 610 Apr 20 23:27 task.sh
-rw-rw-r-- 1 sadikm sadikm 368 Apr 20 23:27 task.json
-rw-rw-r-- 1 sadik sadikm 296 Apr 20 23:27 template.sh

contents of fc.2467347.err

falcon-kit 1.8.1 (pip thinks "falcon-kit 1.8.1")
pypeflow 2.3.0
[INFO]Setup logging from file "None".
[INFO]$ lfs setstripe -c 12 /home/toolsTest.data/FAlCON-Unzip/F1_bull_test_data >
[INFO]Apparently '/home/toolsTest.data/FAlCON-Unzip/F1_bull_test_data' is not in lustre filesystem, which is fine.
[INFO]fc_run started with configuration fc_run.cfg
[WARNING]Option contains flags with "_":
 "falcon_sense_option=--output_multi --min_idt 0.70 --min_cov 4 --max_n_read 200 --n_core 24". Those should be "-", as in
 "falcon_sense_option=--output-multi --min-idt 0.70 --min-cov 4 --max-n-read 200 --n-core 24". Auto-replaced.
[WARNING]Option contains flags with "_":
 "overlap_filtering_setting=--max_diff 120 --max_cov 120 --min_cov 4 --n_core 24". Those should be "-", as in
 "overlap_filtering_setting=--max-diff 120 --max-cov 120 --min-cov 4 --n-core 24". Auto-replaced.
[WARNING]Unexpected keys in input config: {'falcon_greedy'}
[WARNING]Please supply a default for "njobs" (aka concurrency) in section [job.defaults]. For now, we will use 8
[INFO]cfg=
{
  "General": {
    "LA4Falcon_preload": false,
    "avoid_text_file_busy": true,
    "bestn": 12,
    "dazcon": false,
    "falcon_greedy": "False",
    "falcon_sense_greedy": false,
    "falcon_sense_option": "--output-multi --min-idt 0.70 --min-cov 4 --max-n-read 200 --n-core 24",
    "falcon_sense_skip_contained": false,
    "fc_ovlp_to_graph_option": " --min-len 12000",
    "genome_size": "20000000",
    "input_fofn": "subreads.fasta.fofn",
    "input_type": "raw",
    "length_cutoff": "-1",
    "length_cutoff_pr": "12000",
    "overlap_filtering_setting": "--max-diff 120 --max-cov 120 --min-cov 4 --n-core 24",
    "ovlp_DBdust_option": "",
    "ovlp_DBsplit_option": "-s400",
    "ovlp_HPCdaligner_option": "-v -B128 -M24",
    "ovlp_daligner_option": "-k24 -h480 -e.95 -l1800 -s100",
    "pa_DBdust_option": "",
    "pa_DBsplit_option": "-x500 -s200",
    "pa_HPCREPmask_option": "-k18 -h480 -w8 -e.8 -s100",
    "pa_HPCTANmask_option": "-k18 -h480 -w8 -e.8 -s100",
    "pa_HPCdaligner_option": "-v -B128 -M24",
    "pa_REPmask_code": "0,300/0,300/0,300",
    "pa_daligner_option": "-e0.76 -l1200 -k18 -h480 -w8 -s100",
    "pa_dazcon_option": "-j 4 -x -l 500",
    "pa_fasta_filter_option": "streamed-internal-median",
    "pa_subsample_coverage": 0,
    "pa_subsample_random_seed": 12345,
    "pa_subsample_strategy": "random",
    "seed_coverage": "40",
    "skip_checks": false,
    "target": "assembly"
  },
  "job.defaults": {
    "JOB_QUEUE": "bigmem",
    "MB": "30000",
    "NPROC": "6",
    "job_type": "slurm",
    "njobs": 8,
    "pwatcher_type": "fs_based",
    "submit": "srun --wait=0 -p myqueue  \\\n-J ${JOB_NAME}           \\\n-o ${JOB_STDOUT}         \\\n-e ${JOB_STDERR}         \\\n--mem-per-cpu=${MB}M     \\\n--cpus-per-task=${NPROC} \\\n${JOB_SCRIPT}\n\"${CMD}\"",
    "use_tmpdir": false
  },
  "job.step.asm": {
    "MB": "192000",
    "NPROC": "24",
    "njobs": "1"
  },
  "job.step.cns": {
    "MB": "64000",
    "NPROC": "8",
    "njobs": "200"
  },
  "job.step.da": {
    "MB": "32000",
    "NPROC": "4",
    "njobs": "300"
  },
  "job.step.dust": {},
  "job.step.la": {
    "MB": "64000",
    "NPROC": "8",
    "njobs": "200"
  },
  "job.step.pda": {
    "MB": "64000",
    "NPROC": "8",
    "njobs": "200"
  },
  "job.step.pla": {
    "MB": "32000",
    "NPROC": "4",
    "njobs": "300"
  }
}
[INFO]In simple_pwatcher_bridge, pwatcher_impl=<module 'pwatcher.fs_based' from '/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pwatcher/fs_based.py'>
[INFO]job_type='slurm', (default)job_defaults={'JOB_QUEUE': 'bigmem', 'MB': '30000', 'NPROC': '6', 'job_type': 'slurm', 'submit': 'srun --wait=0 -p myqueue  \\\n-J ${JOB_NAME}           \\\n-o ${JOB_STDOUT}         \\\n-e ${JOB_STDERR}         \\\n--mem-per-cpu=${MB}M     \\\n--cpus-per-task=${NPROC} \\\n${JOB_SCRIPT}\n"${CMD}"', 'pwatcher_type': 'fs_based', 'njobs': 8, 'use_tmpdir': False}, use_tmpdir=False, squash=False, job_name_style=0
[INFO]Setting max_jobs to 8; was None
[INFO]Num unsatisfied: 2, graph: 2
[INFO]About to submit: Node(0-rawreads/build)
[INFO] starting job Job(jobid='Pac163ff68e5c9f', cmd='/bin/bash run.sh', rundir='/home/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/0-rawreads/build', options={'JOB_QUEUE': 'bigmem', 'MB': 4000, 'NPROC': 1, 'job_type': 'slurm', 'submit': 'srun --wait=0 -p myqueue  \\\n-J ${JOB_NAME}           \\\n-o ${JOB_STDOUT}         \\\n-e ${JOB_STDERR}         \\\n--mem-per-cpu=${MB}M     \\\n--cpus-per-task=${NPROC} \\\n${JOB_SCRIPT}\n"${CMD}"', 'pwatcher_type': 'fs_based', 'njobs': 8, 'use_tmpdir': False}) w/ job_type=SLURM
[INFO]$ srun --wait=0 -p myqueue  \
-J Pac163ff68e5c9f           \
-o stdout         \
-e stderr         \
--mem-per-cpu=4000M     \
--cpus-per-task=1 \
/home/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/mypwatcher/wrappers/run-Pac163ff68e5c9f.bash
"/home/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/mypwatcher/wrappers/run-Pac163ff68e5c9f.bash" >
[ERROR]In pwatcher.fs_based.cmd_run(), failed to submit background-job:
MetaJobSlurm(MetaJob(job=Job(jobid='Pac163ff68e5c9f', cmd='/bin/bash run.sh', rundir='/home/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/0-rawreads/build', options={'JOB_QUEUE': 'bigmem', 'MB': 4000, 'NPROC': 1, 'job_type': 'slurm', 'submit': 'srun --wait=0 -p myqueue  \\\n-J ${JOB_NAME}           \\\n-o ${JOB_STDOUT}         \\\n-e ${JOB_STDERR}         \\\n--mem-per-cpu=${MB}M     \\\n--cpus-per-task=${NPROC} \\\n${JOB_SCRIPT}\n"${CMD}"', 'pwatcher_type': 'fs_based', 'njobs': 8, 'use_tmpdir': False}), lang_exe='/bin/bash'))
Traceback (most recent call last):
  File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pwatcher/fs_based.py", line 474, in cmd_run
    state.submit_background(bjob)
  File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pwatcher/fs_based.py", line 121, in submit_background
    bjob.submit(self, exe, script_fn) # Can raise
  File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pwatcher/fs_based.py", line 325, in submit
    self.submit_capture = capture(sge_cmd)
  File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pypeflow/io.py", line 50, in capture
    raise Exception(msg)
Exception: 126 <- 'srun --wait=0 -p myqueue  \\\n-J Pac163ff68e5c9f           \\\n-o stdout         \\\n-e stderr         \\\n--mem-per-cpu=4000M     \\\n--cpus-per-task=1 \\\n/home/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/mypwatcher/wrappers/run-Pac163ff68e5c9f.bash\n"/home/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/mypwatcher/wrappers/run-Pac163ff68e5c9f.bash"':
srun: error: cnode10: tasks 0-1: Exited with exit code 13
srun: Terminating job step 247347.0
/bin/sh: line 7: /home/toolsTest.data/FAlCON-Unzip/F1_bull_test_data/mypwatcher/wrappers/run-Pac163ff68e5c9f.bash: Permission denied

[ERROR]Failed to enqueue 1 of 1 jobs: {Node(0-rawreads/build)}
[INFO]CALL:
 scancel -n Pac163ff68e5c9f
Traceback (most recent call last):
  File "/home/miniconda3/envs/falcon/bin/fc_run", line 11, in <module>
    load_entry_point('falcon-kit==1.8.1', 'console_scripts', 'fc_run')()
  File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/falcon_kit/mains/run1.py", line 706, in main
    main1(argv[0], args.config, args.logger)
  File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/falcon_kit/mains/run1.py", line 73, in main1
    input_fofn_fn=input_fofn_fn,
  File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/falcon_kit/mains/run1.py", line 235, in run
    dist=Dist(NPROC=4, MB=4000, job_dict=config['job.step.da']),
  File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/falcon_kit/pype.py", line 106, in gen_parallel_tasks
    wf.refreshTargets()
  File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 278, in refreshTargets
    self._refreshTargets(updateFreq, exitOnFailure)
  File "/home/miniconda3/envs/falcon/lib/python3.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 323, in _refreshTargets
    raise Exception(msg)
Exception: Failed to enqueue 1 of 1 jobs: {Node(0-rawreads/build)}
shannonekj commented 4 years ago

Hi @sadikmu this appears to be a problem I'm running into as well. I believe it is due to the permissions of the .bash scripts that are created in the mypwatcher/wrappers/ directory.

If you change the permissions of the file to be executable and then submit your run script again it will work for that particular step (pertaining to the bash script you just made into an executable). However, it seems we'll have to do this for each individual step & I'd like it to just run. Do any of the PacBio folks have any suggestions/workarounds?

sadikmu commented 4 years ago

hi @shannonekj, It would be unrealistic to rerun on slurm queue every time it fails modifying in the file permission to executable which should be autogenerated executable the job is submitted.

The F1_bull_test data (~462Mb) and the fc_run.cfg script here worked successfully on local server. However, running falcon on actual data on the local sever after modifying genome size in fc_run.cfg used above just failed.

I tried to used various parameter options within fc_run.cfg including parameters in #708 -unfortunately it did not work at all and throws the same error shown below. I have been struggling on this for a while and could not figure it out.

Unable to move forward on this and highly appreciate anyones suggestion. 

[INFO]Setup logging from file "None".
[INFO]$ lfs setstripe -c 12 /home/assembly/falcon >
[INFO]Apparently '/home/assembly/falcon' is not in lustre filesystem, which is fine.
[INFO]fc_run started with configuration fc_run.cfg
[WARNING]Unexpected keys in input config: set(['ovlp_concurrent_jobs', 'falcon_greedy'])
[INFO]cfg=
{
  "General": {
    "LA4Falcon_preload": false,
    "avoid_text_file_busy": true,
    "bestn": 12,
    "dazcon": false,
    "falcon_greedy": "False",
    "falcon_sense_greedy": false,
    "falcon_sense_option": "--output-multi --min-idt 0.70 --min-cov 4 --max-n-read 200 --n-core 24",
    "falcon_sense_skip_contained": false,
    "fc_ovlp_to_graph_option": " --min-len 10000",
    "genome_size": "570000000",
    "input_fofn": "subreads.fasta.fofn",
    "input_type": "raw",
    "length_cutoff": "-1",
    "length_cutoff_pr": "10000",
    "overlap_filtering_setting": "--max-diff 100 --max-cov 100 --min-cov 4 --n-core 24",
    "ovlp_DBdust_option": "",
    "ovlp_DBsplit_option": "-s400",
    "ovlp_HPCdaligner_option": "-v -B128 -M24 -k24 -h600 -e.95 -l1800 -s100",
    "ovlp_concurrent_jobs": "288 -s50",
    "ovlp_daligner_option": "-k24 -h480 -e.95 -l1800 -s100",
    "pa_DBdust_option": "",
    "pa_DBsplit_option": "-x500 -s50",
    "pa_HPCREPmask_option": "-k18 -h480 -w8 -e.8 -s100",
    "pa_HPCTANmask_option": "-k18 -h480 -w8 -e.8 -s100",
    "pa_HPCdaligner_option": "-v -B128 -e0.75 -M24 -l1200 -k14 -h256 -w8 -s100 -t16",
    "pa_REPmask_code": "0,300/0,300/0,300",
    "pa_daligner_option": "-e0.75 -l1200 -k14 -h256 -w8 -s100",
    "pa_dazcon_option": "-j 4 -x -l 500",
    "pa_fasta_filter_option": "streamed-internal-median",
    "pa_subsample_coverage": 0,
    "pa_subsample_random_seed": 12345,
    "pa_subsample_strategy": "random",
    "seed_coverage": "40",
    "skip_checks": false,
    "target": "assembly"
  },
  "job.defaults": {
    "JOB_QUEUE": "default",
    "MB": "32768",
    "NPROC": "6",
    "job_type": "local",
    "njobs": "32",
    "pwatcher_type": "blocking",
    "submit": "/bin/bash -c \"${JOB_SCRIPT}\" > \"${JOB_STDOUT}\" 2> \"${JOB_STDERR}\"",
    "use_tmpdir": false
  },
  "job.step.asm": {
    "MB": "192000",
    "NPROC": "24",
    "njobs": "1"
  },
  "job.step.cns": {
    "MB": "64000",
    "NPROC": "8",
    "njobs": "200"
  },
  "job.step.da": {
    "MB": "32000",
    "NPROC": "4",
    "njobs": "300"
  },
  "job.step.dust": {},
  "job.step.la": {
    "MB": "64000",
    "NPROC": "8",
    "njobs": "200"
  },
  "job.step.pda": {
    "MB": "64000",
    "NPROC": "8",
    "njobs": "200"
  },
  "job.step.pla": {
    "MB": "32000",
    "NPROC": "4",
    "njobs": "300"
  }
}
[INFO]In simple_pwatcher_bridge, pwatcher_impl=<module 'pwatcher.blocking' from '/home/sadik/.conda/envs/falcon/lib/python2.7/site-packages/pwatcher/blocking.pyc'>
[INFO]job_type='local', (default)job_defaults={'JOB_QUEUE': 'default', 'pwatcher_type': 'blocking', 'use_tmpdir': False, 'MB': '32768', 'job_type': 'local', 'submit': '/bin/bash -c "${JOB_SCRIPT}" > "${JOB_STDOUT}" 2> "${JOB_STDERR}"', 'NPROC': '6', 'njobs': '32'}, use_tmpdir=False, squash=False, job_name_style=0
[INFO]Setting max_jobs to 32; was None
[INFO]Num unsatisfied: 2, graph: 2
[INFO]About to submit: Node(0-rawreads/build)
[INFO]Popen: '/bin/bash -c "/home/sadik/.conda/envs/falcon/lib/python2.7/site-packages/pwatcher/mains/job_start.sh" > "/home/assembly/falcon/0-rawreads/build/run-P593808baa38b0f.bash.stdout" 2> "/home/assembly/falcon/0-rawreads/build/run-P593808baa38b0f.bash.stderr"'
[INFO](slept for another 0.0s -- another 1 loop iterations)
[INFO](slept for another 0.3s -- another 2 loop iterations)
[INFO](slept for another 1.2s -- another 3 loop iterations)
[INFO](slept for another 3.0s -- another 4 loop iterations)
[INFO](slept for another 6.0s -- another 5 loop iterations)
[INFO](slept for another 10.5s -- another 6 loop iterations)
[INFO](slept for another 16.8s -- another 7 loop iterations)
[INFO](slept for another 25.2s -- another 8 loop iterations)
[INFO](slept for another 36.0s -- another 9 loop iterations)
[INFO](slept for another 49.5s -- another 10 loop iterations)
[INFO](slept for another 66.0s -- another 11 loop iterations)
[INFO](slept for another 85.8s -- another 12 loop iterations)
[INFO](slept for another 109.2s -- another 13 loop iterations)
[INFO](slept for another 135.6s -- another 14 loop iterations)
[INFO](slept for another 150.0s -- another 15 loop iterations)
[ERROR]Task Node(0-rawreads/build) failed with exit-code=1
[ERROR]Some tasks are recently_done but not satisfied: set([Node(0-rawreads/build)])
[ERROR]ready: set([])
    submitted: set([])
[ERROR]Noop. We cannot kill blocked threads. Hopefully, everything will die on SIGTERM.
Traceback (most recent call last):
  File "/home/sadik.conda/envs/falcon/bin/fc_run", line 11, in <module>
    load_entry_point('falcon-kit==1.3.0', 'console_scripts', 'fc_run')()
  File "/home/sadik/.conda/envs/falcon/lib/python2.7/site-packages/falcon_kit/mains/run1.py", line 706, in main
    main1(argv[0], args.config, args.logger)
  File "/home/sadik/.conda/envs/falcon/lib/python2.7/site-packages/falcon_kit/mains/run1.py", line 73, in main1
    input_fofn_fn=input_fofn_fn,
  File "/home/sadik/.conda/envs/falcon/lib/python2.7/site-packages/falcon_kit/mains/run1.py", line 235, in run
    dist=Dist(NPROC=4, MB=4000, job_dict=config['job.step.da']),
  File "/home/sadik/.conda/envs/falcon/lib/python2.7/site-packages/falcon_kit/pype.py", line 109, in gen_parallel_tasks
    wf.refreshTargets()
  File "/home/sadik/.conda/envs/falcon/lib/python2.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 278, in refreshTargets
    self._refreshTargets(updateFreq, exitOnFailure)
  File "/home/sadik/.conda/envs/falcon/lib/python2.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 362, in _refreshTargets
    raise Exception(msg)
Exception: Some tasks are recently_done but not satisfied: set([Node(0-rawreads/build)])
sadikmu commented 4 years ago

Apologise @pb-cdunn, Any suggestion please? I got stuck on this.

pb-cdunn commented 3 years ago

We've been concentrating on pbipa, which now works really well if you have HiFi data (i.e. ccs reads).

Try using an absolute path for input_fofn = subreads.fasta.fofn I think it's having trouble finding that.