sbatch: error: Batch script is empty!

[Kirito-Ma]

Hi team, I generated a makeflow and tried to run in slurm. I tried the Rscript in the terminal which ran it normally but when I tried to run it in the slurm, I met the error. Do you have any ideas about this error? Here is my code: (base) [ma.m@slurm-login02 ~]$ /stornext/Home/data/allstaff/m/ma.m/cctools-7.4.3-x86_64-centos7.tar.gz-dir/bin/makeflow -T slurm -J 200 /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/R_scripts/Overall_summary/GSEA.makeflow parsing /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/R_scripts/Overall_summary/GSEA.makeflow... local resources: 32.000 cores, 193277 MB memory, 80791602 MB disk max running remote jobs: 200 max running local jobs: 32

And here is the error: 2022/08/15 19:29:42.12 makeflow[7406] notice: job submission failed: no output from slurm couldn't submit batch job, still trying... sbatch: error: Batch script is empty!

[btovar]

@Kirito-Ma It looks like makeflow cannot find the slurm installation. From the terminal you are launching makeflow, could you type:

squeue

and

sbatch

If from any of those you get a message such as "command not found", then we need to figure out why your slurm installation is not available.

[Kirito-Ma]

Hello botvar,

When I ran the sbatch, it didn't get any output:

(base) [ma.m@slurm-login02 ~]$ sbatch

[Kirito-Ma]

Hello botvar,

Do you know how to solve this problem? Do I need to re-install the package again?

[btovar]

The "sbatch: error: Batch script is empty!" error tells me that makeflow is having trouble writing its wrapper files where it needs to. Do you have write permissions to the directory /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/R_scripts? Is the /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/R_scripts/ directory part of the shared filesystem?

[Kirito-Ma]

Yes, I think this directory is part of the shared filesystem. But I have permission to write in the directories.

[btovar]

Something we can try as a sanity check is:

cd /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/R_scripts/ /stornext/Home/data/allstaff/m/ma.m/cctools-7.4.3-x86_64-centos7.tar.gz-dir/bin/makeflow -T slurm -J 1 -dall Overall_summary/GSEA.makeflow (and when it fails, if it does) cat slurm.wrapper

[Kirito-Ma]

It fails and I ran cat slurm.wrapper. Here are the response:

(base) [ma.m@slurm-login02 R_scripts]$ cat slurm.wrapper

!/bin/sh

$ -S /bin/sh

[ -n "${SLURM_JOB_ID}" ] && JOB_ID=echo ${SLURM_JOB_ID} | cut -d . -f 1 logfile=slurm.status.${JOB_ID} starttime=date +%s echo start $starttime > $logfile (while true; do sleep 30; echo alive $(date +%s) >> $logfile; done) & pid_heartbeat=$! eval "$BATCH_JOB_COMMAND" status=$? kill $pid_heartbeat stoptime=date +%s echo stop $status $stoptime >> $logfile exit 0

[btovar]

Ok, can you send me the debug output?

[Kirito-Ma]

Sorry to ask a silly question, where can I find the debug output?

[btovar]

Oh, no problem, it is the output we activated by using the -dall flag. So you can do:

cd /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/R_scripts/Overall_summary/ /stornext/Home/data/allstaff/m/ma.m/cctools-7.4.3-x86_64-centos7.tar.gz-dir/bin/makeflow -T slurm -J 1 -dall GSEA.makeflow > output.log 2>&1 and send me the output.log when it fails.

[Kirito-Ma]

It really helps. Here is the ouput.log

2022/08/22 22:03:17.69 makeflow[24216] makeflow: /stornext/Home/data/allstaff/m/ma.m/cctools-7.4.3-x86_64-centos7.tar.gz-dir/bin/makeflow version 7.4.3 FINAL (released 2022-02-03 15:36:20 +0000) 2022/08/22 22:03:17.70 makeflow[24216] makeflow: Built by root@1a5df52c28be on 2022-02-03 15:36:20 +0000 2022/08/22 22:03:17.70 makeflow[24216] makeflow: System: Linux 1a5df52c28be 5.11.0-1028-azure #31~20.04.2-Ubuntu SMP Tue Jan 18 08:46:15 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux 2022/08/22 22:03:17.70 makeflow[24216] makeflow: Configuration: --debug --strict --with-ext2fs-path /opt/vc3/cctools-deps/ext2fs --with-fuse-path /opt/vc3/cctools-deps/fuse --with-mysql-path /opt/vc3/cctools-deps/mysql --with-swig-path /opt/vc3/cctools-deps/swig --with-uuid-path /opt/vc3/cctools-deps/uuid --with-perl-path no --with-python2-path /usr --with-python3-path /usr --with-cvmfs-path /usr --prefix /tmp/cctools-7.4.3-x86_64-centos7.tar.gz-dir 2022/08/22 22:03:17.70 makeflow[24216] auth: kerberos: not compiled in 2022/08/22 22:03:17.70 makeflow[24216] auth: globus: not compiled in 2022/08/22 22:03:17.70 makeflow[24216] auth: unix: registered 2022/08/22 22:03:17.70 makeflow[24216] auth: ticket: registered 2022/08/22 22:03:17.70 makeflow[24216] auth: hostname: registered 2022/08/22 22:03:17.70 makeflow[24216] auth: address: registered 2022/08/22 22:03:17.70 makeflow[24216] makeflow_hook: Hook Fail Dir:trying to register 2022/08/22 22:03:17.70 makeflow[24216] makeflow_hook: Hook Fail Dir:registered 2022/08/22 22:03:17.70 makeflow[24216] makeflow_hook: hook Fail Dir:initializing 2022/08/22 22:03:17.70 makeflow[24216] makeflow: VARIABLE: = 2022/08/22 22:03:17.70 makeflow[24216] makeflow: LITERAL: CATEGORY 2022/08/22 22:03:17.70 makeflow[24216] makeflow: LITERAL: GSEA 2022/08/22 22:03:17.70 makeflow[24216] makeflow: dag variable name=CATEGORY, value=GSEA 2022/08/22 22:03:17.70 makeflow[24216] makeflow: VARIABLE: = 2022/08/22 22:03:17.70 makeflow[24216] makeflow: LITERAL: MEMORY 2022/08/22 22:03:17.70 makeflow[24216] makeflow: LITERAL: 30024 2022/08/22 22:03:17.70 makeflow[24216] makeflow: dag variable name=MEMORY, value=30024 2022/08/22 22:03:17.70 makeflow[24216] makeflow: VARIABLE: = 2022/08/22 22:03:17.70 makeflow[24216] makeflow: LITERAL: CORES 2022/08/22 22:03:17.70 makeflow[24216] makeflow: LITERAL: 4 2022/08/22 22:03:17.70 makeflow[24216] makeflow: dag variable name=CORES, value=4 2022/08/22 22:03:17.70 makeflow[24216] makeflow: FILES: 2022/08/22 22:03:17.70 makeflow[24216] makeflow: LITERAL: /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/GSEA_data/edgeR_quasi_likelihood_B_cell_GSEA.rds 2022/08/22 22:03:17.70 makeflow[24216] makeflow: COLON 2022/08/22 22:03:17.70 makeflow[24216] makeflow: LITERAL: /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/all_raw_data/B_cell_raw.rds 2022/08/22 22:03:17.70 makeflow[24216] makeflow: NEWLINE 2022/08/22 22:03:17.70 makeflow[24216] makeflow: COMMAND: 2022/08/22 22:03:17.70 makeflow[24216] makeflow: COMMAND_MOD_END: 2022/08/22 22:03:17.70 makeflow[24216] makeflow: LITERAL: Rscript 2022/08/22 22:03:17.70 makeflow[24216] makeflow: SPACE 2022/08/22 22:03:17.70 makeflow[24216] makeflow: LITERAL: /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/R_scripts/Overall_summary/perform_GSEA.R 2022/08/22 22:03:17.70 makeflow[24216] makeflow: SPACE 2022/08/22 22:03:17.70 makeflow[24216] makeflow: LITERAL: /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/all_raw_data/B_cell_raw.rds 2022/08/22 22:03:17.70 makeflow[24216] makeflow: SPACE 2022/08/22 22:03:17.70 makeflow[24216] makeflow: LITERAL: /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/GSEA_data/edgeR_quasi_likelihood_B_cell_GSEA.rds 2022/08/22 22:03:17.70 makeflow[24216] makeflow: SPACE 2022/08/22 22:03:17.70 makeflow[24216] makeflow: LITERAL: /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/ppcseq_data/edgeR_quasi_likelihood_B_cell_ppcseq.rds 2022/08/22 22:03:17.70 makeflow[24216] makeflow: SPACE 2022/08/22 22:03:17.70 makeflow[24216] makeflow: LITERAL: B_cell 2022/08/22 22:03:17.70 makeflow[24216] makeflow: SPACE 2022/08/22 22:03:17.70 makeflow[24216] makeflow: IO_REDIRECT: > 2022/08/22 22:03:17.70 makeflow[24216] makeflow: SPACE 2022/08/22 22:03:17.70 makeflow[24216] makeflow: LITERAL: ERROR-1.output 2022/08/22 22:03:17.70 makeflow[24216] makeflow: SPACE 2022/08/22 22:03:17.70 makeflow[24216] makeflow: LITERAL: 2>&1 2022/08/22 22:03:17.70 makeflow[24216] makeflow: NEWLINE 2022/08/22 22:03:17.70 makeflow[24216] makeflow: node command=Rscript /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/R_scripts/Overall_summary/perform_GSEA.R /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/all_raw_data/B_cell_raw.rds /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/GSEA_data/edgeR_quasi_likelihood_B_cell_GSEA.rds /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/ppcseq_data/edgeR_quasi_likelihood_B_cell_ppcseq.rds B_cell > ERROR-1.output 2>&1 2022/08/22 22:03:17.70 makeflow[24216] makeflow: Category default defined as: {} 2022/08/22 22:03:17.70 makeflow[24216] makeflow: Category GSEA defined as: {"memory":[30024,"MB"],"cores":[4,"cores"]} 2022/08/22 22:03:17.70 makeflow[24216] batch: set feature local_job_queue' toyes' 2022/08/22 22:03:17.70 makeflow[24216] batch: set feature absolute_path' toyes' 2022/08/22 22:03:17.70 makeflow[24216] batch: set feature output_directories' toyes' 2022/08/22 22:03:17.70 makeflow[24216] batch: set feature batch_log_name' to%s.batchlog' 2022/08/22 22:03:17.70 makeflow[24216] batch: set feature gc_size' toyes' 2022/08/22 22:03:17.70 makeflow[24216] batch: created queue 0x138baa0 (slurm) 2022/08/22 22:03:17.70 makeflow[24216] batch: set logfile to GSEA.makeflow.batchlog' 2022/08/22 22:03:17.70 makeflow[24216] batch: cleared optionbatch-options' 2022/08/22 22:03:17.70 makeflow[24216] batch: cleared option password' 2022/08/22 22:03:17.70 makeflow[24216] batch: set optionmanager-mode' to standalone' 2022/08/22 22:03:17.70 makeflow[24216] batch: cleared optionname' 2022/08/22 22:03:17.70 makeflow[24216] batch: cleared option debug' 2022/08/22 22:03:17.70 makeflow[24216] batch: cleared optionpriority' 2022/08/22 22:03:17.70 makeflow[24216] batch: cleared option keepalive-interval' 2022/08/22 22:03:17.70 makeflow[24216] batch: cleared optionkeepalive-timeout' 2022/08/22 22:03:17.70 makeflow[24216] batch: set option caching' toyes' 2022/08/22 22:03:17.70 makeflow[24216] batch: cleared option wait-queue-size' 2022/08/22 22:03:17.70 makeflow[24216] batch: cleared optionamazon-config' 2022/08/22 22:03:17.70 makeflow[24216] batch: cleared option lambda-config' 2022/08/22 22:03:17.70 makeflow[24216] batch: cleared optionworking-dir' 2022/08/22 22:03:17.70 makeflow[24216] batch: cleared option manager-preferred-connection' 2022/08/22 22:03:17.70 makeflow[24216] batch: cleared optionamazon-batch-config' 2022/08/22 22:03:17.70 makeflow[24216] batch: cleared option amazon-batch-img' 2022/08/22 22:03:17.70 makeflow[24216] batch: set optionsafe-submit-mode' to no' 2022/08/22 22:03:17.70 makeflow[24216] batch: set optionignore-mem-spec' to no' 2022/08/22 22:03:17.70 makeflow[24216] batch: cleared optionmem-type' 2022/08/22 22:03:17.70 makeflow[24216] batch: set option keep-wrapper-stdout' tono' 2022/08/22 22:03:17.70 makeflow[24216] batch: set option tlq-port' to0' 2022/08/22 22:03:17.70 makeflow[24216] batch: set option fast-abort' to-1.000000' 2022/08/22 22:03:17.70 makeflow[24216] batch: set feature local_job_queue' toyes' 2022/08/22 22:03:17.70 makeflow[24216] batch: set feature absolute_path' toyes' 2022/08/22 22:03:17.70 makeflow[24216] batch: set feature output_directories' toyes' 2022/08/22 22:03:17.70 makeflow[24216] batch: set feature batch_log_name' to%s.batchlog' 2022/08/22 22:03:17.70 makeflow[24216] batch: set feature gc_size' toyes' 2022/08/22 22:03:17.70 makeflow[24216] batch: cleared feature `local_job_queue' 2022/08/22 22:03:17.70 makeflow[24216] batch: created queue 0x138ddd0 (local) 2022/08/22 22:03:17.70 makeflow[24216] makeflow: MAKEFLOW_INPUTS is not specified 2022/08/22 22:03:17.70 makeflow[24216] makeflow: Added /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/all_raw_data/B_cell_raw.rds to input list 2022/08/22 22:03:17.70 makeflow[24216] makeflow: MAKEFLOW_OUTPUTS is not specified 2022/08/22 22:03:17.70 makeflow[24216] makeflow: Added /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/GSEA_data/edgeR_quasi_likelihood_B_cell_GSEA.rds to output list 2022/08/22 22:03:17.70 makeflow[24216] makeflow: checking for consistency of batch system support... parsing GSEA.makeflow... local resources: 32.000 cores, 193277 MB memory, 24583695 MB disk max running remote jobs: 1 max running local jobs: 32 checking GSEA.makeflow for consistency... GSEA.makeflow has 1 rules. recovering from log file GSEA.makeflow.makeflowlog... checking for old running or failed jobs... checking files for unexpected changes... (use --skip-file-check to skip this step) 2022/08/22 22:03:17.70 makeflow[24216] makeflow: file GSEA.makeflow.batchlog waiting -> running starting workflow.... nothing left to do. 2022/08/22 22:03:17.70 makeflow[24216] batch: deleting queue 0x138baa0 2022/08/22 22:03:17.71 makeflow[24216] batch: deleting queue 0x138ddd0

[btovar]

Uhm, that completed successfully.

Let's try: cd /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/R_scripts/Overall_summary/ /stornext/Home/data/allstaff/m/ma.m/cctools-7.4.3-x86_64-centos7.tar.gz-dir/bin/makeflow -c GSEA.makeflow /stornext/Home/data/allstaff/m/ma.m/cctools-7.4.3-x86_64-centos7.tar.gz-dir/bin/makeflow -T slurm -J 1 -dall GSEA.makeflow > output.log 2>&1

[Kirito-Ma]

Hi btovar,

Here are the responses:

2022/08/22 22:11:32.27 makeflow[26669] makeflow: /stornext/Home/data/allstaff/m/ma.m/cctools-7.4.3-x86_64-centos7.tar.gz-dir/bin/makeflow version 7.4.3 FINAL (released 2022-02-03 15:36:20 +0000) 2022/08/22 22:11:32.27 makeflow[26669] makeflow: Built by root@1a5df52c28be on 2022-02-03 15:36:20 +0000 2022/08/22 22:11:32.27 makeflow[26669] makeflow: System: Linux 1a5df52c28be 5.11.0-1028-azure #31~20.04.2-Ubuntu SMP Tue Jan 18 08:46:15 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux 2022/08/22 22:11:32.27 makeflow[26669] makeflow: Configuration: --debug --strict --with-ext2fs-path /opt/vc3/cctools-deps/ext2fs --with-fuse-path /opt/vc3/cctools-deps/fuse --with-mysql-path /opt/vc3/cctools-deps/mysql --with-swig-path /opt/vc3/cctools-deps/swig --with-uuid-path /opt/vc3/cctools-deps/uuid --with-perl-path no --with-python2-path /usr --with-python3-path /usr --with-cvmfs-path /usr --prefix /tmp/cctools-7.4.3-x86_64-centos7.tar.gz-dir 2022/08/22 22:11:32.27 makeflow[26669] auth: kerberos: not compiled in 2022/08/22 22:11:32.27 makeflow[26669] auth: globus: not compiled in 2022/08/22 22:11:32.27 makeflow[26669] auth: unix: registered 2022/08/22 22:11:32.27 makeflow[26669] auth: ticket: registered 2022/08/22 22:11:32.27 makeflow[26669] auth: hostname: registered 2022/08/22 22:11:32.27 makeflow[26669] auth: address: registered 2022/08/22 22:11:32.27 makeflow[26669] makeflow_hook: Hook Fail Dir:trying to register 2022/08/22 22:11:32.27 makeflow[26669] makeflow_hook: Hook Fail Dir:registered 2022/08/22 22:11:32.27 makeflow[26669] makeflow_hook: hook Fail Dir:initializing 2022/08/22 22:11:32.27 makeflow[26669] makeflow: VARIABLE: = 2022/08/22 22:11:32.27 makeflow[26669] makeflow: LITERAL: CATEGORY 2022/08/22 22:11:32.27 makeflow[26669] makeflow: LITERAL: GSEA 2022/08/22 22:11:32.27 makeflow[26669] makeflow: dag variable name=CATEGORY, value=GSEA 2022/08/22 22:11:32.27 makeflow[26669] makeflow: VARIABLE: = 2022/08/22 22:11:32.27 makeflow[26669] makeflow: LITERAL: MEMORY 2022/08/22 22:11:32.27 makeflow[26669] makeflow: LITERAL: 30024 2022/08/22 22:11:32.27 makeflow[26669] makeflow: dag variable name=MEMORY, value=30024 2022/08/22 22:11:32.27 makeflow[26669] makeflow: VARIABLE: = 2022/08/22 22:11:32.27 makeflow[26669] makeflow: LITERAL: CORES 2022/08/22 22:11:32.27 makeflow[26669] makeflow: LITERAL: 4 2022/08/22 22:11:32.27 makeflow[26669] makeflow: dag variable name=CORES, value=4 2022/08/22 22:11:32.27 makeflow[26669] makeflow: FILES: 2022/08/22 22:11:32.27 makeflow[26669] makeflow: LITERAL: /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/GSEA_data/edgeR_quasi_likelihood_B_cell_GSEA.rds 2022/08/22 22:11:32.27 makeflow[26669] makeflow: COLON 2022/08/22 22:11:32.27 makeflow[26669] makeflow: LITERAL: /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/all_raw_data/B_cell_raw.rds 2022/08/22 22:11:32.27 makeflow[26669] makeflow: NEWLINE 2022/08/22 22:11:32.27 makeflow[26669] makeflow: COMMAND: 2022/08/22 22:11:32.27 makeflow[26669] makeflow: COMMAND_MOD_END: 2022/08/22 22:11:32.27 makeflow[26669] makeflow: LITERAL: Rscript 2022/08/22 22:11:32.27 makeflow[26669] makeflow: SPACE 2022/08/22 22:11:32.27 makeflow[26669] makeflow: LITERAL: /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/R_scripts/Overall_summary/perform_GSEA.R 2022/08/22 22:11:32.27 makeflow[26669] makeflow: SPACE 2022/08/22 22:11:32.27 makeflow[26669] makeflow: LITERAL: /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/all_raw_data/B_cell_raw.rds 2022/08/22 22:11:32.27 makeflow[26669] makeflow: SPACE 2022/08/22 22:11:32.27 makeflow[26669] makeflow: LITERAL: /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/GSEA_data/edgeR_quasi_likelihood_B_cell_GSEA.rds 2022/08/22 22:11:32.27 makeflow[26669] makeflow: SPACE 2022/08/22 22:11:32.27 makeflow[26669] makeflow: LITERAL: /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/ppcseq_data/edgeR_quasi_likelihood_B_cell_ppcseq.rds 2022/08/22 22:11:32.27 makeflow[26669] makeflow: SPACE 2022/08/22 22:11:32.27 makeflow[26669] makeflow: LITERAL: B_cell 2022/08/22 22:11:32.27 makeflow[26669] makeflow: SPACE 2022/08/22 22:11:32.27 makeflow[26669] makeflow: IO_REDIRECT: > 2022/08/22 22:11:32.27 makeflow[26669] makeflow: SPACE 2022/08/22 22:11:32.27 makeflow[26669] makeflow: LITERAL: ERROR-1.output 2022/08/22 22:11:32.27 makeflow[26669] makeflow: SPACE 2022/08/22 22:11:32.27 makeflow[26669] makeflow: LITERAL: 2>&1 2022/08/22 22:11:32.27 makeflow[26669] makeflow: NEWLINE 2022/08/22 22:11:32.27 makeflow[26669] makeflow: node command=Rscript /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/R_scripts/Overall_summary/perform_GSEA.R /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/all_raw_data/B_cell_raw.rds /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/GSEA_data/edgeR_quasi_likelihood_B_cell_GSEA.rds /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/ppcseq_data/edgeR_quasi_likelihood_B_cell_ppcseq.rds B_cell > ERROR-1.output 2>&1 2022/08/22 22:11:32.27 makeflow[26669] makeflow: Category default defined as: {} 2022/08/22 22:11:32.27 makeflow[26669] makeflow: Category GSEA defined as: {"memory":[30024,"MB"],"cores":[4,"cores"]} 2022/08/22 22:11:32.27 makeflow[26669] batch: set feature local_job_queue' toyes' 2022/08/22 22:11:32.27 makeflow[26669] batch: set feature absolute_path' toyes' 2022/08/22 22:11:32.27 makeflow[26669] batch: set feature output_directories' toyes' 2022/08/22 22:11:32.27 makeflow[26669] batch: set feature batch_log_name' to%s.batchlog' 2022/08/22 22:11:32.27 makeflow[26669] batch: set feature gc_size' toyes' 2022/08/22 22:11:32.27 makeflow[26669] batch: created queue 0x20d1aa0 (slurm) 2022/08/22 22:11:32.27 makeflow[26669] batch: set logfile to GSEA.makeflow.batchlog' 2022/08/22 22:11:32.27 makeflow[26669] batch: cleared optionbatch-options' 2022/08/22 22:11:32.27 makeflow[26669] batch: cleared option password' 2022/08/22 22:11:32.27 makeflow[26669] batch: set optionmanager-mode' to standalone' 2022/08/22 22:11:32.27 makeflow[26669] batch: cleared optionname' 2022/08/22 22:11:32.27 makeflow[26669] batch: cleared option debug' 2022/08/22 22:11:32.27 makeflow[26669] batch: cleared optionpriority' 2022/08/22 22:11:32.27 makeflow[26669] batch: cleared option keepalive-interval' 2022/08/22 22:11:32.27 makeflow[26669] batch: cleared optionkeepalive-timeout' 2022/08/22 22:11:32.27 makeflow[26669] batch: set option caching' toyes' 2022/08/22 22:11:32.27 makeflow[26669] batch: cleared option wait-queue-size' 2022/08/22 22:11:32.27 makeflow[26669] batch: cleared optionamazon-config' 2022/08/22 22:11:32.27 makeflow[26669] batch: cleared option lambda-config' 2022/08/22 22:11:32.27 makeflow[26669] batch: cleared optionworking-dir' 2022/08/22 22:11:32.27 makeflow[26669] batch: cleared option manager-preferred-connection' 2022/08/22 22:11:32.27 makeflow[26669] batch: cleared optionamazon-batch-config' 2022/08/22 22:11:32.27 makeflow[26669] batch: cleared option amazon-batch-img' 2022/08/22 22:11:32.27 makeflow[26669] batch: set optionsafe-submit-mode' to no' 2022/08/22 22:11:32.27 makeflow[26669] batch: set optionignore-mem-spec' to no' 2022/08/22 22:11:32.27 makeflow[26669] batch: cleared optionmem-type' 2022/08/22 22:11:32.27 makeflow[26669] batch: set option keep-wrapper-stdout' tono' 2022/08/22 22:11:32.27 makeflow[26669] batch: set option tlq-port' to0' 2022/08/22 22:11:32.27 makeflow[26669] batch: set option fast-abort' to-1.000000' 2022/08/22 22:11:32.27 makeflow[26669] batch: set feature local_job_queue' toyes' 2022/08/22 22:11:32.27 makeflow[26669] batch: set feature absolute_path' toyes' 2022/08/22 22:11:32.27 makeflow[26669] batch: set feature output_directories' toyes' 2022/08/22 22:11:32.27 makeflow[26669] batch: set feature batch_log_name' to%s.batchlog' 2022/08/22 22:11:32.27 makeflow[26669] batch: set feature gc_size' toyes' 2022/08/22 22:11:32.27 makeflow[26669] batch: cleared feature local_job_queue' 2022/08/22 22:11:32.27 makeflow[26669] batch: created queue 0x20d3dd0 (local) 2022/08/22 22:11:32.27 makeflow[26669] makeflow: MAKEFLOW_INPUTS is not specified 2022/08/22 22:11:32.27 makeflow[26669] makeflow: Added /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/all_raw_data/B_cell_raw.rds to input list 2022/08/22 22:11:32.27 makeflow[26669] makeflow: MAKEFLOW_OUTPUTS is not specified 2022/08/22 22:11:32.27 makeflow[26669] makeflow: Added /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/GSEA_data/edgeR_quasi_likelihood_B_cell_GSEA.rds to output list 2022/08/22 22:11:32.27 makeflow[26669] makeflow: checking for consistency of batch system support... parsing GSEA.makeflow... local resources: 32.000 cores, 193277 MB memory, 24583665 MB disk max running remote jobs: 1 max running local jobs: 32 checking GSEA.makeflow for consistency... GSEA.makeflow has 1 rules. creating new log file GSEA.makeflow.makeflowlog... checking files for unexpected changes... (use --skip-file-check to skip this step) 2022/08/22 22:11:32.28 makeflow[26669] makeflow: file GSEA.makeflow.batchlog waiting -> running starting workflow.... 2022/08/22 22:11:32.28 makeflow[26669] makeflow: export CORES=4 2022/08/22 22:11:32.28 makeflow[26669] makeflow: export MEMORY=30024 2022/08/22 22:11:32.28 makeflow[26669] makeflow: export CATEGORY=GSEA 2022/08/22 22:11:32.28 makeflow[26669] batch: set optiontask-id' to `0' 2022/08/22 22:11:32.28 makeflow[26669] makeflow: file /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/GSEA_data/edgeR_quasi_likelihood_B_cell_GSEA.rds waiting -> running submitting job: Rscript /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/R_scripts/Overall_summary/perform_GSEA.R /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/all_raw_data/B_cell_raw.rds /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/GSEA_data/edgeR_quasi_likelihood_B_cell_GSEA.rds /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/ppcseq_data/edgeR_quasi_likelihood_B_cell_ppcseq.rds B_cell > ERROR-1.output 2>&1 2022/08/22 22:11:32.28 makeflow[26669] batch: sbatch --mem=30024M -N 1 -n 1 -c 4 -D . -e /dev/null --export=ALL -o /dev/null -J makeflow0 ./slurm.wrapper 2022/08/22 22:11:32.30 makeflow[26669] batch: job 7622321 submitted submitted job 7622321 2022/08/22 22:11:32.30 makeflow[26669] makeflow: node 0 was successfully submitted. 2022/08/22 22:11:32.30 makeflow[26669] makeflow: node 0 waiting -> running 2022/08/22 22:11:32.30 makeflow[26669] batch: could not open status file "slurm.status.7622321" 2022/08/22 22:11:33.30 makeflow[26669] batch: could not open status file "slurm.status.7622321" 2022/08/22 22:13:01.34 makeflow[26669] batch: job 7622321 complete job 7622321 completed 2022/08/22 22:13:01.34 makeflow[26669] makeflow: Job 7622321 has returned. 2022/08/22 22:13:01.34 makeflow[26669] makeflow: File /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/GSEA_data/edgeR_quasi_likelihood_B_cell_GSEA.rds created by rule 0. 2022/08/22 22:13:01.34 makeflow[26669] makeflow: file /stornext/Bioinf/data/bioinf-data/Papenfuss_lab/projects/ma.m/single_cell_outliers/single_cell_database/COVID_19/data/GSEA_data/edgeR_quasi_likelihood_B_cell_GSEA.rds running -> receive 2022/08/22 22:13:01.35 makeflow[26669] makeflow: node 0 running -> complete nothing left to do. 2022/08/22 22:13:01.35 makeflow[26669] batch: deleting queue 0x20d1aa0 2022/08/22 22:13:01.35 makeflow[26669] batch: deleting queue 0x20d3dd0

[btovar]

All right, these seem to be working. It does seem that makeflow want's to run in the same place as the workflow file is. For now, could you cd to where the makeflow file is, and run from there? I can check the code in the meanwhile.

[Kirito-Ma]

I am not sure if I get your ideas. For now, the makeflow is in my home directory. And my makefile in another directory (/stornext/Biof/...). You mean I should use cd to the direct directory my R script is. And then I can use the makeflow : /stornext/Home/data/allstaff/m/ma.m/cctools-7.4.3-x86_64-centos7.tar.gz-dir/bin/makeflow -T slurm -J 200 ./GSEA.makeflow

Did I misunderstand something?

[btovar]

Yes, I mean that you should change to the directory where the makefile is, and then use a command as you posted above, e.g.: /stornext/Home/data/allstaff/m/ma.m/cctools-7.4.3-x86_64-centos7.tar.gz-dir/bin/makeflow -T slurm -J 200 ./GSEA.makeflow

[Kirito-Ma]

Sure. I will test it now. Many thanks to that.

[btovar]

Yes, I apologize for the late response, I was out for the weekend. Checking the code, it does seem that when using slurm, makeflow expects to run in the same place as the makefile. I'm investigating how to make this more flexible so that it runs with your original command.

[Kirito-Ma]

No worries at all! I think it works this time. This situation is really weird since I worked well by using this code in slurm. I am not sure why it doesn't work anymore. Many thanks for this help. Do you have any ideas why this situation occurs?

[btovar]

I think it has something to do with some implicit expectations that makeflow has about the directory the makefile is. I need to run a couple of tests to confirm this idea. At least you can continue to make progress for now!

[Kirito-Ma]

Yes, I will continue with my work. Really thanks for your help. For this problem, I will wait for your response. If it doesn't work, just leave it here since I can still use this strategy!

[btovar]

Closing this one so that all further comments go to https://github.com/cooperative-computing-lab/cctools/issues/2957