snakemake incompatible to the provided slurm profile

[zhanxw]

Describe the bug In snakemake (v8.4.7), "--cluster-status" is not supported, and thus the "cluster-status" in config.yaml will not work.

For more details on this (or incompatible between snakemake and profile), see: hybracter

[aforestsomewhere]

It seems possible to use the new cluster-executor plugins with Hybracter with a few tweaks, that is working in my hands on the test data.

To add to requirements/recipe (I installed these into the virtual conda environment in which I have installed hybracter):

• snakemake-minimal=8.10.6
• snakemake-executor-cluster-generic
• snakemake-executor-plugin-slurm

config.yaml to be renamed to config.v8+.yaml and contents updated:

• add "executor: cluster-generic";
• amend "cluster:" to "cluster-generic-submit-cmd:"
• amend "cluster-status:" to "cluster-generic-status-cmd:"

• (optional): add "cluster-generic-cancel-cmd: scancel"

  executor: cluster-generic
  cluster-generic-submit-cmd:
  mkdir -p logs/{rule} &&
  sbatch
  --nodes=1
  --cpus-per-task={threads}
  --time={resources.time}
  --job-name=smk-{rule}
  --parsable
  --output=logs/{rule}/{jobid}.out
  --error=logs/{rule}/{jobid}.err
  #cluster-generic-status-cmd: ~/.config/snakemake/slurm/slurm_status.py
  cluster-generic-cancel-cmd: scancel
  default-resources:
  - time=1320
  jobs: 200
  latency-wait: 30
  local-cores: 8
  restart-times: 1
  max-jobs-per-second: 2
  max-status-checks-per-second: 10
  local-cores: 8
  software-deployment-method: conda
  rerun-incomplete: True
  printshellcmds: True
  scheduler: greedy

Sharing in case it is helpful - probably worth testing on other HPC setups (for example I have issues with the sacct command in snakemake workflows locally so could not test the slurm_status.py checker)

[gbouras13]

Hi @aforestsomewhere ,

Do you mind if I add this to the documentation eventually - thanks so much for this!

George

[aforestsomewhere]

Hi @aforestsomewhere ,

Do you mind if I add this to the documentation eventually - thanks so much for this!

George

Of course, glad it was somewhat useful!

Katie

[momsane]

Hi, I was wondering if the latest versions of Hybracter now account for how snakemake v8 uses plugins instead of profiles? I see that in version 0.9.0 snakemake-interface-common, snakemake-interface-executor-plugins, snakemake-interface-report-plugins, snakemake-interface-storage-plugins and snakemake-minimal are included, but not snakemake-executor-cluster-generic and snakemake-executor-plugin-slurm.