cmeesters
commented
1 month ago Hi,
this is really weird. scancel as triggered by the plugin, does not have an --exclusive flag. And why should a SLURM cluster state, that the command sbatch is not found?
Right now, I am travelling, but I will find some time next week, to look into issues. Meanwhile, can you please indicate where you submitted your workflow (within a job or on a login/head node)? And perhaps run it in verbose mode (just add to the command line --verbose) and attach a full log of Snakemake, please?
I would like to know whether sbatch points to a binary or is overwritten with a wrapper (the rather informative output is not a default and admins have several methods to give you that feedback). Can you post the output of which sbatch, too?
I'm afraid, I am not familiar with Snellius, what is your output of sacct during or after the run (same day)? (Background: the plugin keeps track of job states using SLURM's accounting mechanism.)
freekvh
r-blanchet
ifariasg
fgvieira
Software Versions
$ snakemake --version 8.23.0 $ mamba list | grep "snakemake-executor-plugin-slurm" $ conda list | grep "snakemake-executor-plugin-slurm" snakemake-executor-plugin-slurm 0.11.0 pyhdfd78af_0 bioconda snakemake-executor-plugin-slurm-jobstep 0.2.1 pyhdfd78af_0 bioconda $ sinfo --version slurm 24.05.3
Describe the bug When starting a snakemake workflow on a slurm cluster (surf/sara Snellius), the workflow starts but hangs on the initially submitted jobs (that seem to complete and disappear from the squeue overview). It's like Snakemake does not get the signal that the jobs finish. This is my config:
Logs This is how it ends (I pressed ctrl-c at this line
^CTerminating processes on user request, this might take some time.):Minimal example This is a big effort, if it's really required, I'll try to make a minimal pipelines. Apologies.
Additional context I have also posted a question on https://bioinformatics.stackexchange.com/questions/22963/snakemake-on-a-slurm-cluster, however there I ask for support for the generic executor as well (which doesn't let met specify the partition).
I think in general, some more extensive examples of profiles where all options are used would be nice to have. And, perhaps Snellius is deviating from normal Slurm?