search

snakemake / snakemake-executor-plugin-slurm

A Snakemake executor plugin for submitting jobs to a SLURM cluster
MIT License
15 stars 17 forks source link

option to set the log directory? #94

Open jbloom opened 4 months ago

jbloom commented 4 months ago

Is it possible to enable the log directory to be set to something other than a directory in .snakemake? For large workflows that submit many jobs, it would be nice to be able to write these to a scratch directory rather than putting tens of thousands of files in .snakemake.

bentyeh commented 4 months ago

I'd like to second the above request and consolidate several previous requests across multiple issues for giving the user more flexibility in specifying how to log stdout and stderr from rules submitted as SLURM jobs:

  1. Specify log directory (this issue, #94)
  2. Specify which wildcards to use, not just concatenate all of them (#11, #34, #84)
  3. Specify separate names and paths for stdout (to pass to sbatch --output=...) and stderr (to pass to sbatch --error=...) (#95)

More generally, I hope that this new executor framework can at least achieve feature parity with the previous version (e.g., v7) of Snakemake that supported a very flexible logging setup via snakemake --cluster-config cluster.yaml ... <Snakefile>

where cluster.yaml looked something like

__default__:
    time: 01:00:00
    threads: 2
    mem: 10G
    error: "logs/{rule}.{wildcards.sample}.err"
    output: "logs/{rule}.{wildcards.sample}.out"
cmeesters commented 4 months ago

@jbloom no, however, refactoring the logging feature (particularly, deleting log files older than n days) is on my TODO list. The log directory is a core feature - I will discuss this internally. We will work on this, it might take a little while.

@bentyeh please do not mix issues. I ensure, that we take all feature requests seriously. ad 2) I can see your point, however, this too needs some discussion. After all, implementing it the way it is, was a feature request. And a change is not as straightforward as it seems: configurability must be maintained at three points: the command line, the Snakefiles and the profile. Adding a keep-those-wildcadrs-option for the CLI would be awkward, right?

Regarding your other critics: the workflow profile offers just that, albeit that I too would like to see something changed. It ought to be possible to specify a single file, not a directory.

jbloom commented 4 months ago

@cmeesters, great, thank you so much for considering customizability of the log directory as a feature. That would be tremendously useful and I appreciate your time and consideration of this.