option to set the log directory?

[jbloom]

Is it possible to enable the log directory to be set to something other than a directory in .snakemake? For large workflows that submit many jobs, it would be nice to be able to write these to a scratch directory rather than putting tens of thousands of files in .snakemake.

[bentyeh]

I'd like to second the above request and consolidate several previous requests across multiple issues for giving the user more flexibility in specifying how to log stdout and stderr from rules submitted as SLURM jobs:

1. Specify log directory (this issue, #94)
2. Specify which wildcards to use, not just concatenate all of them (#11, #34, #84)
3. Specify separate names and paths for stdout (to pass to sbatch --output=...) and stderr (to pass to sbatch --error=...) (#95)

More generally, I hope that this new executor framework can at least achieve feature parity with the previous version (e.g., v7) of Snakemake that supported a very flexible logging setup via snakemake --cluster-config cluster.yaml ... <Snakefile>

where cluster.yaml looked something like

__default__:
    time: 01:00:00
    threads: 2
    mem: 10G
    error: "logs/{rule}.{wildcards.sample}.err"
    output: "logs/{rule}.{wildcards.sample}.out"

[cmeesters]

@jbloom no, however, refactoring the logging feature (particularly, deleting log files older than n days) is on my TODO list. The log directory is a core feature - I will discuss this internally. We will work on this, it might take a little while.

@bentyeh please do not mix issues. I ensure, that we take all feature requests seriously. ad 2) I can see your point, however, this too needs some discussion. After all, implementing it the way it is, was a feature request. And a change is not as straightforward as it seems: configurability must be maintained at three points: the command line, the Snakefiles and the profile. Adding a keep-those-wildcadrs-option for the CLI would be awkward, right?

Regarding your other critics: the workflow profile offers just that, albeit that I too would like to see something changed. It ought to be possible to specify a single file, not a directory.

[jbloom]

@cmeesters, great, thank you so much for considering customizability of the log directory as a feature. That would be tremendously useful and I appreciate your time and consideration of this.

[mbhall88]

Agreed, this would be very useful.

A second thing I want to ask for is documenting where the logs are actually output to in the plugin docs (unless I've missed it).

[cmeesters]

it's on my todo list