slurm fails when log path contains a Wildcards string that is a directory

[maarten-k]

by default, the log name contains the wildcard information (logs/slurm/%r/%j-%w).

When using a wildcard that is a path, something ugly happens: the filename contains a "\" and SLURM can not handle this(as expected).

I suggest replacing all slash with an underscore inside the %w string before returning the log name.

I welcome other solutions. (as short-term solution you can remove the "%w" from your log files)

[maarten-k]

hmm, I think the problem also arises with the job name.

I think the solution can be implemented here:

https://github.com/Snakemake-Profiles/slurm/blob/dec0b2e9f29320f9bfdbcf3546d8dd65e0cff95d/%7B%7Bcookiecutter.profile_name%7D%7D/slurm_utils.py#L330

[jdblischak]

The job name can include slashes:

$ sbatch --job-name="/a/b/c/" --wrap='echo $SLURM_JOB_NAME'
Submitted batch job 51924721

$ cat slurm-51924721.out
/a/b/c/

[maarten-k]

The job name can include slashes:

$ sbatch --job-name="/a/b/c/" --wrap='echo $SLURM_JOB_NAME'
Submitted batch job 51924721

$ cat slurm-51924721.out
/a/b/c/

Thanks for letting me know. Probably I did something wrong here while trying to sort out this issue.

[mbhall88]

Can you provide a MWE?

[maarten-k]

Hereby an MWE:

Snakefile

rule index:
    input:
        "{test}.gz",
    output:
        "{test}.gz.tbi",
    shell:
        "touch {output}"

Command to execute

mkdir results
touch results/example.gz
snakemake results/example.gz.tbi -j 1 --profile slurm

I think it would also be a good idea to remove/replace awkward characters that have special meaning in bash like:

<>[]*%$

[mbhall88]

What is the consensus one how to handle wildcards that contain / characters? I can't think of anything obvious...

[mbhall88]

Thinking more about this, I guess changing the default would be sufficient and people can use %w at their own "risk" - i.e. if they have / in their wildcards

any preferences on a default?

[maarten-k]

I think "cluster_logpath": "logs/slurm/%r/%j", should be fine. The job number is provided by the snakemake log and in another case grep is your friend.