How to set output and error files?

[mufernando]

Hi, I'm running into a problem with setting output and error.

My cluster.yaml looks like this:

__default__:
    time: 2-00:00:00
    nodes: 1
    ntasks: 1
    account: "kernlab"
    partition: "kern"
    job-name: "{rule}"
    output: "{rule}.{wildcards}.out"
    error: "{rule}_{wildcards.rand_id}_{wildcards.rep}_%j.err"

but I keep getting

Traceback (most recent call last):
  File "/home/murillor/.config/snakemake/slurm/slurm-submit.py", line 54, in <module>
    slurm_utils.ensure_dirs_exist(sbatch_options[o]) if o in sbatch_options else None
  File "/gpfs/home/murillor/.config/snakemake/slurm/slurm_utils.py", line 52, in ensure_dirs_exist
    os.makedirs(di, exist_ok=True)
  File "/home/murillor/projects/anaconda3/envs/greatapes/lib/python3.7/os.py", line 221, in makedirs
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: ''

I noticed the error goes away when I comment out the output and error, but either way the job name isn't being set correctly.

What am I missing? Should I be using the profile and pass this cluster.yaml as a cluster-config? I didn't want to do that bc the cluster-config was deprecated.

Thank you!

[johnstonmj]

Hi @percyfal

As we discussed briefly before, I'm experiencing the same issue as @mufernando .

I'd personally prefer not to be limited to slurm's logs/slurm-%x-%j.out. I find the snakemake {rule}.{wildcards} notation much easier to identify which job a particular log corresponds to.

Additionally, I'd prefer to organize my log directories such that logs are generated at:

logs/{rule}/{rule}.{wildcards}.out
logs/{rule}/{rule}.{wildcards}.err

With many samples, the flat structure of logs/{rule}.{wildcards}.err (without an intermediate directory) can lead to a very overwhelming 'logs' directory.

1) Is using the (deprecated) --cluster-config still the best option?

2) If not, how should we pass non-integer, per-rule cluster options? Partitions and log files are my specific interests at the moment.

3) Would it be possible to map non-integer cluster options from the 'params' block? Similar to the way --cluster-config can access {rule} and {threads}, would it be possible to access a {params.slurm_opts} containing the string "--partition=gpu --output=logs/{rule}/{rule}.{wildcards}.out"

Thanks for any suggestions!

[johnstonmj]

By using: CLUSTER_CONFIG = "cluster_config.yaml" within slurm-submit.py

And ~/.config/snakemake/slurm/cluster_config.yaml:

__default__ :
    ntasks    : "1"
    cpus-per-task : "{threads}"
    nodes     : "1"
    output    : "logs/{rule}/{rule}.{wildcards}.out"
    error     : "logs/{rule}/{rule}.{wildcards}.err"

I can get error logs generated of logs/{rule}/{rule}.{wildcards}.out, however, the brace substitution is not performed. If the brace substitution could be performed as part of the slurm profile, this would be a great improvement.

[percyfal]

Hi,

@mufernando this is hopefully resolved in #41 ; could you verify?

@johnstonmj as you noted in your last comment, now you can point logs to e.g. logs/{rule}.... Brace substitution works as expected, AFAICT. If it still doesn't work as you expect, could you provide an example?

[mufernando]

Hi @percyfal

It is not throwing an error anymore, but the brace substitution isn't working for me.

Thanks for looking into this further btw!

[johnstonmj]

Hi @percyfal

Correct, this are the output names that I see:

[percyfal]

Ok, now I see what's happening; the brace substitution is not performed if cluster-config is processed via slurm_utils. I got it to work by submitting jobs with snakemake --profile profile --cluster-config cluster-config.yaml (and setting CLUSTER_CONFIG="" in slurm-submit.py). I'll look into this, but in the meantime, could you use this workaround?

[johnstonmj]

Hi @percyfal Submitting with snakemake -nq --profile slurm --cluster-config ~/.config/snakemake/slurm/cluster_config.yaml seems to be working. Thanks!

[sbamin]

I ended up referencing output and error paths in slurm-submit.py based on code from bnprks/snakemake-slurm-profile by @bnprks

Tested OK on hpc running slurm, v18.08.8.

[percyfal]

Hi @mufernando, this should now be fixed (see PR #43).

Cheers,

Per