Specifying non-standard rule-specific cluster resource settings

[kpj]

Hello,

In order to specify e.g. a time limit (which can be needed to select an appropriate queue) for a certain rule, one needs to create lsf.yaml (see also https://github.com/Snakemake-Profiles/snakemake-lsf/issues/7 and https://github.com/Snakemake-Profiles/snakemake-lsf/issues/13). This file needs to be in the working directory of the pipeline.

This can be difficult/slightly annoying to handle when lsf.yaml needs to be moved manually to the correct working directory before executing the pipeline is possible.

Is there a better way of doing this that I am missing? Or asked differently, what is the difference between standard and non-standard resources?

[mbhall88]

Yes, it does need to be in the working directory sorry. There is no way of propagating information about the location of this file to the submitter (that I know of - happy to see a PR if you can do this).

[kpj]

My guess would have been to use snakemake.workflow.srcdir in https://github.com/Snakemake-Profiles/snakemake-lsf/blob/2e6f23cbea58bb07bde5eff873be6bc87f2a4018/%7B%7Bcookiecutter.profile_name%7D%7D/lsf_submit.py#L223 but I am not sure if the global workflow variable needed to execute srcdir is available.

I can check whether this could work. If not, do you think that it's worth asking in the main Snakemake repository about it?

[mbhall88]

My guess would have been to use snakemake.workflow.srcdir in but I am not sure if the global workflow variable needed to execute srcdir is available.

I can check whether this could work. If not, do you think that it's worth asking in the main Snakemake repository about it?

I see. We could have it as an option when creating the template from cookiecutter. i.e. you put in the location to the config file, relative to the working directory?

[kpj]

This would only work for a single specific working directory, wouldn't it? If the working directory was changed, one would have to rerun cookiecutter.

[mbhall88]

Correct. If snakemake.workflow.srcdir is available in the submitter then that would obviously be the better option.

[kpj]

As expected, using snakemake.workflow.srcdir in lsf/lsf_submit.py does not work due to the global workflow variable:

In [2]: import snakemake

In [3]: snakemake.workflow.srcdir('.')
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
~/Library/Application Support/snakemake/lsf/lsf_submit.py in <module>
----> 1 snakemake.workflow.srcdir('.')

~/Library/Python/3.7/lib/python/site-packages/snakemake/workflow.py in srcdir(path)
   1548 def srcdir(path):
   1549     """Return the absolute path, relative to the source directory of the current Snakefile."""
-> 1550     if not workflow.included_stack:
   1551         return None
   1552     return os.path.join(os.path.dirname(workflow.included_stack[-1]), path)

NameError: name 'workflow' is not defined

I have asked for a better solution in the main snakemake repository: https://github.com/snakemake/snakemake/issues/424.