snakemake version compatibility

[brantfaircloth]

Now testing on HPC. It seems the snakemake version installed by default w/ the phyloacc bioconda package (currently v8.18.2) is not compatible with the slurm profile setup. --dryrun fails with:

snakemake -p -s /ddnA/work/brant/phyloacc-test/phyloacc-outgroup/phyloacc-job-files/snakemake/run_phyloacc.smk --configfile /ddnA/work/brant/phyloacc-test/phyloacc-outgroup/phyloacc-job-files/snakemake/phyloacc-config.yaml --profile /ddnA/work/brant/phyloacc-test/phyloacc-outgroup/phyloacc-job-files/snakemake/profiles/slurm_profile --dryrun
Using profile /ddnA/work/brant/phyloacc-test/phyloacc-outgroup/phyloacc-job-files/snakemake/profiles/slurm_profile for setting default command line arguments.
Building DAG of jobs...
Full Traceback (most recent call last):
  File "/project/brant/db-home/miniconda/envs/phyloacc/lib/python3.12/site-packages/snakemake/cli.py", line 2116, in args_to_api
    dag_api.execute_workflow(
  File "/project/brant/db-home/miniconda/envs/phyloacc/lib/python3.12/site-packages/snakemake/api.py", line 594, in execute_workflow
    workflow.execute(
  File "/project/brant/db-home/miniconda/envs/phyloacc/lib/python3.12/site-packages/snakemake/workflow.py", line 1082, in execute
    self._build_dag()
  File "/project/brant/db-home/miniconda/envs/phyloacc/lib/python3.12/site-packages/snakemake/workflow.py", line 1038, in _build_dag
    async_run(self.dag.init())
  File "/project/brant/db-home/miniconda/envs/phyloacc/lib/python3.12/site-packages/snakemake/common/__init__.py", line 94, in async_run
    return asyncio.run(coroutine)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/project/brant/db-home/miniconda/envs/phyloacc/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/project/brant/db-home/miniconda/envs/phyloacc/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/project/brant/db-home/miniconda/envs/phyloacc/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/project/brant/db-home/miniconda/envs/phyloacc/lib/python3.12/site-packages/snakemake/dag.py", line 184, in init
    await self.file2jobs(file),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/project/brant/db-home/miniconda/envs/phyloacc/lib/python3.12/site-packages/snakemake/dag.py", line 2199, in file2jobs
    raise MissingRuleException(targetfile)
snakemake.exceptions.MissingRuleException: No rule to produce --cluster-sync=mkdir -p /ddnA/work/brant/phyloacc-test/phyloacc-outgroup/phyloacc-job-files/slurm-logs/{rule}/ && sbatch --wait --partition={resources.partition} --nodes={resources.nodes} --cpus-per-task={resources.cpus} --job-name={rule}-{wildcards} --mem={resources.mem} --time={resources.time} --output=/ddnA/work/brant/phyloacc-test/phyloacc-outgroup/phyloacc-job-files/slurm-logs/{rule}/{rule}-{wildcards}-%j.out (if you use input functions make sure that they don't raise unexpected exceptions).

Downgrading to an earlier version of snakemake (in phyloacc conda environment):

conda install snakemake=7.32.4

is a workaround (I believe because the way to run slurm jobs changed in snakemake 8). After making that downgrade, it seems things are functioning as expected (at least --dryrun succeeds).

It seems like pinning the snakemake version to 7.32.4 should resolve the issue for the time being.

[gwct]

Hi Brant, Thanks for pointing this out. I thought I had restricted the snakemake version to 7 since I haven't had time to update the profile yet, but I guess not. It is one of my top priorities, however I'm currently on parental leave for a couple more weeks so I'm not sure if I'll be able to get to it for a while. -Gregg

[brantfaircloth]

Hi Gregg,

Right on - certainly not a giant rush. Just wanted to report these while I was working with the program.

Thanks for your work on it!

-b

[gwct]

I've completed the update to snakemake 8 in v2.4.1. Please let me know if any other issues arise with this... the snakemake 8 docs are far from clear so it was difficult to tell if I caught everything that needed to be changed.