Support Snakemake version 8 across core pathogen workflows

nextstrain / public

repo to house broad planning issues and other cross-repo concerns; see also https://github.com/nextstrain/private

0 stars 0 forks source link

Support Snakemake version 8 across core pathogen workflows #4

Open victorlin opened 4 months ago

victorlin commented 4 months ago

Currently, our managed runtimes come with Snakemake version 7 so our core pathogen workflows have been written to work with that.

Snakemake version 8 was released in December 2023.

It's known to not work in ncov, reported by an external user. Not sure about other repos as we haven't tried yet.

Tasks

[x] create docker-base image to simplify testing of compatibility: docker.io/nextstrain/base:branch-victorlin-install-snakemake-8
[ ] Update ncov to be compatible with version 8
[ ] https://github.com/nextstrain/ncov-ingest/issues/452
[ ] add other pathogen repos
[ ] Bump Snakemake version in managed runtimes

corneliusroemer commented 3 months ago

Thanks @victorlin!

How about we create a snakemake 8 image and do test runs with it in repos? Then we'll know what needs to change.

I've added it as a task

victorlin commented 3 months ago

Sure! That seems pretty low effort. I'll draft a docker-base PR now.

victorlin commented 3 months ago

Test image is available as nextstrain/base:branch-victorlin-install-snakemake-8. It's not possible to use the proposed simple name nextstrain/base:snakemake8 without manually pushing another tag which could get out of sync with PR. The next simplest would be nextstrain/base:branch-snakemake8. Mine is lengthier due to branch naming habits, but copy/pasting this value is probably a good idea anyways.

victorlin commented 3 months ago

I'm wondering how we would follow through with a big change like this. Is it possible to update to be Snakemake 8 compatible and still work on Snakemake 7? Or are the two mutually exclusive (this is my impression)? Currently all our pathogen repos pull the latest Docker image, so this would require either pinning specific runtime versions or a well-coordinated update.

genehack commented 3 months ago

I'm wondering how we would follow through with a big change like this. Is it possible to update to be Snakemake 8 compatible and still work on Snakemake 7? Or are the two mutually exclusive (this is my impression)?

Seems like confirming whether or not 8 is backcompat with 7 would be a first step? Has anybody built a list of changes? (That would also help with the inevitable "why are we doing this" question I expect.)

Currently all our pathogen repos pull the latest Docker image, so this would require either pinning specific runtime versions or a well-coordinated update.

This feels like a small piece of a larger conversation about long-term pathogen repo maintenance practices.

tsibley commented 3 months ago

This feels like a small piece of a larger conversation about long-term pathogen repo maintenance practices.

This, for sure. While the current practices worked well for a smaller number of production pathogens, they won't necessarily work as well for the larger number we're growing to.