New ingest one snakefile

[j23414]

Description of proposed changes

Explore using Snakemake modules for ingesting data into the Nextstrain build system. This PR would merge into the new_ingest branch and continue the dengue ingest work.

The running behavior is:

• If data directory exists, proceed with build as normal.
• If s3_src is pointing to a pre-cleaned dengue dataset, fetch dataset and proceed as normal.
• Otherwise, fetch dataset using ingest, clean it, and proceed as normal.

In order to implement the above, we are using Snakemake modules¹ which allows us to define a prefix for the ingest input/output paths. However, these do not get applied to the scripts called within shelL . So we are using this workaround² to add a base dir to the scripts.

¹ https://snakemake.readthedocs.io/en/stable/snakefiles/modularization.html#modules ² https://stackoverflow.com/a/66890412

Testing

• [ ] Checks pass

Can run a manual check with the following commands:

git clone https://github.com/nextstrain/dengue.git
cd dengue
git checkout new_ingest_one_snakefile 

# Runs both ingest and build
nextstrain build .

Although the following should still work too:

cd ingest
rm -rf data logs  # cleanup any previously generated files if there
nextstrain build .

[j23414]

The purpose of this was some early exploration of https://github.com/nextstrain/dengue/pull/10/commits/dafdd12c2bc8117469560d058a93a0e233df0d1f to support placing workflows in subdirectories.

Closing this, since we ended up going a different direction: https://github.com/nextstrain/cli/releases/tag/8.2.0.