Closed j23414 closed 3 months ago
The purpose of this was some early exploration of https://github.com/nextstrain/dengue/pull/10/commits/dafdd12c2bc8117469560d058a93a0e233df0d1f to support placing workflows in subdirectories.
Closing this, since we ended up going a different direction: https://github.com/nextstrain/cli/releases/tag/8.2.0.
Description of proposed changes
Explore using Snakemake modules for ingesting data into the Nextstrain build system. This PR would merge into the
new_ingest
branch and continue thedengue ingest
work.The running behavior is:
data
directory exists, proceed with build as normal.s3_src
is pointing to a pre-cleaned dengue dataset, fetch dataset and proceed as normal.In order to implement the above, we are using Snakemake modules¹ which allows us to define a
prefix
for the ingest input/output paths. However, these do not get applied to the scripts called withinshelL
. So we are using this workaround² to add a base dir to the scripts.¹ https://snakemake.readthedocs.io/en/stable/snakefiles/modularization.html#modules ² https://stackoverflow.com/a/66890412
Testing
Can run a manual check with the following commands:
Although the following should still work too: