hubverse-org / hubverse-transform

Data transform functions for hubverse model-output files
MIT License
1 stars 0 forks source link

Add a utility script that re-triggers hubverse-transform-model-output_lambda #28

Closed bsweger closed 3 months ago

bsweger commented 3 months ago

Resolves #27

First iteration of a way to re-trigger the lambda function that fires when new data is added to the raw/model-output/ folder of a hub's S3 bucket.

To try this as part of code review:

  1. Make sure you have write access to S3 buckets in the Hubverse AWS account
  2. Pull this branch and python faas/lambda_retrigger_model_output_add.py bsweger-flusight-forecast

I ran this against the bsweger-flusight-forecast test hub and confirmed that:

➜ python faas/lambda_retrigger_model_output_add.py bsweger-flusight-forecast
Updating metadata for all files in bsweger-flusight-forecast/raw/model-output/

Processing raw/model-output/CADPH-FluCAT_Ensemble/2023-10-14-CADPH-FluCAT_Ensemble.csv
Processing raw/model-output/CADPH-FluCAT_Ensemble/2023-10-21-CADPH-FluCAT_Ensemble.csv
Processing raw/model-output/CADPH-FluCAT_Ensemble/2023-10-28-CADPH-FluCAT_Ensemble.csv
[snip]
Updated metadata for 1129 files in bsweger-flusight-forecast/raw/model-output/