hubverse-org / hubverse-transform

Data transform functions for hubverse model-output files
MIT License
1 stars 0 forks source link

Add a hubverse-transform method to response to S3 object deletion triggers #29

Open bsweger opened 2 months ago

bsweger commented 2 months ago

Background

This package's ModelOutputTransform class has a method to transform a hub's incoming model-output files to Hubverse format. ModelOutputTransform is bundled into an AWS Lambda package that is triggered whenever files are added/updated in the raw/model-output folder of a hub's S3 bucket.

The next step is ensuring that ModelOutputTransform also contains a function to delete files in a hub's S3 model-output directory if the file is removed from the hub.

Definition of done

This issue to to create the

bsweger commented 2 months ago

Discussed this with @matthewcornell, and we decided that the delete method does indeed belong in the ModelOutputTransform class (instead of a new class) because there's some shared code, and the delete operation is simple.

We did decide to split the new delete code into separate unit test files, however, since the existing test_model_output.py is getting long-ish.