search

hubverse-org / hubverse-transform

Data transform functions for hubverse model-output files
MIT License
1 stars 0 forks source link

Throw warnings when the incoming file is not valid #16

Closed bsweger closed 5 months ago

bsweger commented 5 months ago

We've recently encountered two situations when testing the transform lambdas that should not raise errors:

  1. someone manually creates a folder in a hub's raw/ S3 folder, which triggers an S3 creation event
  2. model-output folders contain an .md or .txt file that should not be included in the transform process

Raising a UserWarning allows the invoking process (in our case, the lambda handler) to handle these scenarios as they see fit.

bsweger commented 5 months ago

It would also be nice to tell lambda to skip the retries when it encounters one of these warnings, but it happens infrequently enough that we can punt on that.

bsweger commented 5 months ago

Opened an issue to automatically include the test coverage report on a PR, but in the meantime:

image