search

openclimatefix / nwp-consumer

Microservice for consuming NWP data.
9 stars 3 forks source link

Enable option to not store raw files with storer #106

Closed devsjc closed 7 months ago

devsjc commented 7 months ago

Saving raw files to huggingface does in fact result in us hitting the commit limit for huggingface. As such, modify the NWP Consumer so that we can have the option not to save raw files to the storer.

devsjc commented 7 months ago

Decided instead of creating an optional storage parameter, a quicker to implement (and perhaps more versatile option) is instead to enable a StorageInterface per Raw and Zarr data. These would default to being the same.