openclimatefix / Satip

Satip contains the code necessary for retrieving, transforming and storing EUMETSAT data
https://satip.readthedocs.io/
MIT License
41 stars 28 forks source link

Add Ability to save to different sinks in app.py #230 #232

Closed vikasgrewal16 closed 6 months ago

vikasgrewal16 commented 6 months ago

Pull Request

Description

i have updated the code that updates the function 'save_to_zarr_to_s3' to accept a parameter indicating the desired backend. I use fsspec to create a file system instance based on the specified backend and save the dataset accordingly.

Fixes #

save_to_zarr function now accepts a backend parameter, which defaults to 's3'. The function checks if the specified backend is supported, and then it uses fsspec to create the appropriate file system instance (fs). The dataset is then saved to Zarr using this file system instance.

Example usage: save_to_zarr(my_dataset, 's3://my-bucket/my-data.zarr', backend='s3') save_to_zarr(my_dataset, 'file:///path/to/local/data.zarr', backend='file') save_to_zarr(my_dataset, 'gcs://my-bucket/my-data.zarr', backend='gcs')

If your changes affect data processing, have you plotted any changes? i.e. have you done a quick sanity check?

Checklist: