Data ingestor: convert CBAM short-term forecast into zarr

kartoza / tomorrownow_gap

https://kartoza.github.io/tomorrownow_gap/

0 stars 0 forks source link

Data ingestor: convert CBAM short-term forecast into zarr #48

Open danangmassandy opened 3 months ago

danangmassandy commented 3 months ago

For this ticket, it is still pending confirmation:

Whether we fetch the data from Tomorrow.io API for the whole region, or
There will be netcdf files like Reanalysis data

If the source is Tomorrow.io API, then we also need a list of variables that we want to fetch.

gubuntu commented 3 months ago

you will be fetching now-to-two-week forecasts from the API for the whole area every day and writing to zarr

danangmassandy commented 2 months ago

hi @gubuntu, can we overwrite the past forecast data when we fetch the forecast in the next day? For example, today GAP will fetch data from 07-08-2024 to 21-08-2024, then tomorrow GAP is going to fetch data from 08-08-2024 to 22-08-2024. So, there will be existing forecast data on dates 08-08-2024 until 21-08-2024, this can be overwritten with latest forecast, does this seem correct?

gubuntu commented 2 months ago

no don't overwrite. Long-term we definitely need to keep that day's forecast for each day you fetch. But for the meantime Philip requested that we store the full forecast each day

danangmassandy commented 2 months ago

[ ] From GAP Area of Interest, split into smaller area that Tomorrow.io API accepts the request
[ ] Fetch the forecast data from Tomorrow.io API for each smaller area
[ ] Store the response into a zarr file

danangmassandy commented 1 month ago

Update 29-08-2024:

Brian will confirm to Tomorrow.io whether they can provide short-term forecast in netcdf file