openclimatefix / nwp-consumer

Microservice for consuming NWP data.

9 stars 3 forks source link

Long range ECMWF india data #138

Closed devsjc closed 2 months ago

devsjc commented 3 months ago

In order to do a 7 day forecast for RUVNL, as well as the Satellite (https://github.com/openclimatefix/Satip/issues/223) and Meteomatics () data, we need India ECMWF data to go out to a larger horizon. Currently it is limited to only go as far as the hourly incrementing steps, i.e. 90. We want to enable pulling up to 192 hours. This covers step increments of 1, 3, and 6 hours. We want 2022-2024 initially.

[Part of RUVNL]

devsjc commented 3 months ago

Closed with https://github.com/openclimatefix/nwp-consumer/pull/139

devsjc commented 2 months ago

Whilst MARS listing requests were smart enough to handle requesting steps that did not exist upstream, MARS retrival errors out with the expected number of fields differring. As such, a safety net needs to be put in place to ensure that only exactly what is being pulled from MARS gets requested to begin with, even if it differs from what we are asking for. See #140

devsjc commented 2 months ago

Backfill in progress.

devsjc commented 2 months ago

19-04-2024

2022 fully downloaded. (50% done)

devsjc commented 2 months ago

24-04-2024

Still at 50%. I realised that the UK backfill (https://github.com/openclimatefix/dagster-dags/issues/92) was now taking priority due to 2023 india being its own backfill with an incorrect priority tag (priority instead of dagster/priority). I have recreated the backfill with the corrected priority so that it takes precedence from the UK backfill.

devsjc commented 2 months ago

Moving the backfill part of the process to an issue in dagster-dags: see https://github.com/openclimatefix/dagster-dags/issues/103.