openclimatefix / nwp-consumer

Microservice for consuming NWP data.
9 stars 3 forks source link

ECMWF EPS Mean/std Archive #190

Open devsjc opened 1 month ago

devsjc commented 1 month ago

As part of the GDM work, we require an archive of ECMWF Ensemble data - not full members, but rather mean and standard deviation values. The Operational Archive contains within it a data stream specific to EPS (Ensemble Prediction System). Here follows a summary of the data quality we can expect from the different products available on MARS for EPS:

Derived Probability Products: Ensemble Mean/Std

Derived Probability Products: Time-averaged Ensemble Mean/Std

Neither of these is completely ideal! There is also an ENSEMBLES product available but none of the data on there is near the 2020s. @Sukh-P, and @peterdudfield, which type do you think suits us better? And @Sukh-P, what would you want ideally pulling in terms of region, steps, parameters, years? Or, do you know of any other MARS product that you had in mind with the ensemble data work?

Sukh-P commented 1 month ago

Thanks for this @devsjc! Is the ENSEMBLES product you mentioned without archive data from 2020 the same as the ENS one https://confluence.ecmwf.int/display/FUG/Section+2.1.2.1+ENS+-+Ensemble+Forecasts?

devsjc commented 1 month ago

I think the ENSEMBLES project is something else entirely and not particularly useful to us. However I am a bit confused now: the links on that page take you to a real-time order page where set iii: ENS is clearly defined as the 15- day ensemble model; I've been looking at the archive datasets on MARS, none of which seem to reference anything called ENS, only EPS...

Sukh-P commented 1 month ago

Yeah I also had a look and I am quite confused, it seems like the ENS doesn't have an archive available but maybe worth contacting ECMWF about this to make sure?

devsjc commented 4 weeks ago

Yes, I don't think they do.

devsjc commented 4 weeks ago

In discussion with Sukh:

It would also be beneficial to have probability boundaries if possible (P10, P25, P50(median), P75. P90) in an ens_stat dimension.

devsjc commented 4 days ago

I believe GDM are referring to use of the top option specified in the first comment, so that's what I'll pull. It does not, unfortunately, seem to have probability boundaries - that is in a seperate archive with much lower temporal resolution. I'll get what I can for now.