Add support for forecast information

amotl commented 4 years ago

Introduction

We would like to add support to acquire forecast information from DWD, for both MOSMIX-L and MOSMIX-S.

Background

@drmrbrewer and @onlygecko already asked for this within https://github.com/panodata/dwdweather2/issues/2.

Thoughts

Together with @jlewis91, we already planned to merge dwdweather2 with dwdbulk somehow, but now we should consider to add missing functionality here.

@jlewis91' convert_xml_to_pandas method from forecasts.py might be used 1:1 already. Note that this one has originally been written to read MOSMIX-S only.

meteoDaniel commented 4 years ago

@amotl I have built a parser but it is not advanced in the moment. I can provide the corresponding ode and we can build up a more powerful parser.

amotl commented 4 years ago

@FL550 just added another implementation to grok MOSMIX-L through dwdforecast.py.

amotl commented 4 years ago

Comparison

What I like from @jlewis91' implementation forecasts.py is that it already uses Pandas. However, we will have to expand it to also parse MOSMIX-L, currently it only does MOSMIX-S. Also, it extracts the list of .kml files into a local directory. We should do that in-memory.

What I like from @FL550's implementation dwdforecast.py is that is does everything in-memory already and that it maps the technical meteorological element names to more meaningful field names (see #54). Also, it features a mapping to resolve the weather condition codes to meaningful textual labels.

Both implementations use lxml.etree for parsing the XML.

Resources

The KML format description is available at Data Format of MOSMIX Forecasts.
A number of elements from the MOSMIX-Elemente table is already accessible in machine-readable format through MetElementDefinition.xml.
Mapping information for the numerical weather prediction models - the "significant weather condition" code from field ww - is available at Weather interpretation for the ICON/ICON-EU and COSMO-D2 models as well as Used weather condition codes in the MOSMIX-forecasts. This topic is also discussed within https://github.com/jdemaeyer/brightsky/issues/85.

amotl commented 4 years ago

Hi there,

within #114, I've added a preliminary spike the other day, mostly based on the original dwdbulk::forecasts implementation by @jlewis91 (thanks again!).

I believe we should use this issue here to discuss the future evolution of this feature, even beyond the lifetime of #114.

Backlog

General

[x] Add software tests.
[x] Update documentation.
[x] Adapt get_nearby_stations for MOSMIX.
[x] Integrate into command line interface.
[x] Integrate into HTTP API.
[x] Add docstrings.

Metadata

Adding parameters, column names and such into the metadata/knowledgebase subsystem at wetterdienst/constants would also be awesome if that actually makes sense. I see these details here:

[x] The list of obtained parameters are only strings for now and not Enum-based yet. We might want to map this for both MOSMIX-L and MOSMIX-S, see mosmix.py#L303 vs. mosmix.py#L310.
[x] Output column names should be mapped to sensible Enum-based symbols as well as meaningful humanized English names, see #54. When implementing this, we might want to revisit https://github.com/earthobservations/wetterdienst/issues/54#issuecomment-654353359, where I referenced some work from @meteoDaniel (thanks!) which might help us here.

Borrowing from GribMagic

@meteoDaniel also added some humanized labels through unified_forecast_variables.py, thanks already! Wetterdienst might also draw some inspiration from this.

Remark: For observation data, we added retrieving metadata about column names from PDF files on the DWD data repository as outlined within #112 through #199.
[x] Things like coerce_columns are still in their infancy.

New feature: Station concordance

Now, @gutzbenj asked at https://github.com/earthobservations/wetterdienst/pull/114#issuecomment-690059461:

Also as we have talked about the station ids, it would be awesome to have a translating system from regular DWD station ids to MOSMIX station ids so that one could use the same id for historical observations and MOSMIX forecasts.

That would definitively be awesome. I believe @wetterfrosch did this concordance thing in a hands-on manner while he implemented the Panodata DWD Meteogram. He might be interested in helping us to bring this subsystem into Python land.

With kind regards, Andreas.

earthobservations / wetterdienst