Adding road weather data from DWD

meteoDaniel commented 2 years ago

Is your feature request related to a problem? Please describe. No

Describe the solution you'd like A bufr reading interface to download road weather observations and parse them with odbufr to return a human readable dataframe

Describe alternatives you've considered No alternativs

Additional context No add. context.

I would like to implement that feature, can you give me some instructtions where I should place the feature and which functions I should use?

gutzbenj commented 2 years ago

Dear @meteoDaniel ,

can you provide us some more information about the data? Is it similar to the historical data like does it have a fixed set of stations? Then I'd recommend implementing the API similar to other data with given core classes and enums.

meteoDaniel commented 2 years ago

You can find the data here. So for a good starting point I can copy e.g. provider/dwd/radar directory and built scripts for access, api, metadata and maybe some dataset specific stuff like sites ?

gutzbenj commented 2 years ago

After looking at the data I'd recommend the following:

Use the classes from [1]. Those classes have some attributes that have to be defined but that should (hopefully) be self explanatory.
Work along the classes that exist for DwdObservationRequest
Define enumerations similar to DWD Observations e.g. DwdRoadParameter, DwdRoadParameterUnit, ...
Use parameter names as defined in https://github.com/earthobservations/wetterdienst/issues/502 which I will merge proabably later this day.
Place the files at wetterdienst.provider.dwd.road

I am aware that there's currently no real guide for the implementation and also that some enumerations may be simplified or simply omitted but for the moment you should be able to get 90 % working. Please come back to me or open a PR draft so that I can guide you through.

Btw is there any list for the available stations?

[1] https://github.com/earthobservations/wetterdienst/tree/main/wetterdienst/core/scalar

amotl commented 2 years ago

Hi Daniel,

kudos already for aiming to bring in those data. I am also looking forward to bring in the data from [1], see also #342.

Regarding the data from [2], will you target only the LATEST files in the respective subdirectories? I believe it will be sufficient and will make the addressing a lot easier.

With kind regards, Andreas.

[1] https://opendata.dwd.de/weather/alerts/ [2] https://opendata.dwd.de/weather/weather_reports/road_weather_stations/

gutzbenj commented 2 years ago

Dear @amotl ,

I believe it is important to address all available files in case of a blackout on one of the two ends. This is to keep all available data at your own end so that you can decide later what to do with it but firstly guarantee its preservation, even when your request fails once in a while.

meteoDaniel commented 2 years ago

Hi @amotl,

what do you think about to download the n latest files from the directory? This would be much easier to build an expression to address every single file ? But for sure my primary goal is to download latest and all other stuff will be features coming the can be added later on.

And thanks to @gutzbenj for your guide. I think this will help me a lot.

gutzbenj commented 2 years ago

Hi @amotl,

what do you think about to download the n latest files from the directory? This would be much easier to build an expression to address every single file ? But for sure my primary goal is to download latest and all other stuff will be features coming the can be added later on.

And thanks to @gutzbenj for your guide. I think this will help me a lot.

Dear @meteoDaniel ,

we already have a similar situation for DWD Mosmix, so I believe it wouldn't be difficult to copy that behaviour!

meteoDaniel commented 2 years ago

I am struggling with the development environment again .

Calling:

poetry install --extras=http --extras=sql --extras=export --extras=ui

Yield to:

    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/daniel/projects/forecast-system/wetterdienst/setup.py'"'"'; __file__='"'"'/home/daniel/projects/forecast-system/wetterdienst/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps --user --prefix=
         cwd: /home/daniel/projects/forecast-system/wetterdienst/
    Complete output (3 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'setuptools'
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/daniel/projects/forecast-system/wetterdienst/setup.py'"'"'; __file__='"'"'/home/daniel/projects/forecast-system/wetterdienst/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps --user --prefix= Check the logs for full command output.

But setuptools is installed.

Using the alternative way with poetry:

daniel@daniel-ThinkPad-E490:~/projects/forecast-system/wetterdienst$ poetry install --extras=http --extras=sql --extras=export --extras=ui
Installing dependencies from lock file

  ValueError

  Extra [http] is not specified.

  at ~/.poetry/lib/poetry/installation/installer.py:253 in _do_install
      249│                 )
      250│ 
      251│             for extra in self._extras:
      252│                 if extra not in self._locker.lock_data.get("extras", {}):
    → 253│                     raise ValueError("Extra [{}] is not specified.".format(extra))
      254│ 
      255│             # If we are installing from lock
      256│             # Filter the operations by comparing it with what is
      257│             # currently installed

amotl commented 2 years ago

Hi Daniel,

can you make any progress when using the information from #421, which we compiled the other day you have been tripped by the same thing?

--extras=http has been renamed to --extras=restapi. Is there any spot we overlooked to reflect that within the documentation? From my perspective, the walkthrough at [1] seems to be up to speed. As outlined at https://github.com/earthobservations/wetterdienst/issues/421#issuecomment-824765041, e06c60e fixed it.
If Poetry does not work for you, you can also use pip install --requirement=requirements.txt and python setup.py develop to bootstrap a development environment into a Python virtualenv like outlined at https://github.com/earthobservations/wetterdienst/issues/421#issuecomment-824893583.

With kind regards, Andreas.

[1] https://wetterdienst.readthedocs.io/en/latest/development.html

meteoDaniel commented 2 years ago

@amotl thanks. These 3 lines should be added to the README

poetry install --extras=sql --extras=export --extras=restapi --extras=explorer
poetry shell
poe test

This works for me. Thanks. I have worked on first functions.

gutzbenj commented 2 years ago

Dear @meteoDaniel ,

are there any updates on this project or should I take on the work you started and try to finish it?

meteoDaniel commented 2 years ago

Dear @gutzbenj , the work gots stucked during integration into the unified wetterdienst class and API scheme. So it would be great If you can finish the work. Unfortunately my life kicked in and I was not able to work on such a nice project. Thanks for the offer.

gutzbenj commented 1 year ago

This has actually been implemented with #957

earthobservations / wetterdienst

Adding road weather data from DWD #518