search

earthobservations / wetterdienst

Open weather data for humans.
https://wetterdienst.readthedocs.io/
MIT License
347 stars 54 forks source link

FileNotFoundError when trying to get historical values #549

Closed GezaBoi closed 2 years ago

GezaBoi commented 2 years ago

Describe the bug I can poll the available stations for the historical data, but when I am trying to get the actual measurements I get a file not found error.

To Reproduce The examples in the example forlder and in the documentation all throw errors for me. If I run:

https://github.com/earthobservations/wetterdienst/blob/main/example/observations_sql.py

I get: FileNotFoundError: ['https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/hourly/air_temperature/historical/']

https://github.com/earthobservations/wetterdienst/blob/main/example/observations_stations.py throws an FileNotFoundError as well.

Also https://github.com/earthobservations/wetterdienst/blob/main/example/mosmix_forecasts.py is a different endpoint but also broken. This throws: IndexError: Unable to find LATEST file within https://opendata.dwd.de/weather/local_forecasts/mos/MOSMIX_L/single_stations/01001/kml

Expected behavior Returning the data, not throwing an error.

Desktop (please complete the following information):

Additional context Since I am not able to get a single example to work there seems to be something completly broken on my end or the file paths have changed on the dwd server. I hope you can help me out!

Thank you so much for providing this library! I need the data for my research and the data/information was really hard to understand on the dwd server.

gutzbenj commented 2 years ago

Dear @GezaBoi ,

thanks for writing in! I checked your errors and can report the following:

I would at this moment guess that this was a temporary issue due to the update procedure by DWD that may have led to the vanishing of those files. However I am sceptical since you had run into two different issues of such kind.

On the other hand if there was no strange behavior from the server side, I'd bet that the cache was being built during such a moment where no latest and stations file was listed on the server, but that issue should be gone by now. Can you rerun both examples and tell me the result? Otherwise I believe this is hard to debug.

By the way: Do you run any quality checks on the values?

GezaBoi commented 2 years ago

Dear @gutzbenj now the errors are also fixed on my end without any change in setup. I did test yesterday over the span of a few hours thinking it might be server side but I did not expect the update process to take that long.

I am currently in the phase of just looking at the data and trying to figure out which one I will use. So there are no quality checks there yet. But I am definetly planning on checking and preprocessing the data. I am trying to forecast photovoltaik production and want to use weather data as one input for the model.

One more question: is there historical data for the Mosmix forecasts? I think that would be the most useful for me but I can also Imagine that it is not available due to the amount of data.

Thanks again for your help and work!

gutzbenj commented 2 years ago

Dear @GezaBoi ,

nice to hear. I would leave it that way believing this was an artefact created due to the update procedure of the DWD. However if there will be another higgup we should come back and try and see how to overcome this problem.

That's a really nice project there! And there also seems to be a lot of competition going on. Also there are some nice companies that work in this field! I've checked for the archiving of Mosmix forecasts however it seems that there's no such thing at least for the public. Maybe you can contact the DWD directly and ask for archived forecasts!

Do you already have an estimate on how many pairs of values you need for your training? I'd suggest you start collecting data as early as possible so that you can decide later on whether to take into account what data...

For quality checks etc the documents provided by ECAD may be a good starting point: https://www.ecad.eu/publications/index.php

BuntBaum commented 11 months ago

Hi Everyone,

While routinely crawling the MOSMIX data, as of today, I've encountered a FileNotFoundError.

Error Message: FileNotFoundError: ['https://www.dwd.de/DE/leistungen/met_verfahren_mosmix/mosmix_stationskatalog.cfg?view=nasPublication']

However, when I manually access the URL, the file seems to be in order.

Restarting the job resolved the issue this morning, but the error reappeared this afternoon.

Best regards, BB