search

stephan192 / dwdwfsapi

Python client to retrieve data provided by DWD via their geoserver WFS API
MIT License
6 stars 4 forks source link

Stopped working #2

Closed cdruee closed 3 years ago

cdruee commented 3 years ago

The quickstart example returns nothing.

If the cell Name is given instead of the numeric id, the ows Feature request returns but does not contain a property "WARNCELLID" and line 245 raises: Illegal property name: WARNCELLID for feature type dwd:Warngebiete_Gemeinden

WARNCELLID seems to be no longer supported

Instead AGS (digits 2-9 of warncell id) TYPE (first digit of warncell id) must be used, apparently

martin3000 commented 3 years ago

In the query dwd:Warngebiete_Gemeinden/Kreise, the WARNCELLID is no longer returned. Only "FID" and "WARNCENTER" is returned. https://maps.dwd.de/geoserver/dwd/ows?service=WFS&version=1.1.0&request=GetFeature&typeName=dwd%3AWarngebiete_Kreise&CQL_FILTER=FID%3D%27234%27

stephan192 commented 3 years ago

The problem is that DWD did a lot of, in my opinion bad, changes.

I'm currently testing a version which only supports Landkreise, due to the given constrains.

martin3000 commented 3 years ago

Are these changes documented? Here https://www.dwd.de/DE/leistungen/warnungen/aenderungen_herbst2021.html?nn=16102&lsbId=397560 they write about upcoming changes in autumn and the warncellid. This does not sound like they want to drop the warncellid. Maybe it is only bug in the DWD software?

martin3000 commented 3 years ago

I cannot query the Warnungen by FID: https://maps.dwd.de/geoserver/dwd/ows?service=WFS&version=2.0.0&request=GetFeature&typeName=dwd%3AWarnungen_Landkreise&CQL_FILTER=FID%3D304&OutputFormat=application/json

But what works is NAME and AREADESC: https://maps.dwd.de/geoserver/dwd/ows?service=WFS&version=1.1.0&request=GetFeature&typeName=dwd%3AWarngebiete_Kreise&CQL_FILTER=NAME%3D%27Saale-Holzland-Kreis%27

https://maps.dwd.de/geoserver/dwd/ows?service=WFS&version=2.0.0&request=GetFeature&typeName=dwd:Warnungen_Landkreise&outputFormat=application/json&CQL_FILTER=AREADESC%3D%27Saale-Holzland-Kreis%27

stephan192 commented 3 years ago

That what i meant with:

The FID attribute is not part of the Warning info, so search must be done by name.

My current status:

Still unresolved:

I think DWD messed up a lot. Should i publish an update or should we wait for DWD to fix their error?

martin3000 commented 3 years ago

As of https://geoportal.sachsen.de/cps/capabilities_viewer.html?url=https%3A%2F%2Fmaps.dwd.de%2Fgeoserver%2Fwfs%3FSERVICE%3DWFS%26VERSION%3D2.0.0%26REQUEST%3DGetCapabilities , the feature is dwd:Warnungen_Gemeinden_vereinigt :

https://maps.dwd.de/geoserver/dwd/ows?service=WFS&version=2.0.0&request=GetFeature&typeName=dwd:Warnungen_Gemeinden_vereinigt&OutputFormat=application/json&CQL_FILTER=IDENTIFIER%3D%272.49.0.1.276.0.DWD.PVW.1622965080000.c104d27b-4dd0-4bd0-af7b-714f2f09801b.DEU%27

martin3000 commented 3 years ago

It is very strange that there is no blog post about that changes.

stephan192 commented 3 years ago

The table dwd:Warnungen_Gemeinden_vereinigt is not new. It exists since i started the development of this api. The problem with dwd:Warnungen_Gemeinden_vereinigt is that the warnings in this table do not contain any information to link the warnings to regions defined in dwd:Warngebiete_Gemeinden.

martin3000 commented 3 years ago

In the capabilities document, Warnungen_Gemeinden is not defined. Maybe it was not official? Warnungen_Landkreise can be directly accessed by AREADESC filter: CQL_FILTER=AREADESC:'Saale-Holzland-Kreis'

stephan192 commented 3 years ago

But then the table dwd:Warngebiete_Gemeinden doesn't make sense. I think i will make a fix, which works for everyting execpt "Gemeinden".

martin3000 commented 3 years ago

On my machine, I changed the code in waetherwarnings.py and this works for Kreise: ` 

def __generate_query(self, identifier):

    self.__query = {

                    "typeName": "dwd:Warnungen_Landkreise"
                }

    self.__query[
                   "CQL_FILTER"
                ] = f"AREADESC='{identifier}'"

`

from dwdwfsapi import DwdWeatherWarningsAPI dwd = DwdWeatherWarningsAPI("Saale-Holzland-Kreis")

cdruee commented 3 years ago

Wow, what a lively community in this project! I have poked a lot around in den dwd geoserver and it seems they erased WARNCELLID from all layers. And not only the layer dwd:Warnungen_Gemeinden is missin on the geoserver, but also, in the json-file for use in the WarnApp - where WARNCELLID is still the dictionary key - no WARNCELLIDs starting with "8" for type "Gemeinde" (municipality) are currently reported. I seem the have given up reporting my individual municipalities "Gemeinde", alt all. I have sent a request to opendata@dwd.de ans asked them to clear up, whether this is a malfunction, an error or an intentional change (which is not yet reflected in their documentation)

BTW: Still having the layer "dwd:Warngebiete_Gemeinden" makes sense, since one can match the individual municipality area polyons ("Gemeinde") to the joint municipality warning areas ("Gemeinden_vereinigt") to determine if a warning is issued for an individual municipality. This is a standard operation for a geospacial information system (GIS) and easy in theory but would be an completely unnecessary complication for any project like this.

stephan192 commented 3 years ago

I've just released v1.0.4 of the api to PyPi. It works with "Landkreise". I'll create a pull-request on HA.

@CDMIUB I'll leave this issue open until you report back what DWD answered to your email.

TechHummel commented 3 years ago

Hey there, I got a reply to my email (sent it around the same time):

"es handelt sich wirklich um einen Fehler, der seit letztem Donnerstag auftritt. Der Geoserver liefert die WanrCell-IDs bei diesem Layer nicht mehr auf. Auch unser Feuerwehrinformationssystem ist davon betroffen. Leider ist die Ursache noch nicht gefunden.

Ich hoffe, dass wir im Laufe des Tages das Problem lösen können." (It's a bug on dwd's side which they are hoping to fix today).

Thanks for your effort though! I guess the change should be rolledback.

Edit: I also found this RSS feed which should inform about any upcoming APi changes: https://www.dwd.de/SiteGlobals/Functions/RSSFeed/DE/opendata/opendata_lsb.xml?nn=16102

stephan192 commented 3 years ago

Now they fixed their error. All work done for nothing. But both versions (1.0.3 and 1.0.4) work with the fixed api. 1.0.4 creates higher loads on the dwd servers but may be more robust.

nibblerrick commented 3 years ago

Although all the work was in the end "fürn Ar..." a big thank you to your guys for going to fix it so quickly!