search

noi-techpark / bdp-commons

GNU Affero General Public License v3.0
2 stars 12 forks source link

As an Open Data Hub project manager I would like to integrate static e-charging stations data of DZT (Bayern / Tyrol) #609

Open rcavaliere opened 1 year ago

rcavaliere commented 1 year ago

Information about Data Provider: https://www.germany.travel/de/ueber-uns/die-dzt.html We still have little documentation on how to access the data, but @mrabans has already prepared some call examples on we could get the charging information.

DZT-echarging-single-station-request.md DZT-echarging-single-station-response.pdf DZT-echarging-all-stations-request.md DZT-echarging-all-stations-response.pdf

Tasks to do:

Deadline: possibly before the Open Data Hub Day, so to communicate this international cooperation

Integration Specification: 230727_SpecificheIntegrazione_NOI_v1.2.pdf

clezag commented 1 year ago

This looks to be the applied data structure of the echarging stations: https://semantify.it/ds/E85TgOxMg with the plugs: https://semantify.it/ds/E85TgOxMg?path=odta:hasCharger-odta:Charger

It appears to have all the information that the STA points have, and more, but it remains to be seen how much of the data is actually there once we call the API (aside from "name" all fields seem optional according to this specification). The few stations I called for example did not return any plug information. Pretty much only address and location

I might be missing something as I'm admittedly unfamiliar with this schema.org / knowledge graph stuff.

@mrabans where did you get the information for your initial calls from? Is there any other documentation at all?

mrabans commented 1 year ago

@clezag this is what Elias Kärle wrote by mail. He sent two examples, so I took them, tested them and sent Roberto the curl commands and the responses in order to allow him to have a quick look at it too.

Would you like to join our call on Monday 4pm, so Elias can directly answer to all your questions on the data structure, more documentation, filters etc.

rcavaliere commented 1 year ago

@clezag we will receive some documentation, in the meantime I am trying to draft the usual integration specification document

rcavaliere commented 1 year ago

@clezag @mrabans @ohnewein @sseppi I have checked this dataset more in detail (see draft integration specification document) . I have to say, the quantity of detail we have is very poor. Actually a charging station is modeled just a POI, we don't have any details e.g. if the access type is private / public, which are the supported connectors, etc. I have contacted our reference contact for more clarifications on this. If the data is only this one, we have to discuss if it is worth to integrate it in the Open Data Hub, since I see very few business cases by Data Consumers to use them

230406_SpecificheIntegrazione_NOI_v1.1.pdf

mrabans commented 1 year ago

afaik this is the interactive map of the Bundesnetzagentur which relies basically on the same data from DZT as we should have access to.

https://www.bundesnetzagentur.de/DE/Fachthemen/ElektrizitaetundGas/E-Mobilitaet/Ladesaeulenkarte/Karte/start.html

rcavaliere commented 1 year ago

@mrabans no updates from the Data Provider, what do you suggest to do?

mrabans commented 1 year ago

Here is the full info from Elias on how to do the API calls:

Kurz zur Erklärunbg: es geht dabei um die projection DS. Eine DS die dem Endpunkt sagt welche properties in der projection drin sein sollten. Die Erfahrung hat gezeigt, dass Anfragen gegen einen Graph sehr tief gehen können und daher sehr groß werden können. Das hat zu massiven Performanceproblemen geführt.

Daher geben wir im initialen response nur die ID des Datensatzes mit und für die vollen Daten kann man definieren was man alles braucht - über die projection DS.

In diesem Fall, mit den e-Chargingstations der DZT, macht es Sinn direkt über die einzelnen Stations zu itereieren. Also Beispiel 1: Suche nach allen e-Chargingstations, Beispiel 2: Station abholen

Beispiel 1: Leersuche nach e-Chargingstations (filterDS: https://semantify.it/ds/E85TgOxMg ==> e-Chargingstations)


GET https://proxy.opendatagermany.io/api/ts/v2/kg/things?kw=&filterDs=https://semantify.it/ds/E85TgOxMg
x-api-key: ***
> Response (gekürzt):

{ "metaData": { ... }, "data": [ { "@type": [ "https://odta.io/voc/EChargingStation" ], "https://vocab.sti2.at/ds/compliesWith": { "@id": "https://semantify.it/ds/E85TgOxMg" }, "@id": "http://onlim.com/entity/Ladestationen-Api-Bund/ECarChargingStation/100001001" }, ...


> Beispiel 2: Abholen einer E-charging station mit Details

GET https://proxy.opendatagermany.io/api/ts/v1/kg/things/100001001 ?ns=http%3A%2F%2Fonlim.com%2Fentity%2FLadestationen-Api-Bund%2FECarChargingStation%2F Content-Type: application/json x-api-key: b375e0ff5afd9b812b03ba3df41f64ec

> Response (gekürzt):

[ { "@id": "http://onlim.com/entity/Ladestationen-Api-Bund/ECarChargingStation/100001001", "@type": "https://odta.io/voc/EChargingStation", "https://odta.io/voc/hasCharger": { "@id": "http://onlim.com/entity/7769891b-a020-4ef8-87c7-189c0ebdeb66", "@type": "https://odta.io/voc/Charger", "https://odta.io/voc/power": { "@id": "http://onlim.com/entity/bef653cd-4065-48cb-b81d-8786da2f5d5f", "@type": "https://schema.org/QuantitativeValue", "https://schema.org/unitCode": "kW", "https://schema.org/unitText": "Kilowatt", "https://schema.org/value": { "@type": "http://www.w3.org/2001/XMLSchema#integer", "@value": "22" },

mrabans commented 1 year ago

These are the curl calls I've tested

Example 1: Get all e-charging stations

Call

curl --location 'https://proxy.opendatagermany.io/api/ts/v2/kg/things?kw=&filterDs=https%3A%2F%2Fsemantify.it%2Fds%2FE85TgOxMg' \
--header 'x-api-key: ***'

Response

{
    "metaData": {
        "dsCount": {
            "https://semantify.it/ds/E85TgOxMg": {
                "count": "32730",
                "name": {
                    "en": "e-chargingstation"
                }
            }
        },
        "page-size": 10,
        "sortSeed": "4e2059b5-0546-4af0-ba3b-9de7d313436e",
        "total": 32730,
        "current-page": 1
    },
    "data": [
        {
            "@type": [
                "https://odta.io/voc/EChargingStation"
            ],
            "https://vocab.sti2.at/ds/compliesWith": {
                "@id": "https://semantify.it/ds/E85TgOxMg"
            },
            "@id": "http://onlim.com/entity/Ladestationen-Api-Bund/ECarChargingStation/100001001"
        },
        {
            "@type": [
                "https://odta.io/voc/EChargingStation"
            ],
            "https://vocab.sti2.at/ds/compliesWith": {
                "@id": "https://semantify.it/ds/E85TgOxMg"
            },
            "@id": "http://onlim.com/entity/Ladestationen-Api-Bund/ECarChargingStation/44791001"
        },
...

Example 2: Get all details on station 100001001

Call

curl --location 'https://proxy.opendatagermany.io/api/ts/v1/kg/things/100001001?ns=http%253A%252F%252Fonlim.com%252Fentity%252FLadestationen-Api-Bund%252FECarChargingStation%252F' \
--header 'Content-Transfer-Encoding: application/json' \
--header 'x-api-key: ***'

Response

[
    {
        "@id": "http://onlim.com/entity/Ladestationen-Api-Bund/ECarChargingStation/100001001",
        "@type": "https://odta.io/voc/EChargingStation",
        "https://odta.io/voc/hasCharger": {
            "@id": "http://onlim.com/entity/7769891b-a020-4ef8-87c7-189c0ebdeb66",
            "@type": "https://odta.io/voc/Charger",
            "https://odta.io/voc/power": {
                "@id": "http://onlim.com/entity/bef653cd-4065-48cb-b81d-8786da2f5d5f",
                "@type": "https://schema.org/QuantitativeValue",
                "https://schema.org/unitCode": "kW",
                "https://schema.org/unitText": "Kilowatt",
                "https://schema.org/value": {
                    "@type": "http://www.w3.org/2001/XMLSchema#integer",
                    "@value": "22"
                },
                "https://vocab.sti2.at/ds/compliesWith": {
                    "@id": "https://semantify.it/ds/UO0HFc1Ce"
                }
            },
            "https://odta.io/voc/socket": "AC Steckdose Typ 2",
            "https://schema.org/name": "AC Steckdose Typ 2",
            "https://vocab.sti2.at/ds/compliesWith": {
                "@id": "https://semantify.it/ds/fuRrjo7x-"
            }
        },
        "https://schema.org/address": {
            "@id": "http://onlim.com/entity/4f001747-1891-4ad8-ac63-e3f5398eefc9",
            "@type": "https://schema.org/PostalAddress",
            "https://schema.org/addressCountry": "Deutschland",
            "https://schema.org/addressLocality": "Bocholt",
            "https://schema.org/postalCode": "46399",
            "https://schema.org/streetAddress": "Gildhaus Esch 3",
            "https://vocab.sti2.at/ds/compliesWith": {
                "@id": "https://semantify.it/ds/NP8df6sKy"
            }
        },
        "https://schema.org/geo": {
            "@id": "http://onlim.com/entity/229027df-9985-47e5-a3a5-232546505c2b",
            "@type": "https://schema.org/GeoCoordinates",
            "https://schema.org/latitude": {
                "@type": "http://www.w3.org/2001/XMLSchema#double",
                "@value": "5.184561E1"
            },
            "https://schema.org/longitude": {
                "@type": "http://www.w3.org/2001/XMLSchema#double",
                "@value": "6.587152E0"
            },
            "https://vocab.sti2.at/ds/compliesWith": {
                "@id": "https://semantify.it/ds/2NErTNGpd"
            }
        },
        "https://schema.org/hasMap": {
            "@type": "http://www.w3.org/2001/XMLSchema#anyURI",
            "@value": "https://w3w.co/kohle.l%C3%A4uft.aufgabe"
        },
        "https://schema.org/identifier": {
            "@id": "http://onlim.com/entity/Ladestationen-Api-Bund/ECarChargingStation/100001001/identifier",
            "@type": "https://schema.org/PropertyValue",
            "https://schema.org/name": "Ladestationen-Api-Bund_id",
            "https://schema.org/value": {
                "@type": "http://www.w3.org/2001/XMLSchema#integer",
                "@value": "100001001"
            },
            "https://vocab.sti2.at/ds/compliesWith": {
                "@id": "https://semantify.it/ds/evJvhycX1"
            }
        },
        "https://schema.org/name": "Elektro Siekmann - 100001001 - Normalladeeinrichtung",
        "https://schema.org/sdLicense": {
            "@type": "http://www.w3.org/2001/XMLSchema#anyURI",
            "@value": "https://creativecommons.org/licenses/by/4.0/"
        },
        "https://schema.org/sdPublisher": {
            "@id": "http://onlim.com/entity/e825f59e-0805-4597-86a8-34e15f1bb62c",
            "@type": "https://schema.org/Organization",
            "https://schema.org/name": "Bundesnetzagentur",
            "https://schema.org/url": {
                "@type": "http://www.w3.org/2001/XMLSchema#anyURI",
                "@value": "https://www.bundesnetzagentur.de/"
            },
            "https://vocab.sti2.at/ds/compliesWith": {
                "@id": "https://semantify.it/ds/dA_C4oQ0p"
            }
        },
        "https://vocab.sti2.at/ds/compliesWith": {
            "@id": "https://semantify.it/ds/E85TgOxMg"
        }
    }
]
mrabans commented 1 year ago

@clezag

rcavaliere commented 11 months ago

@mrabans thanks for the information. For the first call, how can we retrieve the different pages so to get the list of all charging stations?

mrabans commented 11 months ago

@rcavaliere I'll ask Elias

mrabans commented 11 months ago

@rcavaliere here is a complete cURL command with pagination, page-size and the dateModified filter:

curl --location 'https://proxy.opendatagermany.io/api/ts/v2/kg/things?filterDsList=https%3A%2F%2Fsemantify.it%2Fds%2FE85TgOxMg' \
--header 'page: 3' \
--header 'page-size: 5' \
--header 'Content-Type: application/ld+json' \
--header 'x-api-key: ***' \
--data-raw '{
    "@context": {
    "ometa": "http://onlim.com/meta/",
    "sq": "http://www.onlim.com/shapequery/",
    "@vocab": "http://www.onlim.com/shapequery/"
  },
    "sq:query":[
        {
          "ometa:dateModified":{
                "sq:value":"2023-02-23T19:00:00",
                "sq:op":">",
                "sq:datatype":"dateTime"

          }
        }
    ]    
}'

and the result 

{
    "metaData": {
        "dsCount": {
            "https://semantify.it/ds/E85TgOxMg": {
                "count": "32730",
                "name": {
                    "en": "e-chargingstation"
                }
            }
        },
        "page-size": 5,
        "sortSeed": "13404025-38dc-4d11-947a-f5245a29bd95",
        "total": 32730,
        "current-page": 3
    },
    "data": [
        {
            "@type": [
                "https://odta.io/voc/EChargingStation"
            ],
            "https://vocab.sti2.at/ds/compliesWith": {
                "@id": "https://semantify.it/ds/E85TgOxMg"
            },
            "@id": "http://onlim.com/entity/Ladestationen-Api-Bund/ECarChargingStation/65102546"
        },
        {
            "@type": [
                "https://odta.io/voc/EChargingStation"
            ],
            "https://vocab.sti2.at/ds/compliesWith": {
                "@id": "https://semantify.it/ds/E85TgOxMg"
            },
            "@id": "http://onlim.com/entity/Ladestationen-Api-Bund/ECarChargingStation/88321013"
        },
        {
            "@type": [
                "https://odta.io/voc/EChargingStation"
            ],
            "https://vocab.sti2.at/ds/compliesWith": {
                "@id": "https://semantify.it/ds/E85TgOxMg"
            },
            "@id": "http://onlim.com/entity/Ladestationen-Api-Bund/ECarChargingStation/67851203"
        },
        {
            "@type": [
                "https://odta.io/voc/EChargingStation"
            ],
            "https://vocab.sti2.at/ds/compliesWith": {
                "@id": "https://semantify.it/ds/E85TgOxMg"
            },
            "@id": "http://onlim.com/entity/Ladestationen-Api-Bund/ECarChargingStation/67851451"
        },
        {
            "@type": [
                "https://odta.io/voc/EChargingStation"
            ],
            "https://vocab.sti2.at/ds/compliesWith": {
                "@id": "https://semantify.it/ds/E85TgOxMg"
            },
            "@id": "http://onlim.com/entity/Ladestationen-Api-Bund/ECarChargingStation/73021005"
        }
    ]
}
rcavaliere commented 11 months ago

@mrabans @clezag @ohnewein good news. I think that we now start with the implementation of the Data Collector, a very basic dataset can be now retrieved. My proposal of specification is there, if there any comments on it we can of course discuss them.

230727_SpecificheIntegrazione_NOI_v1.2.pdf

mrabans commented 11 months ago

@rcavaliere this looks pretty good

clezag commented 11 months ago

@mrabans do they have an official documentation of the API?

For example if I have to request all 32000 echarging stations, and I have to use pagination, I think I have to pass the sort seed on the following calls, but I don't know how I should pass it.

Would also be good to know what happens with defunct or deleted stations. Do they get a new update date or do they simply delete them?

Also requesting the stations one by one is less than ideal (but can be dealt with if we have to)

clezag commented 10 months ago

@rcavaliere @mrabans A first complete import of the data is now available in testing

rcavaliere commented 9 months ago

@clezag sorry for the delay. The data is imported correctly, not further implementation activity is in my opinion needed

rcavaliere commented 9 months ago

However please wait for putting the Data Collector in production, first we need to clarify the agreement stuff (@mrabans and @ohnewein)

rcavaliere commented 5 months ago

@ohnewein @mrabans contacts have been reactivated during this sprint, as you have seen! Let's see the reaction of the counter part