search

noi-techpark / sta-nap-export

NeTEx and SIRI export for STA (MaaS4Italy)
1 stars 0 forks source link

EPIC: As MaaS4Italy I would like to receive from the Open Data Hub a standard interface for planned and real-time mobility data (NeTEx and SIRI) #1

Open rcavaliere opened 4 months ago

rcavaliere commented 4 months ago

In the scope of the new MaaS4Italy project the Open Data Hub is request to provide a standard interface for its mobility data, so that it can be integrated in a new MaaS architecture to be developed at national level. The following specification provides details between available Open Data Hub data and the requested fields.

240701_PianificazioneSviluppi.docx

The services to be considered are:

Below the national specifications to be considered:

230711_Linee-guida-compilazione-NeTEx-IT-v.3.0.pdf

240502_SpecificaSIRI_v.1.0.3.pdf

DSRM-Architettura Target_Dati dinamici e sharing v1.2_signed.pdf (relevant: paragraph 2.4)

NeTEx exports can be compared with an XSD before being published, available there https://github.com/5Tsrl/netex-italian-profile

Update 14.05.2024: STA has also requested to include in the exports bike parking services, since they are bookable and interesting for MaaS providers. I have amended in track change mode the internal specification document with indication on how to include this additional data. Please also note that there has been a new version of the SIRI national standard, some fields for parking have changed (removed).

clezag commented 1 week ago

@rcavaliere There was a bug when creating the FacilityRef ID that I've now fixed. What you are looking at is the car sharing data

rcavaliere commented 1 week ago

@clezag I had today a first meeting with the Accenture, which has the committment by the Ministry of the national MaaS platform. I have presented our first end-point. We need to implement following adaptations:

240701_PianificazioneSviluppi.docx

Can you please take care of this starting from next week?

I also suggest also to activate more "official" end points, which could be:

rcavaliere commented 1 week ago

Example of end-points, as provided by the Ministry:

rcavaliere commented 2 days ago

@clezag I have also seen that the field ParkingVehicleTypes is not fed in the NeTEx export, see e.g. screenshot below.

Image

Can you fix these open points in the next days? Despite on holiday I continue to have pressure to close these points as soon as possible. Moreover, we would like to use these standard end-points so to integrate parking data in our OTP instance (see https://github.com/noi-techpark/odh-mentor-otp/issues/173)

leonardehrenfried commented 2 days ago

Hi, I'm the OpenTripPlanner developer that implements the consumer for parking.

I echo @clezag's doubt about Siri-Lite meaning that it's JSON.

Entur also publishes their SX, VM and ET feeds as what they call "HTTP GET (Standard SIRI 2.0 Lite)" and they are using XML.

Example: https://api.entur.io/realtime/v1/rest/sx

I would love to see some documentation that explains the mapping from XML to JSON. Is it available?

leonardehrenfried commented 2 days ago

@rcavaliere has posted this document https://github.com/user-attachments/files/16069269/240620_RAP.-.Specifiche.interfacce.dati.dinamici.pdf which says

Le risposte previste da questo endpoint anche in questo caso possono essere sia in formato XML che in formato JSON e la decisione viene presa in base all’ header “Accept” fornito nella richiesta: il DSRM predilige il secondo formato tendenzialmente più snello e meno verboso, ma il contenuto informativo è assolutamente corrispondente al SIRI profilo italiano descritto nelle

Google Translate suggests that this means you have a choice between XML and JSON.

Why do I prefer XML? Because OTP already loads the complete Siri XML schema and makes converting it to Java trivial.

rcavaliere commented 1 day ago

@leonardehrenfried @clezag I think we could offer without issues also an XML export. The indication to implement JSON was given during several meetings with the implementers of this national platform.

clezag commented 1 day ago

@leonardehrenfried The JSON implementation has been done in a best effort way, because the only reference I could find was some French implementation from 2016.
It should be fairly easy however to provide a XML version again (I'll base it on the Accept header as outlined in the documentation you quoted).

Have you looked into the content itself? Is it conforming the the "SIRI standard" in OTP? From the little experience I've had with this SIRI/Netex stuff, the word "standard" is to be understood loosely, to say the least.

@rcavaliere 

add an authentication layer. They expect to get from us a client-id and client-secret to make the pull requests

Do they know it's open data and we're not doing any authorization anyway? This is a lot of effort for something that doesn't do anything. I suggest we create them some keycloak credentials, they can then request and supply the authentication token, but we don't implement anything that looks at the token and validates it. If further down the line we need authentication/authorization, we can add the authentication code. it's better to split the end-points in two, one for parking (siri-lite/fm/parking... and siri-lite/fm/sharing) It already is implemented this way. The only thing I will change is making the path siri-lite instead of siri for all our siri endpoints we should be able to implement all possible filtering possibilities by making the API call, as we do for our mobility API To be clear, only the filters outlined in the document?

leonardehrenfried commented 1 day ago

Have you looked into the content itself? Is it conforming the the "SIRI standard" in OTP? From the little experience I've had with this SIRI/Netex stuff, the word "standard" is to be understood loosely, to say the least.

This is also my impression. NeTEx/Siri are really (very big) XML schemas and it's up to the implementers to fill it with life and agree on a subset of it. Most of them call it a profile, so we have EPIP, Nordic profile, UK profile. There appears to be huge room for interpretation, which is a shame.

What I miss most is some form of community where you can ask "Here is my XML, am I using it how it is intended?".

To answer your question: OTP doesn't use the JSON version of Siri so it's hard to say if it conforms to a schema but if I translate it back to XML in my head it looks correct on first glance. However, I would prefer to try it out with a proper parser and then give you feedback.

clezag commented 1 day ago

@leonardehrenfried All good, I'll include the XML output in the next round of implementations and ping you here once it's up

rcavaliere commented 1 day ago

Do they know it's open data and we're not doing any authorization anyway? This is a lot of effort for something that doesn't do anything. I suggest we create them some keycloak credentials, they can then request and supply the authentication token, but we don't implement anything that looks at the token and validates it. If further down the line we need authentication/authorization, we can add the authentication code.

@clezag thanks for your feedback. OK, let's push our message that since all this data is licensed CC0 we won't implement any security mechanism in front of the end-points

rcavaliere commented 1 day ago

To be clear, only the filters outlined in the document?

@clezag, yes only these filters are really necessary!

leonardehrenfried commented 11 hours ago

@clezag I tried out the Netex parking feed today and it conformed flawlessly to the schema (that OTP uses). If the Siri XML is of the same quality, I expect it no problems there as well.