As STA I would like that the real-time availability data of my bike parking structures ("bike boxen") is integrated and make then available in the Open Data Hub, so that I can show this information on the suedtirolmobil channels

[rcavaliere]

To do list:

• [x] @clezag starts testing the end-point according to the information provided by STA supplier (Bicincittà)
• [x] @rcavaliere makes a proposal on how to design the mapping with the Open Data Hub data model
• [x] @clezag implements the Data Collector with the support of @dulvui

Technical information (shared separately)

• Postman information
• Access credentials to testing environment of Bicincittà (only fake data currently available)

Specification integration document: 230308_SpecificheIntegrazione_NOI_v1.4.pdf

[clezag]

I've started to implement the data collector, but there are some problems with the authentication mechanism. The authentication scheme looks very much like Oauth 2 (parameter names, response), but it does not actually conform to the specification:

The token endpoint /connect/token does not accept client credentials via HTTP basic auth, it only accepts them as urlencoded parameters. Using basic auth results in a HTTP 400 error.

In the client_credentials flow, credentials shoud be provided (and accepted) as basic authentication header, while only the grant_type is passed as a an urlencoded param (see the RFC). This is also how tools like Swagger or Spring implement it.

While we could implement a workaround for this issue, it prevents us from using standard libraries such as Spring to handle authentication. I imagine that other consumers of the API will have the same problem, so I suggest passing this feedback on to the API developers.

[rcavaliere]

@clezag thanks for the feedback, issue shared with the Data Provider. Let's wait that they fix this issue...

[clezag]

@rcavaliere The Data Provider has resolved the issue. We can proceed with defining and implementing the data mapping.

[rcavaliere]

@clezag based on the information currently available, my proposal is to integrate this real-time data stream as described in the attached document. Let me know if we need to discuss details together. 230214_SpecificheIntegrazione_NOI_v1.1.pdf

[clezag]

@rcavaliere it's up in testing, you can take a look at the data in the test DB.

[rcavaliere]

@clezag I have tested the Data Collector, nice work! There are some little points to be adjusted:

• [x] the "Bay" stations should have a different stationtype, i.e. "BikeParkingBay" (you implemented it right, I didn't explictly report in the specification document - sorry")
• [x] we should avoid to store the field stationPlaces/state as metadata for the BikeParking station. This field is associated to the measurement of the correspondent bay, so I won't consider this here otherwise we a lot of unuseful records in the metadata table
• [x] I have seen that in the table measurementstring you are storing measurements that already saved in the measurement table (in the test DB, they are associated to type_id = 6029,6125,6126). I think this is a mistake, we just need to store this once in the measurement table
• [x] in the metadata associated to the bike parking bay, I would also add the field stationPlaces/isAssisted (was not specified, I have added it in the specification document)

The updated version of the specification document is available here. 230224_SpecificheIntegrazione_NOI_v1.2.pdf

In general, I have some concerns about the data quality, the naming of certain types and the meaning of certain fields. I will contact the Data Provider for more explanations. In the meantime I would fix these points.

[clezag]

@rcavaliere thanks for the feedback. I've implemented the requested changes, the build is available in testing.

The data in measurementstring(history) is still there from early testing where data types were not configured correctly. The current build should not write to those tables anymore.

[rcavaliere]

@clezag thanks for your work, everything looks fine

I have made a little modification to the specification document based on the feedback of the Data Provider (see attached file), basically:

• [x] "type" field: has also a supported '0' value (see associated meaning in the document)

230227_SpecificheIntegrazione_NOI_v1.3.pdf

Can you please make this final adjustment? Then we should wait that the Data Providers puts everything in its production environment...

[clezag]

@rcavaliere thank you! I've made the change in testing.

[rcavaliere]

@clezag good! Can you please delete all previous data stored in the testing environment and then start the Data Collector again? I would like to test the Data Collector and the modified API of the Data Provider in a more "clean" way

[clezag]

@rcavaliere I've deleted the old data and updated the data collector with the new field names (on our side the data type names remain the same). One thing I've noticed is that the station/place/metadata.type field is still always 0 instead of 1 or 2 like specified in the requirements and API documentation.

[rcavaliere]

@clezag you are right, I have requested to Bicincittà the meaning of this value. Another question: why did you use two different types for the type "state" associated to BikeParking and BikeParkingBay? In theory, we can always use the same type

[clezag]

@rcavaliere sorry, probably missed that because I've implemented the station level "state" measurement only after your feedback. I've changed it so that now both use the same "usageState" data type specified in the requirements doc.

You will continue to see the outdated data with type "state" until we clean up the DB once again.

[rcavaliere]

@clezag based on the final feedback on Bicincittà, I would suggest a final modification, i.e. to not save the field stationPlaces/type for the bike parking bays in case they are referred to a bike sharing station. This should be saved only it is a bike parking station. See revised version of the specification

230308_SpecificheIntegrazione_NOI_v1.4.pdf

[clezag]

@rcavaliere New build is up in testing, containing both your requested change and the serviceType calls detailed below:

Looking through the data and revisiting the postman file provided, I've noticed a bug/inconsistency in the API. We currently don't have any type = 2 (parking) stations in the DB, and the reason is the following:

When interrogating /resources/stations, which returns a list of stations for a city, an additional parameter serviceType can be provided. I assume that this doesn't corresponds to the field station.type, but is just 1 = sharing, 2 = parking.

Currently it behaves as follows:

If I search with

• no serviceType parameter: Webservice only returns stations of type 0 and 1 (this is why currently we have no type 2 stations)
• serviceType = 0: I get a "generic error"
• serviceType = 1: I get only stations with type 0 and 1, same as searching without any type
• serviceType = 2: I get only stations with type 2

I would expect the webservice to return all stations of any type if I don't provide any serviceType parameter to the call. This would both be logically consistent, and save API consumers a second webservice call.

I've just now implemented two separate calls (so we can proceed with testing) with type 1 and type 2 for every city, but we should probably signal this inconsistency to the data provider, since they seem open to feedback.

Another thing: the /resources/services returns multiple services, BicInCittà is only one of them. I currently load all cities of all services. Is this OK? Asking because we called the origin "bicincitta"

[rcavaliere]

@clezag thanks for the feedback. Good that you have fixed this in that way, but the Data Provider should fix this - I will write them again. Let's keep the origin "Bicincittà", in their platform they several services for different clients. We will see if they then will share with us all data when we go into production...

[rcavaliere]

Data Provider should activate their back-end soon. @clezag, testing activities could be planned in the next sprint

[rcavaliere]

@clezag @dulvui the Data Provider has unexpectedly changed their API, which is something bad. This was however done to accomplish other users of their API, which is needed to implement the entire customer journey on suedtirolmobil. In any case, I have seen the changes and are in my point of view quite minor. They also introduced multilinguism which is something more. Please find attached the reviewed version of the integration specifications, both in track change and in clean mode. Let me know if you have doubts!

230630_SpecificheIntegrazione_NOI_v2.1.pdf 230630_SpecificheIntegrazione_NOI_v2.1-clean.pdf

[dulvui]

@rcavaliere Live on testing now BikeParking stations: https://mobility.api.opendatahub.testingmachine.eu/v2/flat,node/BikeParking/ data: https://mobility.api.opendatahub.testingmachine.eu/v2/flat,node/BikeParking/*/latest?where=sactive.eq.true

BikeParkingBay stations: https://mobility.api.opendatahub.testingmachine.eu/v2/flat,node/BikeParkingBay/ data: https://mobility.api.opendatahub.testingmachine.eu/v2/flat,node/BikeParkingBay/*/latest?where=sactive.eq.true

[rcavaliere]

@dulvui looks good. I will test this before sharing our APIs with STA, which wants to immediately start with the integration on their suedtirolmobil channels

[rcavaliere]

@dulvui I am not sure that the data we currently have stored is the one that now this new API is providing. In particular for the metadata of the bike parking station we expect different values. Can you clean the test DB with the data we have and start the Data Collector again?

[dulvui]

@rcavaliere Database is clean now and I made some fixes for the metadata. Let me know if you still find problems

[rcavaliere]

@dulvui thanks a lot. I have checked the data entering, I have some questions:

• [x] the name of the station is saved in the table station only, in multiple languages. What is currently providing the Data Provider. The expectation was to have multiple names in different languages, and to store one reference name as station name in the table station, the other names in the other languages as metadata
• [x] in the metadata of the 'BikeParkingBay' stations, we are also storing the field "state". This is however a measurement of the station and not a static metadata field, so has not to be stored there
• [x] for the measurementstring associated to the stations 'BikeParking', I have seen some strange values 'unknownState0'. Are we receiving here some unexpected values, e.g. '0'? I would suggest that if we receive something different from the expected values we put 'unknown'
• [x] for the measurementstring associated to the stations 'BikeParkingBay', I have seen some unexpected values, e.g. 'free'. What mapping is implemented? It should be 1=in service, 2= occupied – in service; 3 = out of service (+ other values -> unknown, as the point before)

[dulvui]

@rcavaliere Now I added the multilingual fields names and addresses in the BikeParking metadata. Also the mapping should now be correct.

[rcavaliere]

@dulvui thanks a lot, all the comments above are now solved. I will now share the testing API of the Open Data Hub with STA and their supplier, so that they can start the integration. I will ask them if they prefer that we put everything already in production, so that they don't have to change the URL of the requests anymore. I will let you know!

[clezag]

@rcavaliere , @dulvui I've modified the datatype names as requested and put the data collector (and related ninja API changes) into production