search

noi-techpark / bdp-commons

GNU Affero General Public License v3.0
2 stars 12 forks source link

As STA / Municipality of Marlengo / Municipality of Bolzano, I would like to push the data of some parking facilities to the Open Data Hub #667

Open rcavaliere opened 3 months ago

rcavaliere commented 3 months ago

Specification documents and related context information:

To do list:

clezag commented 1 month ago

@rcavaliere The new push API is now available in development environment (note: this is different from our current testing)

The swagger can be found here: https://push.api.dev.testingmachine.eu

I've authorized them for provider=sta and dataset=parking-marlengo, so in their case the full URL will be https://push.api.dev.testingmachine.eu/push/sta/parking-marlengo.

They can just do a HTTP POST with whatever as body, and we will save that as raw data.

For authorization, they have to use the oauth client_credentials flow on our testing Keycloak instance: https://auth.opendatahub.testingmachine.eu/auth/realms/noi/protocol/openid-connect/token

And then pass the obtained access token as Authorization: Bearer <token> header when POSTing the data

Here is some documentation for the BDP API, but the principles are the same

I will send you the credentials separately

rcavaliere commented 1 month ago

@clezag ok, thanks a lot! I will organize the next steps with the Data Provider. Indeed, can you please prepare a similar end-point for another Data Provider? The end-point following this convention should look like:

https://push.api.dev.testingmachine.eu/push/municipality-bolzano/parking-macello

clezag commented 1 month ago

@rcavaliere I've created and sent you the credentials for that new endpoint

clezag commented 2 weeks ago

@rcavaliere FYI they don't seem to have pushed any data yet

rcavaliere commented 2 weeks ago

@rcavaliere FYI they don't seem to have pushed any data yet

@clezag I got a feedback on Friday afternoon from the technological supplier of STA and they told me that they activated the data stream. The messages should be in the format:

{ "id":"teratronik_test", "name_IT":"Teratronik Test", "name_DE":"Teratronik Test", "latitude":0.0, "longitude":0.0, "capacity":10, "measurements":[{ "timestamp":"2024-06-14 02:25:15", "availability":3 }] }

Can you please check? In the next days I would like to draft some minimal specification for the internal integration of both parking places, in both cases should be very simple!

rcavaliere commented 1 week ago

@clezag @dulvui I have completed the specification for these two Data Collectors, and also updated the description of the user story since the activity is now a little bit more broader. The integration is very easy, as you can see. Can you please implement the Data Collectors next week? In particular the parking Macello has a certain priority, they would like to switch on it ASAP since it would be then visible of lot of new variable message signs (through the Open Data Hub) and this new parking area is very important for better coordinating the traffic in the city.

240619_SpecificheIntegrazione_NOI_v1.1_Marlengo.pdf 240620_SpecificheIntegrazione_NOI_v1.1_ParcheggioMacello.pdf

clezag commented 1 week ago

@rcavaliere I will be on holiday next week.

@dulvui This is running on the new infrastructure already, using the new push API. So we already get the data, only the transformer is missing. I will send you some technical info, in case you want to take a crack at this.

rcavaliere commented 1 day ago

@clezag can you also give some priority to this user story? I am also getting some pressure here to have in short times the real-time data of these new parking facilities... thanks!

clezag commented 1 day ago

@rcavaliere First import is now up (not running realtime yet): https://mobility.api.opendatahub.testingmachine.eu/v2/tree/ParkingStation/*/latest?where=scode.eq.parking-bz:8

rcavaliere commented 18 hours ago

@clezag many thanks for this. In relation to your points:

I've changed the station code to be parking-bz:<uid>, because just having "8" there is probably going to cause conflicts with other things.

OK

I've changed the metadata name of lots to capacity to be in line with the other parking data

OK, but let's keep just one value, I see both values available in the metadata table / structure

I think Lots and Tot are reversed. Lots seems to be current capacity, and Tot the total

Let me check this with the data supplier

what does Floor do? Do we have to register separate stations per floor? is the capacity specific to the floor? It seems to be "0" for the cases I've looked at, so it's probably not the number of floors available

This field indicates if we have a parking with multiple floors, we could have information for each single floor. In this case, it is an open space parking without floors. We could also omit this.

rcavaliere commented 18 hours ago

@clezag sorry I just realized that the implementation was done in that way, i.e. lots = current capacity, tot = nominal capacity. Therefore tot has to be mapped with the metadata static field 'capacity, while lots with the measurement free. I have corrected the internal specification document. Can you please fix this?

240703_SpecificheIntegrazione_NOI_v1.1_ParcheggioMacello.pdf

clezag commented 15 hours ago

@rcavaliere I've implemented the changes.
Also, I've appended the floor to the station ID, so in the future when there is multiple floors, a station is created for each floor.

Since the stationcode changed, better use this call to look at the data:

https://mobility.api.opendatahub.testingmachine.eu/v2/tree/ParkingStation/*/latest?where=sorigin.eq.%22Municipality%20Bolzano%22

Regarding the period, I put 300, but since this data is pushed (and currently it looks like somewhat irregular time intervals) we really don't know the update frequency. Do you have some indication from the data provider on it?

clezag commented 11 hours ago

@rcavaliere I've now brought the data collector online, so it should always be up to date.

Something else that I've noticed: The entering/exiting vehicles, which time frame are they referring to? We should probably set the period to that value. Or indicate in the datatype itself that they are for example the values since start of day

rcavaliere commented 11 hours ago

@clezag the integration looks now OK. As far as the data frequency is concerned, we agreed that for each entering or exiting vehicle the back-end triggers the push of a data records towards our platform. Looking at the data (in the DB, since I can not see the time series on analytics DEV) it seems to me that the data is consistent to this. Maybe you can add to the description of the type that they refer to the entering / exiting vehicles since the start of the day (every night the counter is reset, as far as they told me). I suggest to clean the DB with the test data and have a look at the time series for a couple of days, in case let us clarify some doubts with the technological supplier of the Municipality of BZ. Thanks for your work!

clezag commented 10 hours ago

@rcavaliere The data is visible on analytics, it's in regular testing (https://analytics.opendatahub.testingmachine.eu/), not development

rcavaliere commented 10 hours ago

@clezag you are right sorry, I didn't include today in the filter :-)

clezag commented 9 hours ago

@rcavaliere I was wrong :D the data collector that's now running in the dev environment also writes only to dev environment (we can change this if you need): https://analytics.dev.testingmachine.eu/

Since we are on the new infrastructure, I was able to reelaborate all the historical data, and we have the complete time series 🚀🚀

rcavaliere commented 9 hours ago

@clezag good! I would also store the data in the test environment so that it's visible on https://analytics.opendatahub.testingmachine.eu/

clezag commented 8 hours ago

@rcavaliere now it's writing exclusively to testing (https://analytics.opendatahub.testingmachine.eu/). I've also reloaded the history there.