search

noi-techpark / bdp-commons

Open Data Hub time series data collectors (legacy)
GNU Affero General Public License v3.0
2 stars 12 forks source link

As STA / Municipality of Marlengo / Municipality of Bolzano, I would like to push the data of some parking facilities to the Open Data Hub #667

Open rcavaliere opened 8 months ago

rcavaliere commented 8 months ago

Specification documents and related context information:

To do list:

clezag commented 6 months ago

@rcavaliere The new push API is now available in development environment (note: this is different from our current testing)

The swagger can be found here: https://push.api.dev.testingmachine.eu

I've authorized them for provider=sta and dataset=parking-marlengo, so in their case the full URL will be https://push.api.dev.testingmachine.eu/push/sta/parking-marlengo.

They can just do a HTTP POST with whatever as body, and we will save that as raw data.

For authorization, they have to use the oauth client_credentials flow on our testing Keycloak instance: https://auth.opendatahub.testingmachine.eu/auth/realms/noi/protocol/openid-connect/token

And then pass the obtained access token as Authorization: Bearer <token> header when POSTing the data

Here is some documentation for the BDP API, but the principles are the same

I will send you the credentials separately

rcavaliere commented 6 months ago

@clezag ok, thanks a lot! I will organize the next steps with the Data Provider. Indeed, can you please prepare a similar end-point for another Data Provider? The end-point following this convention should look like:

https://push.api.dev.testingmachine.eu/push/municipality-bolzano/parking-macello

clezag commented 6 months ago

@rcavaliere I've created and sent you the credentials for that new endpoint

clezag commented 5 months ago

@rcavaliere FYI they don't seem to have pushed any data yet

rcavaliere commented 5 months ago

@rcavaliere FYI they don't seem to have pushed any data yet

@clezag I got a feedback on Friday afternoon from the technological supplier of STA and they told me that they activated the data stream. The messages should be in the format:

{ "id":"teratronik_test", "name_IT":"Teratronik Test", "name_DE":"Teratronik Test", "latitude":0.0, "longitude":0.0, "capacity":10, "measurements":[{ "timestamp":"2024-06-14 02:25:15", "availability":3 }] }

Can you please check? In the next days I would like to draft some minimal specification for the internal integration of both parking places, in both cases should be very simple!

rcavaliere commented 5 months ago

@clezag @dulvui I have completed the specification for these two Data Collectors, and also updated the description of the user story since the activity is now a little bit more broader. The integration is very easy, as you can see. Can you please implement the Data Collectors next week? In particular the parking Macello has a certain priority, they would like to switch on it ASAP since it would be then visible of lot of new variable message signs (through the Open Data Hub) and this new parking area is very important for better coordinating the traffic in the city.

240619_SpecificheIntegrazione_NOI_v1.1_Marlengo.pdf 240620_SpecificheIntegrazione_NOI_v1.1_ParcheggioMacello.pdf

clezag commented 5 months ago

@rcavaliere I will be on holiday next week.

@dulvui This is running on the new infrastructure already, using the new push API. So we already get the data, only the transformer is missing. I will send you some technical info, in case you want to take a crack at this.

rcavaliere commented 4 months ago

@clezag can you also give some priority to this user story? I am also getting some pressure here to have in short times the real-time data of these new parking facilities... thanks!

clezag commented 4 months ago

@rcavaliere First import is now up (not running realtime yet): https://mobility.api.opendatahub.testingmachine.eu/v2/tree/ParkingStation/*/latest?where=scode.eq.parking-bz:8

rcavaliere commented 4 months ago

@clezag many thanks for this. In relation to your points:

I've changed the station code to be parking-bz:<uid>, because just having "8" there is probably going to cause conflicts with other things.

OK

I've changed the metadata name of lots to capacity to be in line with the other parking data

OK, but let's keep just one value, I see both values available in the metadata table / structure

I think Lots and Tot are reversed. Lots seems to be current capacity, and Tot the total

Let me check this with the data supplier

what does Floor do? Do we have to register separate stations per floor? is the capacity specific to the floor? It seems to be "0" for the cases I've looked at, so it's probably not the number of floors available

This field indicates if we have a parking with multiple floors, we could have information for each single floor. In this case, it is an open space parking without floors. We could also omit this.

rcavaliere commented 4 months ago

@clezag sorry I just realized that the implementation was done in that way, i.e. lots = current capacity, tot = nominal capacity. Therefore tot has to be mapped with the metadata static field 'capacity, while lots with the measurement free. I have corrected the internal specification document. Can you please fix this?

240703_SpecificheIntegrazione_NOI_v1.1_ParcheggioMacello.pdf

clezag commented 4 months ago

@rcavaliere I've implemented the changes.
Also, I've appended the floor to the station ID, so in the future when there is multiple floors, a station is created for each floor.

Since the stationcode changed, better use this call to look at the data:

https://mobility.api.opendatahub.testingmachine.eu/v2/tree/ParkingStation/*/latest?where=sorigin.eq.%22Municipality%20Bolzano%22

Regarding the period, I put 300, but since this data is pushed (and currently it looks like somewhat irregular time intervals) we really don't know the update frequency. Do you have some indication from the data provider on it?

clezag commented 4 months ago

@rcavaliere I've now brought the data collector online, so it should always be up to date.

Something else that I've noticed: The entering/exiting vehicles, which time frame are they referring to? We should probably set the period to that value. Or indicate in the datatype itself that they are for example the values since start of day

rcavaliere commented 4 months ago

@clezag the integration looks now OK. As far as the data frequency is concerned, we agreed that for each entering or exiting vehicle the back-end triggers the push of a data records towards our platform. Looking at the data (in the DB, since I can not see the time series on analytics DEV) it seems to me that the data is consistent to this. Maybe you can add to the description of the type that they refer to the entering / exiting vehicles since the start of the day (every night the counter is reset, as far as they told me). I suggest to clean the DB with the test data and have a look at the time series for a couple of days, in case let us clarify some doubts with the technological supplier of the Municipality of BZ. Thanks for your work!

clezag commented 4 months ago

@rcavaliere The data is visible on analytics, it's in regular testing (https://analytics.opendatahub.testingmachine.eu/), not development

rcavaliere commented 4 months ago

@clezag you are right sorry, I didn't include today in the filter :-)

clezag commented 4 months ago

@rcavaliere I was wrong :D the data collector that's now running in the dev environment also writes only to dev environment (we can change this if you need): https://analytics.dev.testingmachine.eu/

Since we are on the new infrastructure, I was able to reelaborate all the historical data, and we have the complete time series 🚀🚀

rcavaliere commented 4 months ago

@clezag good! I would also store the data in the test environment so that it's visible on https://analytics.opendatahub.testingmachine.eu/

clezag commented 4 months ago

@rcavaliere now it's writing exclusively to testing (https://analytics.opendatahub.testingmachine.eu/). I've also reloaded the history there.

rcavaliere commented 4 months ago

@clezag can you please put in production the Data Collector for parking Macello and go forward with the Data Collector for parking Marlengo? Please give also priority to user story https://github.com/noi-techpark/sta-nap-export/issues/1

clezag commented 4 months ago

@rcavaliere Just a heads up that putting this into production will take a little more time than usual (few days), because some underlying parts of the infrastructure have to first be set up.

clezag commented 4 months ago

@rcavaliere deployed in production, I have sent you the credentials via mail. Once they start pushing data to prod, we can migrate the preceding history from testing.

Will they keep pushing to test as well? I don't think we currently have a clear idea how we will handle test/prod for pushed data in the future. We can set up something to duplicate production to testing on an infrastructure level, but we're not there yet.

clezag commented 4 months ago

@rcavaliere marlengo is now up in testing. We don't have any real data from them yet, and they did not continue sending beyond the first tests. https://mobility.api.opendatahub.testingmachine.eu/v2/flat,node/ParkingStation/*/latest/?where=sorigin.eq.STA

rcavaliere commented 4 months ago

@clezag Data Collector Marlengo is OK. I will ask for real data to be pushed, so to make real tests. Can you check the connection with parking Macello? It seems that yesterday afternoon the data flow has stopped. Is the issue on our side on or their side?

clezag commented 4 months ago

@rcavaliere From the logs it looks like they stopped pushing.

rcavaliere commented 4 months ago

@ohnewein situation here: we have a new parking integrated for the city of Bolzano, is called TURIST PARKING. It's a separate Data Collector which receives the data from the parking itself with a push API. The Data Collector is already in the new core infrastructure of the Open Data Hub.

The second Data Collector is related to the parking of the train station of Marlengo, managed by STA with the Municipality there. The architecture is similar to the previous one. Here we are still in testing phase, so this task is still in progress.

rcavaliere commented 2 months ago

@clezag we are waiting STA configuring the Marlengo parking area so that real-time data is transmitted

clezag commented 1 week ago

We're getting real data in testing now, so we can proceed with the integration:

{
  "id": "fuchs_marling",
  "name_IT": "Parcheggio stazione Marlengo",
  "name_DE": "Parkplatz Marling Bahnhof",
  "latitude": 46.66489445071061,
  "longitude": 11.138228889410987,
  "capacity": 37,
  "measurements": [
    {
      "timestamp": "2024-11-20 08:07:11",
      "availability": 25
    }
  ]
}
clezag commented 1 week ago

@rcavaliere Looks like it's already working: https://mobility.api.opendatahub.testingmachine.eu/v2/flat,node/ParkingStation/*/latest/?where=sorigin.eq.STA

{
  "offset": 0,
  "data": [
    {
      "_timestamp": "2024-11-20 07:11:14.998+0000",
      "tdescription": "free",
      "tmetadata": {},
      "tname": "free",
      "ttype": "Instantaneous",
      "tunit": "",
      "mperiod": 300,
      "mtransactiontime": "2024-07-17 08:02:58.541+0000",
      "mvalidtime": "2024-11-20 07:11:14.998+0000",
      "mvalue": 25,
      "prlineage": "STA",
      "prname": "tr-parking-offstreet-teratronik",
      "prversion": "390fc846a571e9e1b2b55daad17c4c757fd32cd7",
      "sactive": true,
      "savailable": true,
      "scode": "parking-sta-marlengo",
      "scoordinate": {
        "x": 11.138229,
        "y": 46.664894,
        "srid": 4326
      },
      "smetadata": {
        "name_DE": "Parkplatz Marling Bahnhof",
        "name_IT": "Parcheggio stazione Marlengo",
        "capacity": 37
      },
      "sname": "Parcheggio stazione Marlengo",
      "sorigin": "STA",
      "stype": "ParkingStation"
    }
  ],
  "limit": 200
}

Can you take a look if it's ok? I think we can go into production with this fairly fast.