As a project manager I would like to integrate the Data Collector related to the traffic and Bluetooth data of the Province of Bolzano

[Peter080202]

@rcavaliere How do I know from the Api AnagrafichePostazioni (which returns as said in the documentation traffic and bluetooth sensors), that the station is a traffic or bluetooth sensor? Or am I missing something?

[rcavaliere]

@Peter080202 all stations are always both

[Peter080202]

@rcavaliere ok thanks for the quick response

[Peter080202]

@rcavaliere A question about saving the stations (you can also answer me tomorrow at the meeting): In the requirements there is the following text: "For each station provided by the source, multiple ODH stations have to be created: one ODH station for each lane (TrafficSensor): in the majority of the cases, since we have two lanes (two direction of travels), two stations will be created (e.g. station nr. 3 will create the ODH stations "3:verso Bolzano" and "3: verso Trento")". Unfortunately, this is not possible or not at the moment, because the Api says the second time that a station with this Id has already been created/saved.

Warning message from bdp-core: Station with ID 3 already in syncStation list... skipping!

Suggested solution: I simply save the data of stations, since one who uses the data collector can calculate the lane from the data (with the attribute "descendente" and "ascendente"). Since we also have the direction, i.e. "descendente" or "ascendente" given in the api where we retrieve the data about the stations, it´s not needed to store the station twice in my opinion. If it´s not 100% clear we can discuss tomorrow in the meeting

[rcavaliere]

@Peter080202 let's discuss this more in detail tomorrow. In any case, it's expected that the station is created only the first time, while afterwards we just make an update of its metadata, if there are changes. In case you try to create it again the ODH core should recognize it and avoid to create it again.

[rcavaliere]

@Peter080202 please find here the updated specification document. Let me know in case of further requests! 220804_SpecificheIntegrazioneODH_NOI_v1.2.docx

[Peter080202]

@rcavaliere perfect thanks

[Peter080202]

@rcavaliere Should I also include a logic for the traffic stations that distinguishes the measurements by direction? It would be practical to have something like StationId:Direction instead of StationId (like when storing the traffic stations themselves), as this would allow the data or direction to be identified.

[Peter080202]

@rcavaliere And for the traffic station types it always says: Not found, skipping...the type for bluetooth on the other hand is found

[rcavaliere]

Should I also include a logic for the traffic stations that distinguishes the measurements by direction? It would be practical to have something like StationId:Direction instead of StationId (like when storing the traffic stations themselves), as this would allow the data or direction to be identified.

Yes, I had already specified this (see evidenced part from the screenshot below, taken from the specification document)

[rcavaliere]

@rcavaliere And for the traffic station types it always says: Not found, skipping...the type for bluetooth on the other hand is found

Can you provide more details about the types you trying to give back to the ODH?

[Peter080202]

Should I also include a logic for the traffic stations that distinguishes the measurements by direction? It would be practical to have something like StationId:Direction instead of StationId (like when storing the traffic stations themselves), as this would allow the data or direction to be identified.

Yes, I had already specified this (see evidenced part from the screenshot below, taken from the specification document)

Yes for the stations it is clear. I mean for the measurement of the stations, because with the data we get from the api (DatiAggregatiSuPostazioni) I could distinguish between the directions if needed. I don´t know whether it´s clear...otherwise we could schedule a small meeting if you have time

[rcavaliere]

@Peter080202 ok I understand. No, this is not relevant, we could count the vehicles which travel in the opposite direction they should ("veicoli contromano") but I would leave this detail out

[Peter080202]

@rcavaliere And for the traffic station types it always says: Not found, skipping...the type for bluetooth on the other hand is found

Can you provide more details about the types you trying to give back to the ODH?

Thanks for the answer of the other question. Basically when I try to push them to the database I get following logs in the console, meaning that they aren´t there yet. For headway-variance and gap-variance I´m not getting them because I had to register them. Probably just because I´m running it locally and on the server they will be there, right?

[rcavaliere]

I assume so. @dulvui can you please confirm? These types are already in the ODH

[dulvui]

@rcavaliere Yes I can confirm that all types of the image above are already present in the database.

[Peter080202]

@dulvui Ok thanks for the confirmation

[Peter080202]

@rcavaliere Just as an information: I have just completed the pull request

[rcavaliere]

@Peter080202 thanks. @dulvui can you integrate this: https://github.com/noi-techpark/bdp-commons/pull/558 ?

[dulvui]

@Peter080202 @rcavaliere The dc is now running on testing, but I saw that the scheduled jobs are configured to run every 10 days. Is that correct or should i change it to every 10 minutes or hours?

[rcavaliere]

@dulvui please set this to 10 minutes!

[dulvui]

@rcavaliere Now it runs every 10 minutes, but an Exception is thrown:

java.io.FileNotFoundException: jsonfiles/classificationSchemas.json (No such file or directory)
    at java.io.FileInputStream.open0(Native Method)
    at java.io.FileInputStream.open(FileInputStream.java:195)
    at java.io.FileInputStream.<init>(FileInputStream.java:138)
    at java.io.FileInputStream.<init>(FileInputStream.java:93)
    at java.io.FileReader.<init>(FileReader.java:58)
    at it.bz.odh.trafficprovbz.FamasClient.readJsonFile(FamasClient.java:130)
    at it.bz.odh.trafficprovbz.FamasClient.getClassificationSchemas(FamasClient.java:53)

I think the jsonfiles/classificationSchemas.json file path is wrong. @Peter080202 Could you please fix this?

[Peter080202]

@dulvui when I run it on my machine it works but I will fix it now

[Peter080202]

@dulvui just made pull request for fixed version

[dulvui]

@Peter080202 Thank you! The fixed version is now running and seems to work

[rcavaliere]

@dulvui end-point of the Data Collector: webservices.trafficopab.famassystem.it/api/v1/NomeServizio where NomeServizio is the name of the different services provided (see attachment for all the details). The end-point given in this document was updated, the correct one is the one above indicated

211119_SpecificheAPI_FamasSystem.pdf

[rcavaliere]

@Peter080202 we have successfully integrated the Data Collector and let it connect with the external end-point we have at disposal. Following open points have come out from a first testing of the Data Collector:

• [ ] We have seen different values returned for stations metadata (field "stationcode" and "name"). Are you inserting here different values? The codes we receive are not the same.
• [ ] We don't see measurement coming in, but this could be an issue related to the connection with the end-point. @dulvui will investigate more on this (from 23.8)

[Peter080202]

@rcavaliere in theory, it should already work correctly. The only problem I have seen (if you still get this "error" with the test data): the name and the id are not the same in the test data (probably my mistake). Maybe we should adjust the id or name accordingly in the test data. We can discuss it eventually next week during the sprint review.

[rcavaliere]

@Peter080202 thanks. Maybe @dulvui can also have a look at this point next week (we are both on vacation in these days...)

[rcavaliere]

Decision 25.08:

• [x] @dulvui will fix the issues related to the correct integration of measurements (+ the status of the traffic stations - active = FALSE)
• [x] @rcavaliere will contact Famas in relation to the metadata quality issue ("stationcode" and "name")

We will provide you a feedback in relation to the first point, so that we can close this implementation work. The release on production will take place after this clearance with Famas

[dulvui]

@rcavaliere @Peter080202 The problems are now fixed and the measurements are now available in the database. The are still some stations set to ative=false, but all stations affected are the ones with the wrong id/name mapping. So resolving the issue with the FAMAS API will probably solve that problem too.

[Peter080202]

@dulvui perfect thanks

[rcavaliere]

@dulvui I have got a feedback from Famas System. The two fields "Id" and "Nome" in their API have two different meanings:

• ID: is an internal code in the Famas System DB
• Nome: is the "official" code of the traffic station, as known for example by the operators of the PAB

At the beginning these two fields had the same values, at a certain point this alignment was broken.

However, what we should consider is just the field "Nome". As already indicated in the specification, we should feed both our fields "stationcode" and "name" with "Nome"+"CorsieInfo/Descrizione". I think at the moment the two fields consider in one case "Id", and in the other case "Nome".

Can you please fix this, so that we check that everything is correct?

[dulvui]

@rcavaliere Now the filed "Nome" is used for station id and station name is live in testing and it looks fine. Also the active problem seems solved, because all stations with the correct id/name are now active and the old wrong ones are not active.

[rcavaliere]

@dulvui perfect! I tested this, and no works without problem. Let's go in production!

[dulvui]

@rcavaliere Now it is running on production

[rcavaliere]

@dulvui good, I start to see something. What is the visibility of this data? This should be limited to the BrennerLEC users (BDP_BLC user group in Keycloack)

[dulvui]

@rcavaliere At the moment its closed data. I will open it for the BDP_BLC group in testing this afternoon

[dulvui]

@rcavaliere Now the data is open for group BDP_BLC in testing

[rcavaliere]

@dulvui that's fine. Did you do also in production? I think so, since I can't see this data at the moment on analytics...

[dulvui]

@rcavaliere No it was live only on testing. Now it is live on production.

[rcavaliere]

@dulvui are you sure? I am still seeing the same...

[dulvui]

@rcavaliere It is online in production. I think that the problem is that the dc at the moment doesn't write any data into the database for TrafficSensors. I'll investigate now and let you know.

[dulvui]

@rcavaliere I fixed now the problem and the TrafficSesnor data is now written correctly Can you please check again if you an see the data on analytics?

[rcavaliere]

@dulvui I think we still have some issues: metadata is not displayed and data has some strange behaviour....

[dulvui]

@rcavaliere I fixed now the problem with the measurements data and only one value is saved instead of multiple ones. I don't if also the metadata is fixed. Let me know if something is still missing.

[rcavaliere]

@dulvui thanks for the effort, but it seems that the issues are still visible. Let's discuss them together

[dulvui]

@rcavaliere I could delete the old/wrong data, but lets see together.

[dulvui]

@rcavaliere I saw now that the some metadata is saved as JSON and so the analytics website shows [OBJECT] instead of the value. Now I modified the page locally so that I can see the values as you can see in the picture.

I would change the metadata so that we have following keys and values (also translated to english) instead of the nested JSONS: MUNICIPALITY REGION DIRECTION STREET_NAME KILOMETRIC LANES_TOTAL

Let me know if I should implement it like this or use other keys.

[rcavaliere]

@dulvui good that you found the mistake. Please do the following:

• [x] clean up the metadata as you suggested
• [x] ensure that each traffic station has its own metadata, e.g. in this case in corsieinfo just the structure related to "VERSO BOLZANO"