As a project manager I would like to check the status of the DC VMS A22

rcavaliere commented 2 years ago

Open issues:

[x] Dump historical data for the period in which we were not able to receive data
[x] Implement improved logics for correct values reading (texts vs pictograms with paging)
Postponed #509

Piiit commented 2 years ago

@rcavaliere I checked the data collector, but I do not understand what the problem is. Could you please provide some examples and what you would expect to be there... thank you

rcavaliere commented 2 years ago

@Piiit the issue here is that the data stream is stopped at 1.3. Don't we receive updated data? I don't think so...

Piiit commented 2 years ago

@rcavaliere Ah, yes, clear now... looked only at stations which had issues in the past, and not measurements... sorry ... I see that 116 signs of 267 get processed correctly, but then an exception occurs, that is not handled inside the data collector... I will solve that now...

Piiit commented 2 years ago

@rcavaliere This should be solved now, please check the data to be sure... thanks

The problem was a constraint on the length of possible string measurements. It was not a regression, but just a coincidence that it was discovered now...

I tested with the following query:

select timestamp, * from station s 
join measurementstring m on m.station_id = s.id 
where s.active = true
and s.stationtype = 'VMS'
order by timestamp desc;

This gives dates from 2022-03-24, so it should be ok I guess... please verify

rcavaliere commented 2 years ago

@Piiit wonderful, it seems to be ok. Thanks for the clarification. Do you have this long string that caused the problem? So that I can provide this feedback to A22 and the other BrennerLEC partners.

Piiit commented 2 years ago

Unfortunately I do not have that string, because the DB did not insert it due to that constraint... however, now it should insert any string of any size... so in a few hours we should have that string again... let me check tomorrow

Piiit commented 2 years ago

Hehe... it came this moment :-)

String is this:

PARCHEGGIO MEZZI PESANTI ROVERETO SUD COMPLETO | PARCHEGGIO MEZZI PESANTI ROVERETO SUD COMPLETO | PARCHEGGIO MEZZI PESANTI ROVERETO SUD COMPLETO | PARCHEGGIO MEZZI PESANTI ROVERETO SUD COMPLETO | LKW PARKPLATZ ROVERETO SUD AUSGELASTET | LKW PARKPLATZ ROVERETO SUD AUSGELASTET | LKW PARKPLATZ ROVERETO SUD AUSGELASTET | LKW PARKPLATZ ROVERETO SUD AUSGELASTET

from: KM 140.605 SUD - component:1 with code A22:80:1

rcavaliere commented 2 years ago

@Piiit ah ok. Strange, it seems that they transmit the same message more than once. Probably a bug on their site on how they map this information on the API...

Piiit commented 2 years ago

Probably, but now on our side we are more robust with strings

rcavaliere commented 2 years ago

@Piiit CISMA has noticed correctly that when the DC was reactivated we were not able to integrate the historical data between March 1st and March 17th circa... it is strange since the DC should have a logic to start from the last record stored in the ODH core. Can we find a way to integrate the data related to this hole or is it too much effort?

rcavaliere commented 2 years ago

@Piiit we have also another issue with the VMS, see https://issues.opendatahub.bz.it/Ticket/Display.html?id=5592 It seems that in some cases for the VMS that should receive a numeric value (associated to a pictogram shown) we receive something like value | value | value. It's the same that raised the problem for the VMS that receives strings. What is the current situation now? Are we able to correctly parse the data, or what happens in these cases? A strategy could also be to talk to the Data Provider and check why in some cases we receive data like data and if they can fix this on their side

Piiit commented 2 years ago

@Piiit we have also another issue with the VMS, see https://issues.opendatahub.bz.it/Ticket/Display.html?id=5592 It seems that in some cases for the VMS that should receive a numeric value (associated to a pictogram shown) we receive something like value | value | value. It's the same that raised the problem for the VMS that receives strings. What is the current situation now? Are we able to correctly parse the data, or what happens in these cases? A strategy could also be to talk to the Data Provider and check why in some cases we receive data like data and if they can fix this on their side

@rcavaliere The logic of the DC works as follows at the moment: For each sign, it fetches all events. For each event, it fetches all components. It loops over such components and extracts for each a value "esposozione" or "stato". If for the same tuple road/sign/event/component one of these values already exist it concatenates these values with a pipe. This results in something like the following:

string_value                                                                                        |cname      |
----------------------------------------------------------------------------------------------------+-----------+
|                                                                                                   |esposizione|
|||                                                                                                 |esposizione|
|||||||                                                                                             |esposizione|
0                                                                                                   |esposizione|
0|0                                                                                                 |esposizione|
0|0|0                                                                                               |esposizione|
0|0|0|0                                                                                             |esposizione|
1                                                                                                   |stato      |
11                                                                                                  |esposizione|
1|1                                                                                                 |stato      |
1|2                                                                                                 |stato      |
15                                                                                                  |esposizione|
179                                                                                                 |esposizione|
18                                                                                                  |esposizione|
19                                                                                                  |esposizione|
2                                                                                                   |stato      |
203                                                                                                 |esposizione|
212                                                                                                 |esposizione|
22                                                                                                  |esposizione|
2|2                                                                                                 |stato      |
2|2|2                                                                                               |stato      |
2|2|2|2                                                                                             |stato      |
2|2|2|2|2|2|2|2                                                                                     |stato      |
23                                                                                                  |esposizione|
243                                                                                                 |esposizione|
29                                                                                                  |esposizione|
2 KM CODA CARPI ALLACC. A1 |                                                                        |esposizione|
32                                                                                                  |esposizione|
46                                                                                                  |esposizione|
63                                                                                                  |esposizione|
75                                                                                                  |esposizione|
A4 >>>VENEZIA RIENTRARE A VERONA SUD |                                                              |esposizione|
AREA DI SERVIZIO LAIMBURG OVEST INAGIBILE |RASTSTÄTTE LAIMBURG OVEST GESPERRT                       |esposizione|
 AREA DI SERVIZIO PO EST NON EROGA CARBURANTI |                                                     |esposizione|

If these values are not correct, we need to rewrite that part of the DC completely. Let me know...

Piiit commented 2 years ago

@rcavaliere all these numbers above are also stored as strings, because the DC sends them as text inside json, and not as a number

rcavaliere commented 2 years ago

@Piiit there is something strange here. This logic was introduced so to be able to import the translations of the texts, which are bilingual and provided in two different pages on the component, like

AREA DI SERVIZIO LAIMBURG OVEST INAGIBILE |RASTSTÄTTE LAIMBURG OVEST GESPERRT

I would remove this logic and just consider the first value provided in the first page of the component, so to avoid to have this.

Piiit commented 2 years ago

@Piiit CISMA has noticed correctly that when the DC was reactivated we were not able to integrate the historical data between March 1st and March 17th circa... it is strange since the DC should have a logic to start from the last record stored in the ODH core. Can we find a way to integrate the data related to this hole or is it too much effort?

@rcavaliere I had a look at the code, but the logic does not allow to get time frames of the past easily. It is closely connected to the current timestamp System.currentTimeMillis(), which would need a rewrite of that logic to be able to recover gaps. Should we do that? If yes, please create a new issue for that.

Piiit commented 2 years ago

@Piiit there is something strange here. This logic was introduced so to be able to import the translations of the texts, which are bilingual and provided in two different pages on the component, like

AREA DI SERVIZIO LAIMBURG OVEST INAGIBILE |RASTSTÄTTE LAIMBURG OVEST GESPERRT

I would remove this logic and just consider the first value provided in the first page of the component, so to avoid to have this.

I will do the following (DONE = already in production):

if the existing entry is the same as the new one, I will skip it --> to avoid strings like 1|1|1
if the existing entry is different, I will concatenate the strings --> so we can still have AREA DI SERVIZIO LAIMBURG OVEST INAGIBILE |RASTSTÄTTE LAIMBURG OVEST GESPERRT

Problems:

the problem here are mixed situations, that I will not cover for now (tell me if that should be covered as well) --> example: 1|2|1|2
What should we do with empty or space-only strings?

rcavaliere commented 2 years ago

@Piiit I think we need here two developments, but let me discuss this with Catch & Solve who developed the DC... For the problems of your last comment:

this should never happen. These messages are related to pictograms, and they don't change like text messages
we have to integrate also "empty" messages, these are also important to understand when was published what. In this case, you should avoid the check for a concatenation of the string.

Before starting implementing something, however, let me deepen with Catch & Solve and then decide how to proceed

Piiit commented 2 years ago

@rcavaliere Waiting for Catch/Solve

Piiit commented 2 years ago

Inserted historical data that was missing, now I will check the output of what has been changed in the past regarding string messages and their concatenation...

Piiit commented 2 years ago

@rcavaliere I fixed these issues now (on testingmachine) as follows:

Keep a single space string ' ' to avoid gaps in time series
Remove leading and trailing whitespace
Keep only one space in between
Do not concatenate duplicate strings
Avoid trailing pipes

I see that in the DB we still have some wrong concatenations in old string_values, should they be fixed as well? This would mean to have some SQL updates on the production environment

rcavaliere commented 2 years ago

@Piiit I don't know how much effort is to correct all historical values... if high, I wouldn't do it.

Piiit commented 2 years ago

It is a bit of work, because we must be very careful and query time might be quite high... maybe 3-4 hours of work

rcavaliere commented 2 years ago

ok, I would suggest to focus on other user stories now, I will suggest to keep the user story open and to complete it in a latter step

Piiit commented 2 years ago

@rcavaliere I create a separate issue for the historical data correction... everything done

noi-techpark / bdp-commons

As a project manager I would like to check the status of the DC VMS A22 #439