search

noi-techpark / it.bz.opendatahub.api.mobility-ninja

The new home of the Open Data Hub Mobility API v2
Other
2 stars 0 forks source link

Provenance ID of measurements should be exposed through the ODH API #28

Closed rcavaliere closed 2 years ago

rcavaliere commented 2 years ago

Use case: same station with measurements provided by different sources will be differentiated through this parameter

Piiit commented 2 years ago

Several tables reference provenance: event, measurement*, measurementhistory*... in total 7 tables

Piiit commented 2 years ago

@rcavaliere Done, everything on testing ...

We have now a provenance for events and one for each measurement table

rcavaliere commented 2 years ago

@Piiit looks good. Question: how will we manage the field sorigin if we have multiple Data Collectors providing the same stations?

Piiit commented 2 years ago

I suggest to deprecate that field in the future and solely use the lineage field inside the provenance object

Piiit commented 2 years ago

The problem with stations that have multiple origins are that we do not have a history table for stations, just for their measurements. This is a general design flaw in the ODH. For example, if a station gets updated, we do not store the old data in a history table, therefore if we access a measurement of one year ago, the associated station has only the most recent data, that might not reflect the station data of one year ago.

The metadata has history tables, but the actual API does not expose that. This is not easily possible because the measurement table has only the station as foreign key and not that metadata record, so we would need to match two timestamps to find the right metadata.

rcavaliere commented 2 years ago

@Piiit I don't see that critical, in the sense that what we expose is the metadata of the station provided by the "most reliable" source. Since this includes parameters like name, position, etc. this should be not change too much if we have, like in this case, multiple sources for the same stations. So, having the field sorigin is in my opinion important to provide the information about the source we have chosen for the metadata. I therefore recommend that if we have another source station metadata are in general not updated, but can only "append" new measurements, that are characterized by a different provenance ID.

Piiit commented 2 years ago

OK, but at the moment we have data collectors that rapidly change metadata fields of stations... we would need to check them and eventually change that behavior...