Air quality hourly averages computed by ODH for the A22 air quality low-cost sensors sometimes miss some data point in the time series

[rcavaliere]

See notification by CISMA here 20210520_CISMA_B4_realtime.pdf

[bertolla]

@rcavaliere They are missing for a simple reason that we defined together with CISMA. If there are not at least 30 datapoints, we do not calculate an average.

[rcavaliere]

@bertolla thanks for the reminder. OK, let me share this information with the other partner, I will let you know if we want to change this behavior

[rcavaliere]

@bertolla another question: how are the elaborations computed? Are we first making the averages and then the elaborations or viceversa?

[bertolla]

first the elaborations, since those are made by the datacollector which collects the data

[rcavaliere]

@bertolla I have had today a discussion with the technical partners of BrennerLEC about the criteria for not computing the hourly averages. They suggest to reduce the criterion to 16 datapoints, so to have more values in the time series. Can you please update this? Regarding the way elaborations are computed: from a mathematical point of view, the correct workflow would be the following, since the formulas are non linear: 1) compute the hourly averages applied to raw measurements 2) compute the non linear transformations (complex formulas) on the hourly averages (only) Would it be complicated to change the elaboration logic?

[bertolla]

@bertolla I have had today a discussion with the technical partners of BrennerLEC about the criteria for not computing the hourly averages. They suggest to reduce the criterion to 16 datapoints, so to have more values in the time series. Can you please update this?

We can do this, but doing it for data we already calculated means deleting all old data.

Regarding the way elaborations are computed: from a mathematical point of view, the correct workflow would be the following, since the formulas are non linear:

1. compute the hourly averages applied to raw measurements
2. compute the non linear transformations (complex formulas) on the hourly averages (only) Would it be complicated to change the elaboration logic?

Well yes, since they are 2 differenct modules who doesn't know of each other.

[rcavaliere]

@bertolla I would apply the new logic for the 16 datapoints only to the new data entering, without changing the previous elaborations. It's wonderful if you can change the flow of the the two elaborations.

[bertolla]

No, I can't. You asked me if it is complicated and I answered yes, so better not to change.

[rcavaliere]

@bertolla I am testing this right now, it looks better but in some situations I still see holes. See this example:

https://analytics.opendatahub.bz.it/#%7B%22active_tab%22:0,%22height%22:%22400px%22,%22auto_refresh%22:false,%22scale%22:%7B%22from%22:1623621600000,%22to%22:1624399200000%7D,%22graphs%22:%5B%7B%22category%22:%22Air%20quality%22,%22station%22:%22AUGEG4_AIRQ01%22,%22station_name%22:%22103.700_APPA%20BZ%22,%22data_type%22:%22NO2-Alphasense_processed%22,%22unit%22:%22%22,%22period%22:%223600%22,%22yaxis%22:1,%22color%22:0%7D%5D%7D

Can you confirm me that in these situations we have less than 16 datapoints?

[bertolla]

Sure, lookt at this: https://analytics.opendatahub.bz.it/#%7B%22active_tab%22:0,%22height%22:%22400px%22,%22auto_refresh%22:false,%22scale%22:%7B%22from%22:1623621600000,%22to%22:1624399200000%7D,%22graphs%22:%5B%7B%22category%22:%22Air%20quality%22,%22station%22:%22AUGEG4_AIRQ01%22,%22station_name%22:%22103.700_APPA%20BZ%22,%22data_type%22:%22NO2-Alphasense_processed%22,%22unit%22:%22%22,%22period%22:%223600%22,%22yaxis%22:1,%22color%22:0%7D,%7B%22category%22:%22Air%20quality%22,%22station%22:%22AUGEG4_AIRQ01%22,%22station_name%22:%22103.700_APPA%20BZ%22,%22data_type%22:%22NO-Alphasense_processed%22,%22unit%22:%22%22,%22period%22:%22600%22,%22yaxis%22:1,%22color%22:2%7D%5D%7D