orchestracities / ngsi-timeseries-api

QuantumLeap: a FIWARE Generic Enabler to support the usage of NGSIv2 (and NGSI-LD experimentally) data in time-series databases
https://quantumleap.rtfd.io/
MIT License
38 stars 49 forks source link

How to deal with nulls? #204

Open c0c0n3 opened 5 years ago

c0c0n3 commented 5 years ago

Quantum Leap may store a null attribute value even if the attribute wasn't originally sent by the device. This is especially so in batch notifications where we take the union of all the attributes for each entity type in the batch---see #191.

How should a DB null be interpreted? As it stands now it could be either because the device sent an attribute with an empty value or it could be we added that null ourselves.

This may not be a problem if we add a query parameter to filter nulls out. The only problem I can think of would be the following, but am not sure this is actually something that could happen in practice---my IoT experience is very limited, but @chicco785 @pooja1pathak may care to comment.

Say I have a device with two sensors, one for temperature and the other for pressure, and say it sends temperature measurements every minute whereas pressure gets measured only every second minute. I may wind up with entity data that looks like this:

pressure:    null    4   null  null   ...
temperature:  10    15    16    18    ...
time:          1     2     3     4    ...

The nulls at time points 1 and 3 where introduced by Quantum Leap whereas the one at time point 4 was the result of the device failing to measure pressure so it sent us an empty attribute. Now say I'd like figure out the pressure sensor fault rate. Just looking at nulls won't cut it nor can I filter out all the nulls in any meaningful way...

github-actions[bot] commented 3 years ago

Stale issue message