orchestracities / ngsi-timeseries-api

QuantumLeap: a FIWARE Generic Enabler to support the usage of NGSIv2 (and NGSI-LD experimentally) data in time-series databases
https://quantumleap.rtfd.io/
MIT License
38 stars 49 forks source link

Allow specify which attribute to apply the 'fromDate'/'toDate' filter #230

Closed jaimeventura closed 5 years ago

jaimeventura commented 5 years ago

Is your feature request related to a problem? Please describe. Sometimes our instances are updated in batch (several updates for the same entity). As an example, imagine that for some reason, the readings from a sensor were cached (in the sensor or midlware) and later sent to the context broker at '03/01/2019': {'entity_id': 'A', 'dateObserved': '01/01/2019', 'value': 5} {'entity_id': 'A', 'dateObserved': '02/01/2019', 'value': 3} {'entity_id': 'A', 'dateObserved': '03/01/2019', 'value': 8}

This will cause quantumleap to fill the 'time_index' column with '03/01/2019'. And so, if one asks quantumleap 'https://quantunleap/ve/entities/A?fromDate=01/01/2019&toDate=02/01/2019', it wont return any values.

A data consumer doesnt know when the values got into quantumleap, for him to change the query parameters

There are other reasons why an observation date is different from the time it got into quantumpleap.

Describe the solution you'd like Allowing specify which column to use for the 'fromDate'/'toDate' filter, would allow getting those values, regardless the date/time they got into quantumleap. When not specified, 'time_index' should be used.

Describe alternatives you've considered none, at the moment

Additional context Adding the functionality would keep backward compatibility

chicco785 commented 5 years ago

@jaimeventura the trick to solve your issue is to set as time_index attribute dateObserved

chicco785 commented 5 years ago

see the documentation for more details: https://quantumleap.readthedocs.io/en/latest/user/#time-index

jaimeventura commented 5 years ago

Oops, sorry about that. Last time i checked the documentation was a long time ago, i must admit. Its not what i was describing, but fits the purpose. Im closing this. Thank you,

chicco785 commented 5 years ago

@jaimeventura i know that is not exactly what you asked, there reason why we don't support arbitrary time attributes is that timeseries query optimization require to select a specific index, multiple ones are not supported (at least in crate)

jaimeventura commented 5 years ago

@chicco785 Sorry if it passed the wrong ideia. :) I strongly believe/agree the way it is implemented as the right way to solve the problem i presented.

By the way, is there an alternative channel to have community support (maillist, slack, ..). I have some ideias/doubts/... i would prefer to present other in than as "bug" or "feature request" in github.

Thanks

chicco785 commented 5 years ago

We have thought of other channels so far, because being a small team, we prefer to limit distractions. Anyhow, I opened a ticket #232 to discuss this internally.