search

orchestracities / ngsi-timeseries-api

QuantumLeap: a FIWARE Generic Enabler to support the usage of NGSIv2 (and NGSI-LD experimentally) data in time-series databases
https://quantumleap.rtfd.io/
MIT License
38 stars 49 forks source link

Initial Timescale DB support #228

Closed c0c0n3 closed 5 years ago

c0c0n3 commented 5 years ago

The problem

Crate DB's support for window functions and advanced queries is still in its infancy. Ditto for storage and querying of spatial data. Moreover, most of the existing advanced features are only available in the enterprise edition. All this makes it hard for us to implement time series analytics that go just beyond the basics.

Proposed solution

Implement an additional storage backend on Timescale DB. Timescale offers more open source window functionality than Crate, advanced time series support (e.g. hypertables) and, best of all, is implemented as a Postgres extension. This means we get a rock-solid, battle-tested, open source database, Postgres that is. Moreover, we can also plonk in the PostGIS extension to provide advanced spatial functionality. In a nutshell, this new backend we'd like to support is:

Timescale       PostGIS
-----------------------
        Postgres

For the Crate fans out there: this doesn't mean we're going to ditch Crate. Quite the opposite actually. Timescale and Crate will be both supported as QuantumLeap backends going forward. Using a configuration file, you'll be able to specify which backend QuantumLeap should use at runtime based on the tenant. For example say you have two tenants, T and V, and you configure QuantumLeap to use Crate for tenant T and Timescale for V. When T's data comes in, it'll get stashed away in Crate whereas V's data will wind up in Timescale.

Alternatives

We looked at Influx and Rethink in the past and even implemented initial support for them---see the modules in src/translators. Back in the day, Crate seemed to be a better option, so we went full steam ahead with Crate support and left all the rest behind.

c0c0n3 commented 5 years ago

@dherykw: @taliaga mentioned your interest in a Timescale QuantumLeap backend. Get in touch if you'd like to contribute or just find out more about our plans :-)

c0c0n3 commented 5 years ago

closed by #231 , #237