search

timescale / outflux

Export data from InfluxDB to TimescaleDB
Apache License 2.0
86 stars 22 forks source link

Outflux to compressed timseries (with Timescale 1.5) #77

Closed remiremi closed 3 years ago

remiremi commented 4 years ago

Hello,

The new column-oriented compression now makes timescaleDB a viable alternative to influxdb. It would however help to have outflux directly handle the compression. At the moment in my trials I run a command similar to this one after running outflux migrate for each of my measurement

ALTER TABLE co2_concentration SET (
  timescaledb.compress,
  timescaledb.compress_segmentby = 'factory_location,sensor_location'
);
SELECT add_compress_chunks_policy('co2_concentration', INTERVAL '1 day');

It's pretty long though, data is written to disk in uncompressed format and then is compressed, IO is a bit bottleneck, and I have to process measurements one by one to avoid running out of disk space.

Would be nice if outflux could insert directly in a compressed format.

PS: the "compress_segmentby" option in my use case is really easy to figure out, it's simply the tags for the measure

show tag keys on my_database FROM "{measure}"

I suspect this would be a good enough default for outflux?

bboule commented 4 years ago

@remiremi thanks for the feedback let me have a chat with the team and get back with you on this!!

mfreed commented 3 years ago

Sorry for lack of reply. Currently compressed chunks are immutable (although we plan to lesson that requirement in the future).

Although you seem to be asking something slightly different - it probably doesn't make sense to re-implement our customer compression format/algorithms in the outflux service, as opposed to doing it in the database as currently.

But we have heard desire to support direct import of compressed chunks (i.e., they are coming from another timescaledb database/archive), which we are considering for the future.

Going to close this as something where there is no current plans to implement, but feel free to open for future ideas.