search

open-infrastructure-labs / telemetrywg

Repository for the Open Infrastructure Labs Telemetry Working Group
1 stars 1 forks source link

Long term storage for Jaeger and Loki #4

Open tumido opened 3 years ago

tumido commented 3 years ago

Related to: https://github.com/operate-first/apps/issues/232 for the Causter logging for DeCorus

Discussed influxDB IOx and ElasticSearch

cc @jpkrohling @RS1999ent @drbwa @eranra

jpkrohling commented 3 years ago

To give a bit more context, @RS1999ent mentioned a way to get historical data in a format that can be used at a later point in time, to which I mentioned that InfluxDB IOx will store data following the Apache Parquet format, allowing other tools to read the data. The files themselves can be stored using any block solution, like S3 for retention over longer periods of time. We also talked about Jaeger's possible plan to move to InfluxDB IOx in the future as the recommended storage mechanism, moving away from Elasticsearch. Whether that makes sense for other signals (logs, metrics), I don't know.

drbwa commented 3 years ago

I am aware of the trend to move to InfluxDB and maybe that is indeed the right thing to do.

From our experience, Elasticsearch allows for more convenient retrieval when it comes to log files.

What I don't know (yet) is if a) InfluxDB IOx will make it easier to slice and dice and retrieve your logs and b) how much more efficient InfluxDB is in terms of required storage space (guessing there it has an advantage there, but don't know if that's the case).