search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17.79k stars 1.57k forks source link

Clickhouse sink with field arrays scheme #905

Open ssfilatov opened 5 years ago

ssfilatov commented 5 years ago

There is a use case with clickhouse as a timeseries DB when metrics/events are stored in the following format: timestamp, ["key1", "key2"], ["value1", "value2"] original event: {"timestamp": <>, "event": {"key1": "value1", "key2": "value2"} These allows us to send events with arbitrary fields. Do you think vector can support this sink?

LucioFranco commented 5 years ago

@ssfilatov this seems like it would be a good spot for both a transform that can translate that nested json structure into a flat string. That then clickhouse could read and submit.