search

jaegertracing / jaeger-clickhouse

Jaeger ClickHouse storage plugin implementation
Apache License 2.0
236 stars 50 forks source link

Document and add support for deleting data/TTL #43

Closed pavolloffay closed 2 years ago

pavolloffay commented 3 years ago

We should document how the old data can be removed (alter table jager_spans drop partition 20201) and add support for TTL https://clickhouse.tech/docs/en/sql-reference/statements/alter/ttl/ (The user could specify the number of days in the config).

E.g. 

CREATE TABLE IF NOT EXISTS jaeger_index_local (
     timestamp DateTime CODEC(Delta, ZSTD(1)),
     traceID String CODEC(ZSTD(1)),
     service LowCardinality(String) CODEC(ZSTD(1)),
     operation LowCardinality(String) CODEC(ZSTD(1)),
     durationUs UInt64 CODEC(ZSTD(1)),
     tags Array(String) CODEC(ZSTD(1)),
     INDEX idx_tags tags TYPE bloom_filter(0.01) GRANULARITY 64,
     INDEX idx_duration durationUs TYPE minmax GRANULARITY 1
) ENGINE MergeTree()
PARTITION BY toDate(timestamp)
ORDER BY (service, -toUnixTimestamp(timestamp))
TTL timestamp + INTERVAL 90 DAY
SETTINGS index_granularity=1024

cc) @chhetripradeep could you please loop in and document how do you delete old data?

pavolloffay commented 2 years ago

See as well https://github.com/pavolloffay/jaeger-clickhouse/pull/34#discussion_r678140425

EinKrebs commented 2 years ago

I'm gonna add option for ttl in table init scripts.

pavolloffay commented 2 years ago

+1 we should make TTL configurable in the config.

pavolloffay commented 2 years ago

@EinKrebs is there a place where we could chat together? Could you please join CNCF slack https://cloud-native.slack.com/messages/CHY2THYUU and join #jaeger or #jaeger-clickhouse channel or just ping me directly on slack?

EinKrebs commented 2 years ago

@pavolloffay Unfortuately, it seems like I can't use CNCF slack. Maybe Telegram?

pavolloffay commented 2 years ago

Or do you use gitter?

EinKrebs commented 2 years ago

No, I don't. But, I guess, I can try using it.

pavolloffay commented 2 years ago

Try to ping me there directly, use github username