open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
3.06k stars 2.36k forks source link

ClickHouse exporter produces duplicates and poor compression without sorting attributes #33634

Open JustinMason opened 4 months ago

JustinMason commented 4 months ago

Component(s)

exporter/clickhouse

Is your feature request related to a problem? Please describe.

The default table created by the exporter isn't a good pattern for optimizing compression and removing duplicates. ClickHouse does not sort the map values, so even though there may be duplicate records the order of their attributes may be different. This causes ClickHouse to treat them as unique records for storage and merge trees. This also effects ClickHouses compression so the same data takes up a lot more disk.

Describe the solution you'd like

We identified this issue and the solution was to use a NULL Engine for the primary table the Exporter writes to, then using a Materialized View we explicitly sort the attributes before insert. mapSort(Attributes) as Attributes,

After this the compression rate for billions of rows was greater than 250, making the storage needed much less. It also eliminated duplicates and helped streamline the increase functions so we could avoid extra processing.

This makes the initial table creation a bit trickier but it is critical in my experience.

Describe alternatives you've considered

No response

Additional context

No response

github-actions[bot] commented 4 months ago

Pinging code owners:

SpencerTorres commented 4 months ago

Interesting information, we actually have a PR open to update the table (#33611)

Considering the complexity of the materialized view required, it might be best to do this in the exporter code. Maps are unpredictable in Go, so we would need to convert it to a slice and sort it. Any thoughts on this approach?

hanjm commented 4 months ago

Yes, map sort can improve compression, clickhouse-go sdk support column.IterableOrderedMap https://github.com/ClickHouse/clickhouse-go/pull/1152, exporter can use this sdk type to write clickhouse map type with order, welcome a PR to try it.

crobert-1 commented 4 months ago

Removing needs triage based on response from code owners.

github-actions[bot] commented 3 weeks ago

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

hanjm commented 2 weeks ago

still valid

SpencerTorres commented 1 week ago

This can be closed once #35725 is merged