open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
3.04k stars 2.35k forks source link

[question] Is there a way to load balance the exportation of telemetry data to multiple Clickhouse instances using the endpoint param `connection_open_strategy` in exporter/clickhouse? #29155

Closed ceevaaa closed 11 months ago

ceevaaa commented 11 months ago

Component(s)

exporter/clickhouse

Description

Is there a way to balance the exportation load (client-side load balancing) to multiple Clickhouse DBs while using the exporter/clickhouse.

As explained in this configuration here, will it be possible to use this parameter in the endpoint param to achieve the same ?

Something like this -

clickhouse:
    endpoint: http://addr1:port,addr2:port?connection_open_strategy=round_robin
    database: otel

Thanks & Regards Shiva Pundir

Collector version

v0.0.88

github-actions[bot] commented 11 months ago

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

crobert-1 commented 11 months ago

Hello @ceevaaa, from what I'm reading in the code this should be possible.

All data types (logs, metrics, traces) go through the same call stack of creating a ClickHouse client. The client will use the endpoint provided in the config to build a DSN and then open the DB connection. During the buildDSN method all parameters of the provided endpoint will be encoded into the DSN url. (The specified endpoint is validated on startup to make sure it can properly be parsed by ClickHouse).

I may be missing something here though, I don't have much experience with this component, so the code owners may have to correct me here. Have you been able to test it to see if it's working as expected?

ceevaaa commented 11 months ago

Hey @crobert-1, I tested this out today. And it worked well! So as usual, thanks again for being the query solving superman. Cheers!

Regards Shiva Pundir 3u04h5