search

go-graphite / carbon-clickhouse

Graphite metrics receiver with ClickHouse as storage
MIT License
186 stars 47 forks source link

Dynamic Uploader connection #52

Closed Felixoid closed 4 years ago

Felixoid commented 4 years ago

Hello. During huge data migration from the Whisper to ClickHouse I've recognized that the connection to CH is permanent. Because of that, we have an unbalanced loading: only one of 8 CH nodes under an LB receives all data and then redistribute it with a sharding key.

Implementing a mechanism to recognize shouldn't be a hard task, but I'm not sure about the overhead for a new connection per insert. Maybe, implement a config parameter for it?

lomik commented 4 years ago

I assume that the carbon-clickhouse and the graphite-clickhouse are on the same server as the clickhouse. In this configuration, you can use external load balancers for write (carbon-c-relay) and read (carbonapi)

Implementing a mechanism to recognize shouldn't be a hard task, but I'm not sure about the overhead for a new connection per insert. Maybe, implement a config parameter for it?

Overhead will be very small and optional parameter is not required. PR are welcome

Felixoid commented 4 years ago

Solved with 56fceb9