search

housepower / clickhouse_sinker

Easily load data from kafka to ClickHouse
https://housepower.github.io/clickhouse_sinker
Apache License 2.0
515 stars 118 forks source link

sinker consumer long lag messages cause container oom on the host #147

Closed zhengjianfei111 closed 1 year ago

zhengjianfei111 commented 2 years ago 
i find a scenario when finish a pressure test clickhouse_sinker container got killed by the system oom killer as the memory consumption overflow the memory limit set in deployment spec.  i read the sinker code but have no answer ,it seems that there is no flow control mechanism .   
i see there are two task control configuration,flushInterval and buffersize  ,but these only control sink speed ,if kafka topic message is too large,kafka-go task will keep read message and store in the buffer rings that will eventually crash container.   
so is there any way to control kafka read speed when sinker instance get high memory pressure.
zhengjianfei111 commented 2 years ago

buffersize is a good way to control one topic level memory consumption,but what if there are too many topic in task group, seems there is no global control

yuzhichang commented 1 year ago

@zhengjianfei111 go 1.19 introduced GOMEMLIMIT. Have a try.

yuzhichang commented 1 year ago

@zhengjianfei111 v3.0系列优化了内存和性能，试一下