Closed denisov-vlad closed 1 year ago
Hey, we had same issue with clickhouse-bulk and all over unstable work but we did increased config parameters and problems are gone:
{
"listen": ":8124",
"flush_count": 10000,
"flush_interval": 3000,
"clean_interval": 0,
"remove_query_id": true,
"dump_check_interval": 300,
"debug": false,
"dump_dir": "dumps",
"clickhouse": {
"down_timeout": 120,
"connect_timeout": 30,
"tls_server_name": "",
"insecure_tls_skip_verify": false,
"servers": [
"http://*******:8123"
]
},
"use_tls": false,
"tls_cert_file": "",
"tls_key_file": ""
}
So my advice to increase the flush_interval, down_timeout and connect_timeout and observe.
@JohnDoeDC Hi! I've thought about that solution. Maybe such a big timeout would hide the real problem?
@denisov-vlad Sorry for long response, I believe connect timeout and down timeout doesnt make any impact but flush count and flush interval does. When we had default settings there was thread starvation from clickhouse-server, about 16k threads. While now with quieres like every 1-2 seconds there is only a few threads (may be 1-2)
ps -T -p 3734613 | grep HTTPHandler | wc -l
3
3734613 is a clickhouse-server pid
Increasing flush_count / flush_interval helped. Thanks!
The last lines in log file:
After that, data isn't sent to the CH server and the pod's RAM has increased over time.
I've checked app status and it looks ok: