Simple Yandex ClickHouse insert collector. It collect requests and send to ClickHouse servers.
Download binary for you platorm
or
or from sources (Go 1.13+):
git clone https://github.com/nikepan/clickhouse-bulk
cd clickhouse-bulk
go build
For example:
INSERT INTO table3 (c1, c2, c3) VALUES ('v1', 'v2', 'v3')
INSERT INTO table3 (c1, c2, c3) VALUES ('v4', 'v5', 'v6')
sends as
INSERT INTO table3 (c1, c2, c3) VALUES ('v1', 'v2', 'v3')('v4', 'v5', 'v6')
{
"listen": ":8124",
"flush_count": 10000, // check by \n char
"flush_interval": 1000, // milliseconds
"clean_interval": 0, // how often cleanup internal tables - e.g. inserts to different temporary tables, or as workaround for query_id etc. milliseconds
"remove_query_id": true, // some drivers sends query_id which prevents inserts to be batched
"dump_check_interval": 300, // interval for try to send dumps (seconds); -1 to disable
"debug": false, // log incoming requests
"dump_dir": "dumps", // directory for dump unsended data (if clickhouse errors)
"clickhouse": {
"down_timeout": 60, // wait if server in down (seconds)
"connect_timeout": 10, // wait for server connect (seconds)
"tls_server_name": "", // override TLS serverName for certificate verification (e.g. in cases you share same "cluster" certificate across multiple nodes)
"insecure_tls_skip_verify": false, // INSECURE - skip certificate verification at all
"servers": [
"http://127.0.0.1:8123"
]
},
"metrics_prefix": "prefix"
}
CLICKHOUSE_BULK_DEBUG
- enable debug loggingCLICKHOUSE_SERVERS
- comma separated list of serversCLICKHOUSE_FLUSH_COUNT
- count of rows for insertCLICKHOUSE_FLUSH_INTERVAL
- insert intervalCLICKHOUSE_CLEAN_INTERVAL
- internal tables clean intervalDUMP_CHECK_INTERVAL
- interval of resend dumpsCLICKHOUSE_DOWN_TIMEOUT
- wait time if server is downCLICKHOUSE_CONNECT_TIMEOUT
- clickhouse server connect timeoutCLICKHOUSE_TLS_SERVER_NAME
- server name for TLS certificate verificationCLICKHOUSE_INSECURE_TLS_SKIP_VERIFY
- skip certificate verification at allMETRICS_PREFIX
- prefix for prometheus metrics./clickhouse-bulk
and send queries to :8124
manual check main metrics
curl -s http://127.0.0.1:8124/metrics | grep "^ch_"
ch_bad_servers 0
- actual count of bad serversch_dump_count 0
- dumps saved from launchch_queued_dumps 0
- actual dump files id directorych_good_servers 1
- actual good servers countch_received_count 40
- received requests count from launchch_sent_count 1
- sent request count from launchFor better performance words FORMAT and VALUES must be uppercase.