search

nikepan / clickhouse-bulk

Collects many small inserts to ClickHouse and send in big inserts
Apache License 2.0
474 stars 86 forks source link

Q: is bulk a remedy for max_connections_count #29

Closed rsurgiewicz closed 4 years ago

rsurgiewicz commented 4 years ago

Hi there, Could you kindly confirm my initial thoughts about using your tool as a savior to my system?

I have a scenario where small inserts of data are posted to Clickhouse (e.g gps updates from a number of mobile devices). Often Clickhouse returns http 500 error due to max connection count reached or due to timeout. There are some MV that are being calculated on inserts so that might slow it down. I changed default value from 100 to 500 but it doesn't seem to help just more queries are waiting.

I thought that using your tool can improve the situation as bulk inserts are advised due to performance boost. Other option that I think of is usage of buffer tables.

Thanks!

nikepan commented 4 years ago

Yes, this tool is just a buffer for scripts that cannot save requests themselves

rsurgiewicz commented 4 years ago

Thanks Will it work if I have multiple inserts constructed with a number of values ? INSERT INTO TABLE a VALUES (v1,v2), (v3,v4), (v5,v6); INSERT INTO TABLE a VALUES (v7,v8), (v9,v10); Or only inserts with single value is supported/buffered?

Regards

nikepan commented 4 years ago

Yes. It works with multiple values

rsurgiewicz commented 4 years ago

Thank you! I'll be testing the tool!