search

nikepan / clickhouse-bulk

Collects many small inserts to ClickHouse and send in big inserts
Apache License 2.0
474 stars 87 forks source link

combine chproxy and clickhouse-bulk #20

Open yang3808282 opened 4 years ago

yang3808282 commented 4 years ago

I want to add bulk features to chproxy，can you help me how to do it?

nikepan commented 4 years ago

How I can help you? You want import clickhouse-bulk as module or implement this functionality?

yang3808282 commented 4 years ago

How I can help you? You want import clickhouse-bulk as module or implement this functionality?

thanks for your reply, I want to implement this functionality

nikepan commented 4 years ago

As example, you can add bulk before chproxy

asfaltboy commented 4 years ago

I have a different package and I'd love to import clickhouse-bulk to do exactly the same (while kipping the interface slim). Would you be interested in that sort of refactoring, and consider a PR perhaps?

nikepan commented 4 years ago

What do you want to change? Can you send pr or needed changes?

nikepan commented 4 years ago

Возможно стоит по русски) У меня сейчас не особо много времени делать какой-то большой объем работе. Но если изменения небольшие, могу сделать. Если что-то большое, то надо смотреть, что нужно.

asfaltboy commented 4 years ago

Basically I wanted a plug and play solution to use to batch up calls the client, e.g via https://github.com/mailru/dbr .

I ended basically mimicking the collector.go module, collecting and doing batch insertions of records instead of generating the query text. The solution is quite a bit shorter, but not very generalized (my "Table.Rows"field is a slice of my "ClickhouseRow" struct).

Do you think something like that has place in clickbouse-bulk?

nikepan commented 4 years ago

I think that such solutions are quite private. And yet I do not understand what should result? It seems to me that collecting requests from structures is much faster than processing it with a parser in collector.go.