search

nikepan / clickhouse-bulk

Collects many small inserts to ClickHouse and send in big inserts
Apache License 2.0
474 stars 86 forks source link

Option to periodically flush collected data from memory to the disk if needed. #43

Open rtarasov-mg opened 3 years ago

rtarasov-mg commented 3 years ago

This option could be helpful in cases when service might be killed with oom or unexpected reboot server, etc. To prevent losing all of the data collected in memory and guarantee delivery after recovery service. Useful new options:

  1. Enable flush to disk
  2. How often
  3. Retention policy Could this feature be implemented?
saikrishnakethan commented 1 year ago

Is this similar to Asynchronous Inserts ? https://clickhouse.com/docs/en/optimize/asynchronous-inserts

nikepan commented 1 year ago

Is this similar to Asynchronous Inserts ? https://clickhouse.com/docs/en/optimize/asynchronous-inserts

yes, it is almost same