yuzhichang
commented
3 years ago https://github.com/housepower/clickhouse_sinker/releases/tag/v1.8.6 stops task gracefully. Please have a try.
The normal procedure to handle schema change is:
- add some fields to Clickhouse table
- wait clickhouse_sinker consume up old records (consume lag be zero)
- restart clickhouse_sinker pods
- publish new records to kafka
We expierienced that after POD restart in kubernetes OLD records were read (we added some fields therefore we had to restart pod to reread config task JSON and update clickhouse table fields).
Are OS signals (SIGHUP,...) handled the way to minimize data coruption (stoping kafka consume after first OS stop signal, waiting until all backpoll data is processed and zookeeper commit)?