Kafka Checkpoint - Githubissues

pingcap / tidb-binlog

A tool used to collect and merge tidb's binlog for real-time data backup and synchronization.

Apache License 2.0

292 stars 130 forks source link

Kafka Checkpoint #738

Closed thuyenptr closed 4 years ago

thuyenptr commented 5 years ago

Hi,

I have a question about the checkpoint Why you don't persist checkpoint in a Topic in Kafka instead of saving checkpoint in a file (local checkpoint) (code here) I think we can lose checkpoint if drainer die.

Thank you.

july2993 commented 5 years ago

yes, if lost the checkpoint file, we lost the checkpoint. I think maybe we can just consume the latest msg the fetch the checkpoint.

thuyenptr commented 5 years ago

Yeah, I think we should save the checkpoint in Kafka instead of saving in a file. Can I make PullRequest for this? Btw, I am implementing raft protocol for pump instance in pump cluster, can you give me some advice?

july2993 commented 5 years ago

Sure, but I think we can't change the behavior for compatibility, but I think we can add some config option in syncer.to.checkpoint to controller how to save the checkpoint, default save at file for dest=kafka for compatibility.
Well, that will take a lot of work, do you do this for using it later or for personally interest？

thuyenptr commented 5 years ago

Sorry, I don't understand what compatibility mean here, can you explain it?
For using later. Currently, I use TiDB-Binlog to replicate data to another TiDB cluster for business analysis, for detail we must run reconcile for accounting system online. TiDB is HTAP database, we experimental TiDB for all transactional and analytic but the result was terrible. So we decide using TiDB-Binlog to replicate to downstream.
Come back with my problem, I want to increase availability for pump cluster because if 1 pump instance down then the replicate pipeline will stop, we must be manual fix this instance. In my opinion, I can use a raft to increase the availability and fault-tolerant for pump cluster. Currently, I have no idea when pump instance crash situation.

july2993 commented 5 years ago

I means don't surprise user if they upgrader drainer version and the checkpoint ts is got from other place and different from the ts saved in file before.
TiFlash will help, but it's not GA now.
Maybe just try to use some availability storage volumes now.

thuyenptr commented 5 years ago

Thank you, I will try to separate the storage layer and processing layer in pump instance first, then make availability storage