What happens when one pump instance crash?

pingcap / tidb-binlog

A tool used to collect and merge tidb's binlog for real-time data backup and synchronization.

Apache License 2.0

292 stars 131 forks source link

What happens when one pump instance crash? #717

Closed thuyenptr closed 5 years ago

thuyenptr commented 5 years ago

General Question

Hi, I'm researching tidb-binlog and confusing about this, for details:

What happens when Drainer make PullBinlogs(), at the same time Pump instance crash?
Has any mechanism prevented that threat?

Many thanks.

july2993 commented 5 years ago

What happens when Drainer make PullBinlogs(), at the same time Pump instance crash?
- It will retry and once the pump comes back drainer will work again, note drainer must pull data from all non-offline pump instance for merging the data.
Has any mechanism prevented that threat?
- If the crash pump can come back, drainer will work normally again, if not, you can unsafely remove the pump instance(change state to be offline directly using binlogctl) but will lost part of data in that instance

thuyenptr commented 5 years ago

Thank you for your reply.
Another question for Drainer, Drainer is a standalone instance so if it crashes, all synchronize process must be stopped until the instance come back, can we implement raft protocol to prevent this problem?

july2993 commented 5 years ago

do you means implement raft protocol to replicate data in pump? In theory， we can but there's no plan to do this now.

thuyenptr commented 5 years ago

yeah, in pump instance we can implement raft protocol, but I mean can we implement raft protocol in Drainer too?

july2993 commented 5 years ago

There's no data persistence needed by drainer (unless the downstream is backup file saved locally). If the downstream is MySQL/TiDB, it will be simple to start a new drainer in another node with some more simple solutions.

thuyenptr commented 5 years ago

thank you, I really appreciate all your help.

thuyenptr commented 5 years ago

do you means implement raft protocol to replicate data in pump? In theory， we can but there's no plan to do this now.

Continuing with the mechanism to prevent failure in pump instance, I think we don't need to implement raft protocol here, but we can use TiKV to storage log instead of the file system and Level DB. My question is: Can we use TiKV in pump instance?

Thank you.

july2993 commented 5 years ago

@billhcmus yes, we can just save data in TiKV for high reliability instead of saving in the local file system without replication. (but pump don't support this)

thuyenptr commented 5 years ago

why ping cap team did not choose TiKV for log storage? It effects on performance, right?