search

pingcap / tidb

TiDB is an open-source, cloud-native, distributed, MySQL-Compatible database for elastic scale and real-time analytics. Try AI-powered Chat2Query free at : https://www.pingcap.com/tidb-serverless/
https://pingcap.com
Apache License 2.0
37.11k stars 5.83k forks source link

Some questions about the kafka binlog #6408

Closed lixiangdude closed 6 years ago

lixiangdude commented 6 years ago

Hi, there are some questions about binlog and kafka: 1.I used the local binlog before, and I found if the pump file expired, then start up drainer there will be something wrong. Now I use the kafka binlog, will it also has the same problem of message expire?

2.The kafka topic about pump only has 1 replica, will the corresponding binlog lost if the broker which retains this replica crashes?

3.If I want to change the kafka configuration, should I stop the tidb cluster first?

IANTHEREAL commented 6 years ago
  1. maybe, but I don't think it's a problem. Because a single node has limited storage capacity, it's more common to meet local binlog expired than Kafka, and u can control Kafka topic GC yourself.
  2. In fact one pump writes one topic(one partition). u can set default.replication.factor and min.insync.replicas of Kafka broker to get HA and consistency