search

pingcap / tidb-tools

tidb-tools are some useful tool collections for TiDB.
Apache License 2.0
287 stars 192 forks source link

Drainer 配置kafka 仅往一个 topic ,partition 发送 #188

Open kaiball9999 opened 5 years ago

kaiball9999 commented 5 years ago

使用 pump跟drainer 将tidb binlog 同步到kafka ... image 有几个问题： 1、仅往一个partition 写数据。 2、有可能单条记录很大，request超过限制大小，一条日志可以解析出多条操作日志。 image 3、官网文档没有比较详细的解释，kafka的参数太少 4、源码没有找到drainer的相关代码

july2993 commented 5 years ago

1: currently one transaction one msg, we may support to write multi partition and shard by PrimaryKey later. 2,3: one msg for one transaction, what do you means multi op log? you can try increase this now:

message.max.bytes=
replica.fetch.max.bytes=
fetch.message.max.bytes=

4: it's not open source now.

IANTHEREAL commented 5 years ago

@kaiball9999 any more problems and feedback on it？

pxuxian commented 5 years ago

1: currently one transaction one msg, we may support to write multi partition and shard by PrimaryKey later. 2,3: one msg for one transaction, what do you means multi op log? you can try increase this now:

message.max.bytes=
replica.fetch.max.bytes=
fetch.message.max.bytes=

4: it's not open source now.

第一个问题，按PrimaryKey shard，现在已经支持了吗？

july2993 commented 5 years ago

1: currently one transaction one msg, we may support to write multi partition and shard by PrimaryKey later. 2,3: one msg for one transaction, what do you means multi op log? you can try increase this now:

message.max.bytes=
replica.fetch.max.bytes=
fetch.message.max.bytes=

4: it's not open source now.

第一个问题，按PrimaryKey shard，现在已经支持了吗？

sorry, not yet