TiDB Binlog high delay

[thuyenptr]

Hi, I'm testing TiDB Binlog in real-server but got high delay metrics reported. Detail:

• Binlog Reach Duration By Pump metrics: greater than 1 second

• Transaction execution (sql query Time) greater than 1 second

• Then the Checkpoint delay (p99) is 8.7 min and avg is 2.0 min

My Drainer configuration is:

....
txn-batch = 1000
worker-count = 48
flag: --cache-binlog-count=6553600

• So after 1 second Binlog reach to Drainer, but drainer need > 7s to execute this transaction? Is it right?

Hope your help.

[july2993]

seems there's lag at that time near 15:10, may meet high throughput at that time at up stream or run a slow DDL? you can check Drainer Event and ddl at drainer dashboard for the speed it handled DDL and DML events at that time.

• suggest use a lower txn-batch (<= 100)
• don't config the cache-binlog-count, just use the default (with too high value may causes OOM when there's lag), it should be ok to just use the default value after #737 or v3.04

[thuyenptr]

Thank you so much.