Drainer's memory usage exponentially increases when filtering too many DELETE events, leading to OOM

[kennytm]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.

   TiDB -> Pump -> Drainer -> MySQL

   1. On upstream, DELETE many rows only in every transaction, repeat for many times
   2. On drainer, use the binlog-filter-rule to filter out all DELETE events, so nothing will be replicated.

2. What did you expect to see?

   • Synchronize normally.

3. What did you see instead?

   • The memory usage of drainer grows exponentially and eventually triggered kernel OOM.

4. Please provide the relate downstream type and version of drainer. (run drainer -V in terminal to get drainer's version)

   • Both Drainer v5.4.1 and Drainer v6.5.3.

[kennytm]

Config:

[syncer.binlog-filter-rule]
[syncer.binlog-filter-rule.rule1]
ignore-event = ["delete"]

[[syncer.table-migrate-rule]]
binlog-filter-rule = ["rule1"]
source = { schema = "db", table = "tbl1" }

[[syncer.table-migrate-rule]]
binlog-filter-rule = ["rule1"]
source = { schema = "db", table = "tbl2" }

...

[kennytm]

Heap profile (DM me if you need the actual pprof file)