search

pingcap / tidb-binlog

A tool used to collect and merge tidb's binlog for real-time data backup and synchronization.
Apache License 2.0
292 stars 131 forks source link

drainer OOM in old cluster #1112

Open YuJuncen opened 2 years ago

YuJuncen commented 2 years ago

Bug Report

Please answer these questions before submitting your issue. Thanks!

From AskTUG: https://asktug.com/t/topic/274497

  1. What did you do? If possible, provide a recipe for reproducing the error. Start drainer in a cluster with long history.

  2. What did you expect to see? Drainer can startup (even slowly).

  3. What did you see instead? OOM. The peak memory usage is up to 64G. Note that the GlobalID in the cluster is 4593562(which means there probably are millions of DDL jobs).

  4. Please provide the relate downstream type and version of drainer. (run drainer -V in terminal to get drainer's version) v5.1.2

  5. Notes As a solution, we can use the CDC like method to resolve this: loading the snapshot and then send DDL jobs incrementally. Or, we can paginate the scan of history DDL jobs for reducing the peak memory usage.

lichunzhu commented 2 years ago

Possible solution: Use CDC's snapshot method and just load the following jobs. https://github.com/pingcap/tiflow/blob/c063509/cdc/entry/schema_storage.go#L728

kennytm commented 3 months ago

already "fixed" in #1237 i believe?