sync-diff-inspector: avoid eating too many memory

[july2993]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.

It's easy to split a huge chunk. having already 380G memory still oom, this is unacceptable.

fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x1915ed4, 0x16)
        /usr/local/go/src/runtime/panic.go:774 +0x72
runtime.sysMap(0x117c4000000, 0x4000000, 0x2e7acd8)
        /usr/local/go/src/runtime/mem_linux.go:169 +0xc5
runtime.(*mheap).sysAlloc(0x2e60f20, 0x4000, 0x41f2a2, 0x7ff63cc03200)

2. What did you expect to see?

   • better split chunk
   • do not oom even if a chunk contains too many rows.
   • we can fetch data by order by ... limit ... can compare step by step(we can split chunk dynamic doing this?).

3. What did you see instead?

4. What version of TiDB are you using (tidb-server -V or run select tidb_version(); on TiDB)?

5. which tool are you using? sync-diff

6. what versionof tool are you using (pump -V or tidb-lightning -V or syncer -V)?

[july2993]

cc @WangXiangUSTC

[kennytm]

@july2993 please specify which tool you are talking about

[july2993]

@july2993 please specify which tool you are talking about

sync-diff, updated in the description.