BKBASE-Plugin / flink-cdc-connectors

CDC Connectors for Apache Flink®
https://ververica.github.io/flink-cdc-connectors/
Apache License 2.0
0 stars 0 forks source link

Use SampleBucketSplitStrategy for shard collection with hashed keys #21

Closed vanliu-tx closed 1 year ago

vanliu-tx commented 1 year ago

For collection with hashed keys as shard key, the record counts in chunks are varied, some may contains millions of records, other may contains a few thousands.

This would lead to OOM exception during snapshot phrase if a chunk contains millions of records.

This feature use SampleBucketSplitStrategy instead of ShardedSplitStrategy to avoid this OOM situation.

vanliu-tx commented 1 year ago

fixed by #22