Use long windows when catching up

scylladb / scylla-cdc-java

Apache License 2.0

24 stars 15 forks source link

Use long windows when catching up #67

Open avelanarius opened 2 years ago

avelanarius commented 2 years ago

Changes the strategy the CDC library uses to generate windows to read the CDC log. Previously, when a library was started, it would read queryTimeWindowSize large windows starting from |now - ttl|. The issue with that approach is that it requires a large number of windows to catch up, for example: 24 hours / 30 seconds 256 8 = 5898240 for a 8-shard Scylla server.

A new approach just creates a single large window from |now - ttl| to |now - confidence window|.

avelanarius commented 2 years ago

v2: fixed flaky unit test. (off-by-one error)

racevedoo commented 2 years ago

Any news here? I'm pretty interested in this PR