Improves/Fixes TableBackedProgressManager

[hartmut-co-uk]

Hi, I've been playing, testing and debugging with this lib + simple-printer example, also making use of TableBackedProgressManager + NewPeriodicProgressReporter in particular.

For some unexpected behaviour I came up with improvements/fixes I wanted to share and maybe discuss. I've added my current findings + a new example into this PR.

Changes included

• adds method Empty(..) to ChangeConsumer (allows for marking CDC log stream time as processed, fixes 'bug' where second start of a reader begins reading CDC log on generated_created timestamp instead of ChangeAgeLimit / actual latest processed state)
• Simplify AdvancedReaderConfig options [PostEmptyQueryDelay, PostFailedQueryDelay] -> single 'PostQueryDelay' which is also considered when calculating next PollWindow
• adds new example 'simple-printer-stateful' which correctly maintains the CDC reader's state using TableBackedProgressManager + NewPeriodicProgressReporter
• adds some commented out 'logging' lines for debugging purposes

Options for further improvements:

• [ ] gracefully shutdown the reader+consumers allowing to mark/save current progress before exit (~call consumer.End() / reporter.SaveAndStop(ctx context.Context))

Any questions, please reach out!

[hartmut-co-uk]

cc: @avelanarius, @piodul

[hartmut-co-uk]

cc: @haaawk

[hartmut-co-uk]

Latency (from 'write to table' ... 'Reader.Consume()') when the reader has fully caught up can be calculated as simple as Latency = ConfidenceWindowSize + PostQueryDelay

e.g. for

Advanced: scyllacdc.AdvancedReaderConfig{
    ChangeAgeLimit:       15 * time.Minute,
    ConfidenceWindowSize: 10 * time.Second,
    PostQueryDelay:       5 * time.Second,
    PostFailedQueryDelay: 5 * time.Second,
    QueryTimeWindowSize:  5 * 60 * time.Second,
},

=> Latency: 15s *with queries against scylla every 5s (PostQueryDelay) per [reader::table::vnodeId] ... -> with query windows of 5s **also on initial startup or when behind + catching up QueryTimeWindowSize is used allowing for faster progress -> with query window size 5min (with above example).

[hartmut-co-uk]

thx @piodul - good to see some movement on the CDC front... I'm still interested to contribute / help with feedback, but I'm quite out of touch with this PR, so I'll need some focus time to review again myself... :-) I'll try to make time soon.

[hartmut-co-uk]

@piodul shall I close this PR? Or would you want to reuse anything else or some of the suggestions? e.g.

Options for further improvements:

• gracefully shutdown the reader+consumers allowing to mark/save current progress before exit (~call consumer.End() / reporter.SaveAndStop(ctx context.Context))

[dkropachev]

@hartmut-co-uk , thanks for your contribution, this project recently have changed it's maintainers, can you please let me know if you are willing to continue work on this PR