When we resume a changefeed, we only pass the checkpoint to the changefeed aggregators, not the changefeed frontier. This means that if the frontier writes a checkpoint before it receives updates covering the previous checkpoint from the aggregators, it could overwrite the checkpoint with less information (i.e. the checkpoint can regress). This is undesirable because we would end up losing saved progress for no good reason. An easy solution would be to just pass the checkpoint to the changefeed frontier too or we can have the frontier read it off the job progress.
blathers-crl[bot]
commented
4 hours ago
When we resume a changefeed, we only pass the checkpoint to the changefeed aggregators, not the changefeed frontier. This means that if the frontier writes a checkpoint before it receives updates covering the previous checkpoint from the aggregators, it could overwrite the checkpoint with less information (i.e. the checkpoint can regress). This is undesirable because we would end up losing saved progress for no good reason. An easy solution would be to just pass the checkpoint to the changefeed frontier too or we can have the frontier read it off the job progress.
Relevant code comment: https://github.com/cockroachdb/cockroach/blob/1caa3e4ca9d92b98287fe5996f4c3cfa7dd42e3e/pkg/ccl/changefeedccl/changefeed_processors.go#L1280-L1296
Jira issue: CRDB-42593