This is a partial and rough implementation of AndreKurait's idea that sequences bulk put responses into a flux that's effectively a progress report. We can then use that to monitor and update the last watermark that we've made it to for purposes of writing the work continuation record when a lease expires.
TODO:
Unit tests compile/pass
sendBulkRequest takes in the segment/docIds (at least the last one) & that's wired to the SegmentDocumentCursor
exitOnLeaseTimeout() is enhanced to do create a new work item from the lastDocIndexed
Make this easily exitOnLeaseTimeout() easily transferable to MIGRATIONS-2172
Category Enhancement
Why these changes are required? To support preserving backfill work even when a process is terminated.
What is the old behavior before changes and new behavior after changes? See Jira.
Testing
Tests break - this PR is to communicate the idea.
Check List
[ ] New functionality includes testing
[ ] All tests pass, including unit test, integration test and doctest
[ ] New functionality has been documented
[x] Commits are signed per the DCO using --signoff
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.
Description
This is a partial and rough implementation of AndreKurait's idea that sequences bulk put responses into a flux that's effectively a progress report. We can then use that to monitor and update the last watermark that we've made it to for purposes of writing the work continuation record when a lease expires.
TODO:
Unit tests compile/pass
sendBulkRequest takes in the segment/docIds (at least the last one) & that's wired to the SegmentDocumentCursor
exitOnLeaseTimeout() is enhanced to do create a new work item from the lastDocIndexed
Make this easily exitOnLeaseTimeout() easily transferable to MIGRATIONS-2172
Category Enhancement
Why these changes are required? To support preserving backfill work even when a process is terminated.
What is the old behavior before changes and new behavior after changes? See Jira.
Testing
Tests break - this PR is to communicate the idea.
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.