Add a new CommitterSetting that allows the user to choose whether to eagerly commit Committables as soon as they're observed (current behaviour), or once a new commit is observed (new behaviour inspired by cloudflow).
Details
A simple way to ensure that all transformed elements are processed when a source offset is transformed into many elements is to wait to commit an offset until the next Committable is observed. This ensures at-least-once guarantees when input elements are 1:M with output elements. The consequence is that committing will always be behind-by-1 and more records may need to be reprocessed after a failure/shutdown.
seglo
commented
4 years ago
Short description
Add a new
CommitterSettingthat allows the user to choose whether to eagerly commitCommittablesas soon as they're observed (current behaviour), or once a new commit is observed (new behaviour inspired by cloudflow).Details
A simple way to ensure that all transformed elements are processed when a source offset is transformed into many elements is to wait to commit an offset until the next
Committableis observed. This ensures at-least-once guarantees when input elements are 1:M with output elements. The consequence is that committing will always be behind-by-1 and more records may need to be reprocessed after a failure/shutdown.