In Lake Loader, there is an edge-case scenario where the time taken to "commit" events to the Lake becomes longer than the duration of the configured timed window.
Previously, because of how windows got timed, the edge case scenario caused the window sizes to get smaller and smaller as it waited for the events to be committed.
This PR changes how a window length is timed. After this PR, the processing time of a window remains constant, even if the "commit" phase starts to exceed the window size. It allows the loader to stay healthier even during edge-case scenarios.
In Lake Loader, there is an edge-case scenario where the time taken to "commit" events to the Lake becomes longer than the duration of the configured timed window.
Previously, because of how windows got timed, the edge case scenario caused the window sizes to get smaller and smaller as it waited for the events to be committed.
This PR changes how a window length is timed. After this PR, the processing time of a window remains constant, even if the "commit" phase starts to exceed the window size. It allows the loader to stay healthier even during edge-case scenarios.