Writer will continue dequeue all items if it couldn't meet its expected seq number. It will cause blocking queue fails to control, while writer's map will bloat; Plus, writer always sleep there;
Current sequence mechanism is very fragile. Writer always expect a sequence but preprocessor may fail and thus lose this sequence.
Solution:
Sliding window may help here.
On each preprocessor dequeue task, it will insert a sequence into sliding window to occupy the location.
A map is maintained in sliding window, only when its most left item filled with success or error, the window will slide to the right.
Only when the window finds the left most item is filled with susccess/error item, it will notify writer to dequeue;
Here the window slides, and thus move one empty space. Then we allow following preprocessor threads to put item into the window. And it will notify
Background:
Solution: Sliding window may help here. On each preprocessor dequeue task, it will insert a sequence into sliding window to occupy the location. A map is maintained in sliding window, only when its most left item filled with success or error, the window will slide to the right. Only when the window finds the left most item is filled with susccess/error item, it will notify writer to dequeue; Here the window slides, and thus move one empty space. Then we allow following preprocessor threads to put item into the window. And it will notify