HWCloudEngine / storage-gateway

storage gateway to adapt cloud storage to replicate/snapshot/backup to other different cloud storage.
7 stars 10 forks source link

client sequence mechanism is error prone #20

Closed yinweiishere closed 7 years ago

yinweiishere commented 7 years ago

Background:

  1. Writer will continue dequeue all items if it couldn't meet its expected seq number. It will cause blocking queue fails to control, while writer's map will bloat; Plus, writer always sleep there;
  2. Current sequence mechanism is very fragile. Writer always expect a sequence but preprocessor may fail and thus lose this sequence.

Solution: Sliding window may help here. On each preprocessor dequeue task, it will insert a sequence into sliding window to occupy the location. A map is maintained in sliding window, only when its most left item filled with success or error, the window will slide to the right. Only when the window finds the left most item is filled with susccess/error item, it will notify writer to dequeue; Here the window slides, and thus move one empty space. Then we allow following preprocessor threads to put item into the window. And it will notify

yinweiishere commented 7 years ago

sequential blocking queue, on going.

yinweiishere commented 7 years ago

patch ready for merge

yinweiishere commented 7 years ago

patch committed.