fpco / libraries

FP Complete libraries mega-repo
2 stars 0 forks source link

Periodically garbage-collect stale `activeKey`s #133

Closed bitonic closed 8 years ago

bitonic commented 8 years ago

Once #126 is merged, we can end up in this situation:

  1. A worker is detected dead by the heartbeat mechanism, and thus is cleaned up from the list of heartbeats
  2. The worker is actually still alive, and it grabs a request to process putting its request id in the activeKey
  3. The worker dies
  4. Now the worker is gone from the list of heartbeats, so the request id sitting in the worker's activeKey will be forgotten

This can be solved periodically collecting stale activeKeys.