cfpb / qu

:warning: This project was archived on September 25th, 2020 and is no longer maintained
Creative Commons Zero v1.0 Universal
368 stars 80 forks source link

Aggregation worker dies; only fixable by Qu restart #231

Open marcesher opened 9 years ago

marcesher commented 9 years ago
  1. Start an aggregation
  2. Kill mongos
  3. Restart mongos

Actual behavior: The worker dies, because its error handler itself throws an error when trying to reset the job because it can't communicate with mongo

When this happens, that Qu instance's aggregation worker is effectively dead and will not process additional aggregations. This results in a buildup of aggregations until that Qu instance is restarted

Expected behavior:

  1. The error handler should not throw an error, or, if it does, the worker should be restarted
  2. The worker's currently processing job, at the time of failure, would get restarted when a) the worker is restarted and b) able to communicate with mongo successfully
marcesher commented 9 years ago

@m3brown found this, which may be useful in supervising the worker: https://github.com/MichaelDrogalis/dire