search

nabicht / SimpleTaskQueue

A simple task queue used for coordinating distributed, parallel work.
4 stars 0 forks source link

the app hands out multiple open attempts for same task #65

Open nabicht opened 6 years ago

nabicht commented 6 years ago

This really shouldn't happen and it is happening occasionally. 2 attempts are getting handed out back to back for the same task, and based on the logs it is happening right on top of each other.

nabicht commented 6 years ago

As of flask 1.0 threaded=True by default when start a Flask app. This is causing each request to be its own thread and the TaskManager is not at all thread safe.

The quick and dirty fix to is to explicitly set threaded=False. This alleviates the problem for now but isn't the end all be all fix for a couple of reasons:

  1. Not every operation that the restful api does with TaskManager needs to be thread safe so for those I'd rather take advantage of threading when I can
  2. As this gets productionized (Flask really isn't meant to be a production server) then the quick and dirty fix no longer applies so need something more robust.

Everything is done just in python objects in-memory right now, with zero persistence. I'm possibly moving to sqlite as a way to get persistence. I need to double check everything before hand, but I think I can use sqlite as a way to be thread safe as well, which would be an added benefit.