DurhamARC / ManyFEWS

The Many Flood Early Warning System
GNU General Public License v3.0
7 stars 3 forks source link

Implement locking for tasks to prevent race conditions #89

Open sjmf opened 1 year ago

sjmf commented 1 year ago

Problem:

Multiple instances of the same task can be served to celery. For example, loadParameters is a long running task. If it is kicked off in the interface multiple times, or scheduled too closely together, another task should not start to run while the existing one is running.

Solution: A locking mechanism will need to be implemented in the application to stop tasks from colliding in this way. For example, http://loose-bits.com/2010/10/distributed-task-locking-in-celery.html and https://docs.celeryq.dev/en/latest/tutorials/task-cookbook.html#ensuring-a-task-is-only-executed-one-at-a-time