search

ubccr / coldfront

HPC Resource Allocation System
https://coldfront.readthedocs.io
GNU General Public License v3.0
96 stars 76 forks source link

Django Q defaults to catch up mode resulting in possible duplicate emails #464

Open dougnd opened 1 year ago

dougnd commented 1 year ago

By default, if Django Q misses scheduled runs it will "catch up", by running the scheduled task as many times as it should have run. So if the qcluster was offline for 4 days, when it is turned back online it will run the daily tasks 4 times. With the current tasks in coldfront, this may mean duplicate emails.

A simple solution would be set catch_up to false in the Q_CLUSTER config. I'm happy to submit a PR, if this makes sense.

Of course, if the qcluster is offline some emails may be missed so we'll certainly try to not have it offline in production, but it may not always be online on our development tier (especially a local tier).

Thank you!

aebruno commented 1 year ago

@dougnd Thanks for reporting this. It does sound like the default behavior of missed schedules may not be desirable with our current implementation of daily email tasks. More than happy to accept a PR for this. Perhaps we just expose this as a config variable that defaults to false? This way sites can still turn on the catch_up behavior they wish. Thanks!