search

procrastinate-org / procrastinate

PostgreSQL-based Task Queue for Python
https://procrastinate.readthedocs.io/
MIT License
852 stars 52 forks source link

Actively notfiy a job of an abort request #1084

Open medihack opened 3 months ago

medihack commented 3 months ago

With PR #1081, we added a feature to mark a currently running job for abortion, but we have to poll the database in the task ourselves to determine whether this mark was set (represented by the aborting status). As discussed in #511, it would be nice if the task was actively notified of an abortion (without the need for intense database polling) using a PostgreSQL listen-notify. We could then raise an asyncio.CancelledError in an async task. It's unclear what to do with a sync task (maybe a callback method like context.listen_abort(cb: Callable?!). It should be optional and triggered by something like job_manager.cancel_job_by_id(33, abort=True, notify=True). It would be best implemented after the worker refactoring #933, as it must be integrated into the worker class.

ewjoachim commented 3 months ago

It would be best implemented after the worker refactoring https://github.com/procrastinate-org/procrastinate/issues/933

Or be part of the design of #933

ewjoachim commented 2 months ago

If we do #1083, is this worth the complexity ? I won't close this but I think it's interesting to spell out the gains we expect from this before tackling it.

medihack commented 2 months ago

In some cases, it could simplify the abortion process and save the user some code (e.g., no need for another asyncio task that monitors the abort status). On the other hand, it's not a feature that needs to have high priority or be integrated right now as part of the worker refactoring (the refactoring is complicated enough anyway). It's more a nice to have feature and something I have not seen in any other distributed task queue library. So I would say, let us leave this issue open and see what the future brings. Regarding #1083, it's the next thing I'll deal with.