Move potentially long-running tasks to background jobs

webrecorder / browsertrix

Browsertrix is the hosted, high-fidelity, browser-based crawling service from Webrecorder designed to make web archiving easier and more accessible for all!

https://browsertrix.com

GNU Affero General Public License v3.0

178 stars 33 forks source link

Move potentially long-running tasks to background jobs #1898

Open tw4l opened 3 months ago

tw4l commented 3 months ago

We have a few API endpoints where it would be better to respond 200 early and move the logic of the endpoint into a background job that can be queued and retried if necessary.

Specifically,areas where this would be helpful:

[x] Org deletion (same note as above)
[ ] Recalculating org storage
[ ] Org import (possibly?)

tw4l commented 4 days ago

Org deletion merged in #2098, which adds a new main_by entrypoint for background jobs that need access to ops methods. Org stats and import have not yet been moved over, so keeping this issue open for now.