search

biigle / core

:large_blue_circle: Application core of BIIGLE
https://biigle.de
GNU General Public License v3.0
12 stars 16 forks source link

Queue monitoring support #592

Closed mzur closed 11 months ago

mzur commented 1 year ago

We had a case on biigle.de where a volume with lots of annotations and large (tiled) images was cloned. This produced thousands of jobs to generate annotation patches. On tiled images each job can take many seconds or even a few minutes so the worker queue was completely bogged down by this. We had to add additional workers but we only found out about this issue by accident.

We should implement support for Laravel's built-in queue monitoring. This requires an event listener that sends a notification to the instance admins (as described in the Laravel docs). The queue:monitor command should also be added to the scheduled commands. Maybe not once each minute but every 10 minutes or so ("on one server").

I'd like to have the event handler to send only one email every hour and not each time the scheduled command runs.

Also there should be config options to adjust this. Instance admins should be able to enable or disable monitoring (default: disabled) as well as configure the queues and connections that should be monitored and their job thresholds.

mzur commented 1 year ago

This is now developed as https://github.com/biigle/laravel-queue-alert

mzur commented 11 months ago

This is implemented now.