Use dedicated capsule for processing sidekiq-alive queue

andrcuns commented 11 months ago

This introduces a separate capsule for handling sidekiq alive queue and makes it sequential because we don't really need to process multiple alive jobs in parallel.

I think it improves the design by not mutating "default" capsule
Uses capsule local connection pool
It should solve an issue I encountered in my own application, more details, more context here: https://gitlab.com/dependabot-gitlab/dependabot/-/issues/293#note_1408148243. TLDR; basically if an app fills in all threads that are configured for sidekiq and those jobs are long running, we can end up where alive job is not picked up for long enough and start reporting app as unhealthy.

andrcuns commented 11 months ago

@arturictus Could You please review this one and let me know what You think?

andrcuns commented 11 months ago

Did some testing in my app, and it seems we do need more than 1 connection in a pool. This is because we can end up calling unregister_current_instance for example concurrently.

I ended up getting ConnectionPool::TimeoutError: Waited 1 sec, 0/1 available error with just one connection in the pool

arturictus commented 11 months ago

This looks great!, thanks @andrcuns for this improvement.

arturictus / sidekiq_alive

Use dedicated capsule for processing sidekiq-alive queue #96