New processes not spawned with auto balancing, minProcesses=0 and a pending delayed job

[oprypkhantc]

• Horizon Version: 5.7.0
• Laravel Version: 8.29.0
• PHP Version: 7.4.15
• Redis Driver & Version: predis 1.1.6

Description:

auto balancer that has minProcesses set to 0 does not spawn new worker processes when the only job in the queue is a delayed one.

Steps To Reproduce:

1. Add this supervisor:

   'supervisor-main' => [
   'balance' => 'auto',
   'minProcesses' => 0,
   'maxProcesses' => 10,
   ],

2. Run horizon
3. Dispatch a delayed job

[driesvints]

Don't set your minProcesses to zero. That doesn't makes any sense. The latest Horizon version will even warn you about this.

[oprypkhantc]

@driesvints Having 9 different queues, are we supposed to have 9 processes always running locally, given minProcesses of 1? That's quite resource intensive given those 9 processes will be unused 99% of the time.

[paras-malhotra]

@oprypkhantc see https://github.com/laravel/horizon/issues/987#issuecomment-799457280. Min processes needs to be at least 1 to migrate delayed jobs.

[oprypkhantc]

@paras-malhotra I got it, that's what I'm doing right now. However, I don't get how to achieve a single process running instead of X for X different queues then. minProcesses affects all of the queue names listed, setting a minimum for each of them, which isn't ideal.

[mfn]

That's quite resource intensive given those 9 processes will be unused 99% of the time.

Is it really "resource intensive"?

AFAIK the following applies:

• php processes might look like the consume lots of memory, e.g. 100MB but most of it is shared libs
• the workers use blpop which means, ideally, most of the time a non-working worker waits on IO and doesn't do anything

Not arguing about your environments resource requirements, everyone has there own threshold, but in my experience the permanent resource usages for idle workers are very low.

[graemlourens]

@oprypkhantc this is also a question of how you designed / grouped your queues. There are various strategies (by purpose/category, by priority, by job execution speed etc) that can help you shape your horizon/queue config to have minimal 'wasted' idling queue workers. How it will work for you is depending on your setup.

In the beginning we also struggled with setting up the correct layout / grouping of queue workers and horizon supervisors. We ran into this issue of wanting to set minWorkers to zero because we had split our queues into too many different categories and therefore had nearly 20 horizon supervisors, with approx 60 standby queue workers. After lots of tweaking and learning the hard way it is not necessary to setup that fine grained queues and striking the right balance is important (which is highly dependant on your setup). And Btw: you'll only know if you setup your queues & priorities correctly when you start load-testing your application... We thought we had it right, and after the first test, had to change everything radically :)

In our case our workers consume between 22 - 32MB of memory when they are 'warm' which is acceptable for our scale (approx 200-300 jobs / second with approx 120 workers at peak.)

[oprypkhantc]

Thanks @graemlourens. We've decided we'll merge some of the queues we have, lowering the total of queues for that supervisor down to 1. The total would be minimum of 3-4 workers for all supervisors combined on dev env, that's totally fine and much better than current 13.

@mfn htop reports each worker's RES usage as 92MB, which shouldn't include shared memory. Experimentally, it looks more like each worker uses somewhere around 64MB judging from the total memory usage. Agreed, this isn't a lot. Didn't know it shared any memory, thanks.