Concurrency and Waiting Children

[bnason]

Is your feature request related to a problem? Please describe. As far as I can tell, when a job changes from active to waiting children, it is no longer counted against the concurrency limit.

Describe the solution you'd like I would like for it to count against it so only 1 job is running or waiting for children.

Here is a reproduction of the problem: https://github.com/bnason/bullmq-concurrency-children-issue As you can see in the logs, both parent jobs run and sit at waiting-children

BullMQ Concurrency Children Issue
Created parent queue
Created child queue
Created child worker
Created parent worker
Adding parent jobs
Added parent jobs
Created check interval
parent_job:1 - start parent processor
Parent initializing step
parent_job:1:child_job:0 - start child processor
Parent waiting step
Parent should wait
parent_job:2 - start parent processor
Parent initializing step
Parent waiting step
Parent should wait
parent_job:1:child_job:1 - start child processor
parent_job:1:child_job:2 - start child processor
parent_job:1:child_job:3 - start child processor
parent_job:1:child_job:4 - start child processor
parent_job:1:child_job:5 - start child processor
parent_job:1:child_job:6 - start child processor
parent_job:1:child_job:7 - start child processor
parent_job:1:child_job:8 - start child processor
parent_job:1:child_job:9 - start child processor
Parent status=1 isWaitingChildren=true
Parent status=1 isWaitingChildren=true
parent_job:1:child_job:10 - start child processor
parent_job:1:child_job:11 - start child processor
parent_job:1:child_job:12 - start child processor
parent_job:1:child_job:13 - start child processor
parent_job:1:child_job:14 - start child processor
parent_job:1:child_job:15 - start child processor
parent_job:1:child_job:16 - start child processor
parent_job:1:child_job:17 - start child processor
parent_job:1:child_job:18 - start child processor
parent_job:1:child_job:19 - start child processor
Parent status=1 isWaitingChildren=true
Parent status=1 isWaitingChildren=true
parent_job:1:child_job:20 - start child processor
parent_job:1:child_job:21 - start child processor
parent_job:1:child_job:22 - start child processor
parent_job:1:child_job:23 - start child processor
parent_job:1:child_job:24 - start child processor
parent_job:2:child_job:0 - start child processor
parent_job:1 - start parent processor
Parent waiting step
Parent is done waiting, all children should be finished
parent_job:2:child_job:1 - start child processor
parent_job:2:child_job:2 - start child processor
parent_job:2:child_job:3 - start child processor
parent_job:2:child_job:4 - start child processor
Parent status=2 isWaitingChildren=false
Parent status=1 isWaitingChildren=true
parent_job:2:child_job:5 - start child processor
parent_job:2:child_job:6 - start child processor
parent_job:2:child_job:7 - start child processor
parent_job:2:child_job:8 - start child processor
parent_job:2:child_job:9 - start child processor
parent_job:2:child_job:10 - start child processor
parent_job:2:child_job:11 - start child processor
parent_job:2:child_job:12 - start child processor
parent_job:2:child_job:13 - start child processor
parent_job:2:child_job:14 - start child processor
Parent status=2 isWaitingChildren=false
Parent status=1 isWaitingChildren=true
parent_job:2:child_job:15 - start child processor
parent_job:2:child_job:16 - start child processor
parent_job:2:child_job:17 - start child processor
parent_job:2:child_job:18 - start child processor
parent_job:2:child_job:19 - start child processor
parent_job:2:child_job:20 - start child processor
parent_job:2:child_job:21 - start child processor
parent_job:2:child_job:22 - start child processor
parent_job:2:child_job:23 - start child processor
parent_job:2:child_job:24 - start child processor
Parent status=2 isWaitingChildren=false
Parent status=1 isWaitingChildren=true
parent_job:2 - start parent processor
Parent waiting step
Parent is done waiting, all children should be finished

[manast]

The concurrency factor establishes how many jobs are allowed to run concurrently within one worker. If a work is not active, then it is not being considered for the concurrency factor. So I do not think there is an issue here, as the job is waiting and not active.

[bnason]

Is it possible to add an option to count it as "active". For my use case, it is still active, even if technically in bullmq its waiting instead. I'm trying to look at the code to see where I could potentially do that but I'm not seeing anything obvious yet.

[manast]

@bnason No, that would be really difficult to do as the concurrency mechanics are very tight coupled with the active status.