getJobCounts reporting incorrect waiting job numbers when using groups?

[nlochschmidt]

I've been trying to report on the total currently waiting jobs when using groups. During testing I found out that the waiting jobs reported by getJobCounts('waiting') don't make much sense.

I would have expected the count to be the total number of jobs waiting to be processed. My second best assumption would have been that it represents the number of jobs that are ready to be taken up at any moment. Both assumptions seem incorrect.

What I get instead for most of the time, is a fixed 1, independent of number of tasks or group concurrency 🤔

Example code:

import { QueuePro, WorkerPro } from "@taskforcesh/bullmq-pro";

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

describe("Job counts with groups", function () {
  it("return sensible count of waiting jobs", async function () {
    const queue = new QueuePro("test", { connection: { host: "localhost" } });
    let count = 0;

    for (let index = 0; index < 10; index++) {
      await queue.add("task", {});
      await queue.add("task", {}, { group: { id: "group-1" } });
      await queue.add("task", {}, { group: { id: "group-2" } });
      await queue.add("task", {}, { group: { id: "group-3" } });
    }

    const { waiting, active } = await queue.getJobCounts("waiting", "active");
    console.log(
      `Before worker is running: ${waiting} waiting, ${active} active`,
    );

    const worker = new WorkerPro(
      "test",
      async () => {
        await sleep(1000);
        count = count + 1;
      },
      {
        connection: { host: "localhost" },
        concurrency: 5,
        group: {
          concurrency: 5,
        },
      },
    );

    do {
      await sleep(200);
      const { waiting, active } = await queue.getJobCounts("waiting", "active");
      console.log(
        `While worker is running: ${waiting} waiting, ${active} active`,
      );
    } while (count < 40);

    await worker.close();
  });
});

Output

Before worker is running: 10 waiting, 0 active
While worker is running: 5 waiting, 5 active
While worker is running: 1 waiting, 5 active
While worker is running: 1 waiting, 5 active
While worker is running: 1 waiting, 5 active
While worker is running: 1 waiting, 5 active
....
While worker is running: 0 waiting, 5 active
While worker is running: 0 waiting, 5 active
While worker is running: 0 waiting, 5 active
....
While worker is running: 0 waiting, 0 active

Is this the expected behavior?

Coming back to my original intent, is there any way I can get the total waiting jobs without iterating over all groups? We might have ~100.000k groups at times.

BullMQ Pro version: 5.2.0

[manast]

This behavior is by design, when jobs are placed in groups they are not available on the wait list. Only one job can exist on the wait list which is used for triggering workers without relying on polling. As the jobs are placed in groups, we do not currently have any way to know how many jobs are in total other than summing the counts for every group. I guess this could be solved by having a dedicated counter for grouped jobs, where the counter is increased everytime a job is added to the queue and decreased when the job is consumed 🤔

[manast]

Btw, a possible cheap way to get an estimate of number of jobs waiting could be to use the DBSIZE command, which returns the total number of keys in a redis database, of course a lot of assumptions must be made for this to work, as there could only be one queue in said Redis host, and completed, failed, delayed and active job counts must be substracted: https://redis.io/commands/dbsize/

[nlochschmidt]

Thanks for clarifying. Maybe if more people stumble over this it might be good to add a warning to the documentation of getJobCounts

We have multiple queues on the redis host and we already track DBSIZE, but it's unfortunately not granular enough to be used for monitoring.

[nlochschmidt]

I found another bug in my test case and opened a separate issue (#48).

I reran the above test case but without dashes in the group names just to confirm that it doesn't change anything with regards to the waiting counts, which it doesn't.

[geofholbrook]

This is an issue for us as well ... group concurrency is the main feature that prompted us to go pro :) but, we have this problem with visibility, since grouped jobs do not appear as waiting (actually, to me it seems that one job at a time appears, the rest are invisible).

we do not currently have any way to know how many jobs are in total other than summing the counts for every group

If we know the group ids, is there some method for getting the counts per group?

[roggervalf]

Hi @geofholbrook, we have https://api.bullmq.pro/classes/Queue.html#getGroupJobsCount and https://api.bullmq.pro/classes/Queue.html#getGroupsCount

[manast]

@geofholbrook you should also be able to see all the groups in Taskforce.sh. If you are using the connector then you need to install the Pro connector that is also available in your account.

[hicaro]

This behavior is by design, when jobs are placed in groups they are not available on the wait list. Only one job can exist on the wait list which is used for triggering workers without relying on polling. As the jobs are placed in groups, we do not currently have any way to know how many jobs are in total other than summing the counts for every group. I guess this could be solved by having a dedicated counter for grouped jobs, where the counter is increased everytime a job is added to the queue and decreased when the job is consumed 🤔

@manast Should I expect one job count per each existing group, then? I have four groups, which each has ten jobs; however, when I call getWaitingCount, I only get the count of one instead of four. The same is true for getWaiting; it returns just one job.

The same is valid if I pause the queue.

The following is what I get when I call getGroups:

[
  {
    "id": "123-entity.type-1",
    "status": "waiting"
  },
  {
    "id": "123-entity.type-2",
    "status": "waiting"
  },
  {
    "id": "456-entity.type-1",
    "status": "waiting"
  },
  {
    "id": "456-entity.type-2",
    "status": "waiting"
  }
]

This is the job count breakdown:

{
  active: 0,
  completed: 0,
  delayed: 0,
  failed: 0,
  paused: 0,
  prioritized: 0,
  waiting: 1,
  'waiting-children': 0
}

[manast]

You can use getGroupJobsCount to get the count of jobs for a given group: https://api.bullmq.pro/classes/v6.Queue.html#getGroupJobsCount and getGroups to get the groups' IDs: https://api.bullmq.pro/classes/v6.Queue.html#getGroups Furthermore, you can also inspect all this data with Taskforce.sh, as it supports groups as well.

[hicaro]

You can use getGroupJobsCount to get the count of jobs for a given group: https://api.bullmq.pro/classes/v6.Queue.html#getGroupJobsCount and getGroups to get the groups' IDs: https://api.bullmq.pro/classes/v6.Queue.html#getGroups Furthermore, you can also inspect all this data with Taskforce.sh, as it supports groups as well.

I still think that this is a bug. One thing that I forgot to mention in my previous message is that all groups dig not belong to the same job. I had 2 jobs, which each had two groups. I was expecting to have a count of 2 at least since those were distinct jobs.

I did try Taskforce.sh. I paused the queue and added 6 jobs as follows:

Job	Group	Count
type-1	123-entity.type-1	1
type-2	123-entity.type-2	1
type-1	456-entity.type-1	2
type-2	456-entity.type-2	2

Taskforce was not displaying the correct amounts in the group section either. All groups were there, but group 456-entity.type-1 showed a count of 1 instead of 2. At first, I thought there was an issue with adding the jobs to the group, but when I resumed the queue, all 6 jobs got processed, meaning that one job was not being counted.

While in that paused state, both getGroupJobs('456-entity.type-1') and getGroupsJobsCount() returned the incorrect count, 1 and 5 respectively. This tells me that the issue is not with Taskforce.sh but with Bull MQ Pro.

We're very excited to use the pro license of BullMQ, but I don't know if we will also have buy-in for the dashboard. We've been using the bullboard, and folks seem happy with it. If the counts match there (at least one unit per job type), it'll be easier to sell on the idea of using BullMQ Pro since the majority of our use cases rely on visual inspection of the board (we have lots of manually triggered batch processing jobs).

[sakari]

hi, I'm running into this issue. I understood from the above that queue.getGroupsJobCounts would produce the amount of jobs in all queues and we could use that to figure out the total number of waiting jobs (waiting + the group jobs count). However queue.getGroupsJobCount does not seem to produce the expected number of jobs that are in the queue.

      const group = 'group';
      const queueName = `queue-${Math.random() * 100_000}`;
      const queue = new QueuePro(queueName, { connection: redisConnectionConfiguration });
      new WorkerPro(
        queueName,
        async () => {
          // make the processing get stuck on the first job so that the second job gets put to waiting
          await new Promise(() => {});
        },
        { group: { concurrency: 1 }, connection: redisConnectionConfiguration },
      );
      await queue.add('foo', 1, { group: { id: group } });
      await queue.add('foo', 2, { group: { id: group } });
      while (true) {
         // this goes to zero, where the second job is?
        const waitingPerGroup = await queue.getGroupsJobsCount();
        // this shows the active job
        const waitingJobs = await queue.getGroupJobs(group);
        // this has only the active first job
        const allJobs = await queue.getJobs();
        // only the active job is visible here
        const states = await queue.getJobCounts();
      }

It seems that the second job gets lost somewhere as it is not visible in the getGroupsJobsCount nor in getJobCount.

This is not good as we would like to monitor the sizes of our queues. What do.

[sakari]

well. seems that getGroupJobsCount does return better numbers. Would I be correct in thinking that I can sum waiting and that number for all groups to get the amount of waiting jobs

[manast]

getGroupsJobsCount will return the total number of jobs that are waiting in all the groups, but there will most likely 1 job on the wait list too.

[JZhang502]

Hi, running into a similar issue when we are trying to get more visibility for grouped jobs.

While we are able to use getGroupsCountByStatus to see the # of groups in each state, we haven't found a efficient way to get jobs in each group. The current idea is first getting all the groups via getGroups, and then looping through getGroupJobs for each group and filtering for jobs in the desired state (ie. waiting). Which I think is inefficient since our queue has upwards of 50k jobs for which most belongs to groups.

Wondering if it is possible to implement a getJobs function that can filter on status for the pro version that grabs all the jobs (both grouped and ungrouped), similar to the non-pro version here. Also open to any other suggestions!

[manast]

@JZhang502 By definition the jobs that are in a group are in the waiting status, or rate-limited. Individual jobs inside a group cannot have different statuses.

By the way, have you tried to examine groups in the taskforce.sh dashboard, I think that would give you a good picture of the groups and their jobs.