When I use group, Can I use priority together?

[for2gles]

I'm currently using 2.4.14 version of BullMQ Pro.

And I want to use group concurrency with job's priority.

If I try like below

Code

const worker = createWorkerPro<TestJobData>(
  queueName,
  async job => {
    if (job.data.priority !== 5) {
      console.log(`higher priority ${job.data.priority} processed first`);
    }
    await Bluebird.delay(2 * 1000);
  },
  {
    group: {
      concurrency: 1,
    },
    concurrency: 500,
  },
);
await queue.add(
  `test-${groupKey1}`,
  { groupKey: groupKey1, priority: 5 },
  { jobId: uuidv4(), group: { id: groupKey1 } },
);
await queue.add(
  `test-${groupKey2}`,
  { groupKey: groupKey2, priority: 5 },
  { jobId: uuidv4(), group: { id: groupKey2 } },
);

But if then, creating job logic ignore the group option, and it makes to use full of concurrency

[roggervalf]

hi @for2gles, currently this is not possible, you should or use priority or groups. By using priority option, groups will be ignored

[for2gles]

Hi @roggervalf, then when do you think is it possible to use both option at once? Actually, I need both at once to use BullMQ Pro. So, it would be nice if there's plan for this. Thank you.

[manast]

@for2gles could you please share a bit more about your use case so we understand better what you want to achieve?

[for2gles]

@manast We are getting data from other system and they have their own rate limit rule by each specific key.

So actually, what we really need is group-key + rate limiter(group-key) + priority(global or group). But as I know, because of some reason, we can't use priority with rate limiter from Bull MQ (link)

So, for alternative, we thought to use group + concurrency(group) + priority + our own rate limiter. So we tried above and found priority seemed not work.

[manast]

We are getting data from other systems and they have their own rate limit rule for each specific key.

Hmm, how does priority help in this case? wouldn't you need a rate limit that is configurable per group instead?

[for2gles]

Ah, there are some data that important and fast response and other that less important and slow response. So normally fast response requests are done. But during processing slow response requests, some time we need to get fast response request first using higher priority.

[manast]

@for2gles we will study today if it is feasible to implement priorities for groups, I will update this issue later.

[for2gles]

Thank you!

[manast]

@for2gles just to clarify, do you need a priority inside every group, meaning that groups are processed in order but the jobs within the group are prioritized or do you need a global priority, meaning that the groups are picked not in order but based on a priority?

[for2gles]

global priority, meaning that the groups are picked not in order but based on a priority

@manast What we need is second option

But anyway after the group picked, it also should processed in order based on a priority

[manast]

@manast What we need is second option

But in that case, you do not need groups at all. You can just use the global priority from BullMQ, unless I am missing something 🤔

[manast]

For example, let's say that you have groups with priorities A(p1), B(p2), C(p3), and D(p4), then it would first process all jobs from A, then from B, then from C and finally all from D. So this is the same as just not having groups and using priorities on the jobs.

[for2gles]

That's If I run the worker(global concurrency: 1) and each group has only one job. I want to use global concurrency like 500 and group concurrency as 1.

For example1 A(p1, p2, p2, p3), B(p2, p3), C(p3), and D(p3, p4) and global concurrency: 4, group concurrency 1

It should process

1. A-p1, B-p2, C-p3, D-p3
2. A-p2, B-p3, D-p4
3. A-p2,
4. A-p3

For example 2 A(p1, p2, p2, p3), B(p2, p3), C(p3), and D(p3, p4) and global concurrency: 2, group concurrency 1

It should process

1. A-p1, B-p2
2. A-p2, B-p3
3. A-p2, C-p3
4. A-p3, D-p3
5. D-p4

[manast]

Those examples were quite useful, thank you. The case of having only intra-group priority is much more simple to implement than having also an inter-group priority. So we could do the former, but adding inter-group is more challenging so we need to discuss it. Without inter-group your examples would look like this: Example 1:

• A-p1, B-p2, C-p3, D-p3
• A-p2, B-p3, D-p4
• A-p2,
• A-p3

Example 2:

• A-p1, B-p2
• C-p3, D-p3
• A-p2, B-p3
• D-p4, A-p2
• A-p3

[for2gles]

Oh thank you. It will really useful for me The Best for us is the example2 that I wrote above but what you wrote also very useful!

[maxichar74]

Hi, This feature would be very useful for me too. Do you have some approximate date when this feature would be accessible ?

[for2gles]

Hi @manast. Can you let me know is any progress about it?

[manast]

We have started an implementation but it is quite complex, so we are working on other smaller issues the reminder of this year. We will pick up again this feature if you are still interested in the beginning of the next year.

[for2gles]

Hi, we are still really interested in this feature. I hope Taskforce also interested in :)

[for2gles]

@manast Hi there :) Just want to know can you start to implement this feature 🙂 We will really get help from this. BTW, We already started to use the BullMQ Pro 2 weeks ago. Thanks

[manast]

It is the next feature coming to the Pro version, but it is a bit involved so it is taking more time than I expected.

[roggervalf]

Hi guys, be aware that we are actively working on local group priorities

[vlrevolution]

Hey, did this ever get implemented?

[roggervalf]

hi @vlrevolution, intra groups priority was implemented https://docs.bullmq.io/bullmq-pro/groups/prioritized some time ago

[vlrevolution]

hi @vlrevolution, intra groups priority was implemented https://docs.bullmq.io/bullmq-pro/groups/prioritized some time ago

Amazing, I'm using them, thank you!