possibility of group concurrency failure

[geofholbrook]

It is possible to defeat the group concurrency if jobs are added at the same time (within ~10ms). To reproduce:

const {QueuePro, WorkerPro} = require('@taskforcesh/bullmq-pro');

const queue = new QueuePro('test', {connection: {host: 'localhost', port: 6379}});

const processFn = async (job) => {
  console.log(`-> ${job.data.foo}`);
  await new Promise((res) => setTimeout(res, 1000));
  console.log(`\t<- ${job.data.foo}`);
};

new WorkerPro('test', processFn, {
  connection: {host: 'localhost', port: 6379},
  group: {
    concurrency: 1
  },
  concurrency: 100
});

queue.add('test', {foo: 'A-1'}, {group: {id: 'A'}});
queue.add('test', {foo: 'A-2'}, {group: {id: 'A'}});
queue.add('test', {foo: 'A-3'}, {group: {id: 'A'}});
queue.add('test', {foo: 'B-1'}, {group: {id: 'B'}});
queue.add('test', {foo: 'B-2'}, {group: {id: 'B'}});
queue.add('test', {foo: 'B-3'}, {group: {id: 'B'}});

Intermittently, I'm observing one group executing twice simultaneously (always group A which is submitted first), although the group concurrency is set to 1. This seems to be solved by using addBulk, or by adding a delay of 10ms, but it could still happen if two processes happen to submit a job within 10ms for the same group.

[manast]

Thanks for reporting this. We are investigating this.

[geofholbrook]

in case it helps, I've since discovered that this breaks with version 5.1.3 ... up to 5.1.2 works fine for me.

[manast]

This has been fixed on version v5.1.8. Please give it a try and let us know how it goes.

[geofholbrook]

Hi @manast sorry for the delay in answering ... yes, my tests pass now using the latest version 5.1.11 Thank you!

[manast]

Closing since the issue seems to have been resolved.