search

taskforcesh / bullmq

BullMQ - Message Queue and Batch processing for NodeJS and Python based on Redis
https://bullmq.io
MIT License
6k stars 387 forks source link

Keep order of jobs in group upon retry #2774

Open freiondrej-lokalise opened 2 weeks ago

freiondrej-lokalise commented 2 weeks ago

I am using group mode with per-group concurrency = 1. Let's say I add three jobs in it, job A, job B and job C, to be executed as A-B-C. Job A is being executed first. However, when it fails and is being retried, the order changes to B-C-A. I'd like the per-group order to be kept as A-B-C and only if A exhausts its attempts, it would be discarded, so only B-C would be executed.

I tried to use per-group priority for this, but didn't find a reliable way of setting the priority accordingly to my needs. What I tried was to use queue.getGroupJobsCount() when adding to the queue and incrementing it by 1, but it seems it's not fully reliable as sometimes the count is not up to date, so I end up with two identical priorities which then does not work as I'd need.

Maybe a new option would be needed for this, something like workerOptions: {group: {concurrency: 1, keepOrderOnRetry: true}}?

roggervalf commented 2 weeks ago

hi @freiondrej-lokalise, we only support priorities intra groups, it may be possible to use priorities for groups but it may imply a breaking change. We are in the building the new breaking change for bullmq that also will affect pro version, in that new breaking change could be included

freiondrej-lokalise commented 1 week ago

Thanks for your comment @roggervalf ! 🙌

we only support priorities intra groups, it may be possible to use priorities for groups but it may imply a breaking change

I'm not suggesting to use priorities for groups, only to allow configuring how prioritization intra groups is handled. Let me know if I should elaborate or illustrate with a better example 🙏

roggervalf commented 1 week ago

hey, I misunderstood you. In reality I worked on some ideas for this case https://github.com/taskforcesh/bullmq/pull/2465 or https://github.com/taskforcesh/bullmq/pull/2687, we are still evaluating which is the best way to handle it. Also I have one extra approach but probably we will revisit this after our next breaking change that we are currently working