taskforcesh / bullmq

BullMQ - Message Queue and Batch processing for NodeJS and Python based on Redis
https://bullmq.io
MIT License
6.24k stars 408 forks source link

[Bug]: Debounced job ID not being returned on new flow job creation #2780

Open fakhir opened 2 months ago

fakhir commented 2 months ago

Version

v5.12.12

Platform

NodeJS

What happened?

I had been using plain queue based jobs in the past, along with the debouncing feature. But recently I needed to create jobs with debouncing + parent/child jobs, so I switched to using Flows.

Now the problem that I'm facing is that when I call flowProducer.add() with the following options:

const jobNode = await flowProducer.add({
        name: myJobName,
        queueName: myQueueName,
        data: myJobData,
        opts: {
          debounce: { id: myDebounceId },
          removeOnComplete: 200,
          removeOnFail: 200
        },
        children: myChildJobs
      });

// jobNode.job.id here is always unique even for debounced jobs

The above call successfully debounces the job, but the returned jobNode.job.id is always a unique UUID identifier. If the job was debounced, then a job with the returned UUID doesn't exist in the queue, but the ID is still returned by the flowProducer.add() function. I am not specifying any job ID in the options, so I am relying on bullmq-generated job IDs.

I've been using similar code for debouncing jobs when using plain queues (instead of flows). When I call queue.add(), if the job is debounced, then the original job's ID is returned, which is very convenient. I would expect flows to behave in a similar manner, for consistency.

Reporting this as a bug, since I assume this isn't intentional behavior.

How to reproduce.

Create a flow based job using a flow producer using a debounce ID. And then create a second job using the same debounce ID (before the first job completes). The second job created will have a unique job id (based on UUID) even though it has been debounced and no such job has actually been created in the queue.

Relevant log output

No response

Code of Conduct

roggervalf commented 2 months ago

this functionality is not supported as this combinación implies also supporting more than one parent in a flow. We should throw an error in this case

fakhir commented 2 months ago

Okay, so this relates to the discussion under #2416.

Can this still be supported with a limitation, for example, that debouncing would only be performed for the parent job and not for child jobs? That would probably avoid the multi-parent condition.

roggervalf commented 2 months ago

it could work for the root parent and yes it's related to that issue