taskforcesh / bullmq

BullMQ - Message Queue and Batch processing for NodeJS and Python based on Redis
https://bullmq.io
MIT License
6.06k stars 396 forks source link

Is it possible to share a job between different flows ? #2573

Open guillaumeklaus opened 5 months ago

guillaumeklaus commented 5 months ago

On my current project I use bullmq (with NestJS) to queue jobs needed to generate a rather big pdf.

Each page of my pdf gathers informations from different db or api calls.

Some pages need the same first job, currently this first job is duplicated, my question is : is it possible to share this job between different flows ?

My code example :

   const firstPageGenerationFlow = await this.flowProducer.add({
      name: 'firstPageGeneration',
      queueName: 'db-api-calls',
      data: { someObject },
      children: [
        {
          name: 'mapStaticImageGeneration',
          queueName: 'db-api-calls',
          data: { someObject },
          children: [
            {
              name: 'parcelComplementaryInformations',
              queueName: 'db-api-calls',
              data: { someObject },
              children: [
                {
                  name: 'parcelRetrieval',
                  queueName: 'db-api-calls',
                  data: { someObject }
                }
              ]
            }
          ]
        }
      ]
    });
    const secondPageGenerationFlow = await this.flowProducer.add({
      name: 'secondPageGeneration',
      queueName: 'db-api-calls',
      data: { someObject },
      children: [
        {
          name: 'mapStaticImageGenerationUrbanism',
          queueName: 'db-api-calls',
          data: { someObject },
          children: [
            {
              name: 'getNearbyUrbanismPermits',
              queueName: 'db-api-calls',
              data: { someObject },
              children: [
                {
                  name: 'parcelRetrieval',
                  queueName: 'db-api-calls',
                  data: { someObject }
                }
              ]
            }
          ]
        }
      ]
    });

Both flows depend on parcelRetrieval, my goal would be to deduplicate this, any thoughts ?

manast commented 4 months ago

I think this is a case of the multiple parents feature request that some people have been requesting in the past. Unfortunatelly not supported. I am not sure if in your case you could solve this by having the job parcelRetrieval be a standard job, that as last thing it does is creating the two flows firstPageGeneration and secondPageGeneration, passing its result to both of them.