search

nestjs / bull

Bull module for Nest framework (node.js) :cow:
https://nestjs.com
MIT License
589 stars 96 forks source link

On application shutdown it closes all the queues. If any active job is creating more jobs it will fail. #2069

Open akshatflx opened 3 months ago

akshatflx commented 3 months ago

Is there an existing issue for this?

Current behavior

We have a long-running Bull job which processes a big CSV file, validates each line and creates a new Bull job for every row in the CSV for further processing.

When we send SIGTERM signal to the process running this job, we ideally want that it should not process any new jobs, but still complete the pending job and then close the app.

But we observed that if we try to create any new jobs inside the active job after SIGTERM event, it fails with Error: Connection is closed redis error. We believe that's because queue.close() gets called on application shutdown - https://github.com/nestjs/bull/blob/197079daa3400b9d0b487a5ace7f27f12762b920/packages/bull/lib/bull.providers.ts#L43C3-L47C5

My suggestion is to do queue.pause(true) to only pause the processing of new jobs locally, but still allow adding new jobs. Or provide an option in the BullModuleOptions/BullModuleAsyncOptions to control this behaviour.

Minimum reproduction code

https://github.com/akshatflx/bull-graceful-shutdown

Steps to reproduce

  1. Install and run the app
  2. Copy the process ID from process listening on ... log
  3. Run the script sh test.sh <pid>

The test.sh script will publish a job in the "long-job" queue, which will internally start creating a job in the "internal-jobs" queue. It will create one job every second for the next 10 seconds. After 3 seconds the test.sh will kill the node process, which triggers the error.

Expected behavior

We shouldn't get the Connection is closed error. Instead the internal jobs should get created successfully. The node process should only stop processing any new jobs, it should still be allowed to add new jobs.

Package version

10.1.1

Bull version

4.12.2

NestJS version

10.0.0

Node.js version

20.10.0

In which operating systems have you tested?

Other

No response

kamilmysliwiec commented 3 months ago

Or provide an option in the BullModuleOptions/BullModuleAsyncOptions to control this behaviour.

This sounds like a useful addition. Would you like to create a PR for this?

akshatflx commented 3 months ago

Or provide an option in the BullModuleOptions/BullModuleAsyncOptions to control this behaviour.

This sounds like a useful addition. Would you like to create a PR for this?

@kamilmysliwiec raised a PR #2070 for this, looking forward to your feedback on it.