[Bug]: Job completed but the underlying process is still running

[alirizwan]

Version

5.10.3

Platform

NodeJS

What happened?

I am using BullMQ to manage FFMPEG and other notifications. FFMPEG is running as sandbox jobs and these works can run for days. In sandbox job I am doing something similar to this:

const command = ffmpeg(url).run();

while my worker looks like this:

const cameraProcessor = pathToFileURL(__dirname + '/processes/camera-process/index.js');
const cameraWorker = new Worker(CameraQueue, cameraProcessor, { concurrency: 50, connection});

above FFMPEG process will then keep running until it's closed via backend. What is happening atm is that the Job gets completed immediately, while this process keeps going, even if FFMPEG command fails, the worker logs properly but never fails.

What I would expect the behavior to be that the process only get's completed when all sub processes and promises and everything else is actually completed, or even better if I can manually complete the job by calling a done or completed function and same for failing it.

I also tried to add a long running timeout to see if that might keep the job running but it didn't work either.

Maybe I am missing something, but I checked the API and documentation and can't find something to do this.

EDIT: I have tried this with normal workers, with those behavior was same but at least I was getting progress event properly, which BullMQ wasn't able to catch with sandbox

How to reproduce.

You should be able to reproduce it by invoking ffmpeg or spawn a child process from within a sandbox job.

Relevant log output

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[manast]

It is difficult to asses what is causing this issue without knowing what is the content of the file '/processes/camera-process/index.js'. Maybe this blog post is useful for you as it includes working code: https://blog.taskforce.sh/splitting-heavy-jobs-using-bullmq-flows/

[fabianboerner]

Sounds a bit that something in the process started asynchronously without using the promise or await/async.