Graceful termination fails for workers marked for termination during job processing

[tarexme]

Horizon Version

5.24.3

Laravel Version

11.5.0

PHP Version

8.3.6

Redis Driver

PhpRedis

Redis Version

7.2.4

Database Driver & Version

No response

Description

There appears to be an issue where workers marked for termination while processing jobs do not terminate gracefully when horizon:terminate is subsequently invoked. These workers, while still actively running, are overlooked during the supervisor's termination process. As a result, instead of terminating gracefully, they are killed upon the supervisor's exit.

Steps To Reproduce

1. Launch the master supervisor with the fast_termination option set to false using horizon command.
2. Send a long-running job to the queue. Ensure that this job is being processed by a worker.
3. Wait for scaleDown() method to be triggered on ProcessPool, ensuring that the process handling the long-running job is marked for termination. For consistent test results, use the code snippet below to simulate a supervisor restart during which all worker processes are marked for termination by scaling process pools down to 0.
4. Terminate horizon using horizon:terminate command.

// Dispatch a long-running job that sleeps for 60 seconds
SleepJob::dispatch(60);
sleep(10); // Make sure that job is picked up

// Trigger a restart on all supervisors, marking workers for termination
foreach (app(SupervisorRepository::class)->names() as $name) {
    app(HorizonCommandQueue::class)->push(
        $name, Restart::class
    );
}

sleep(10); // Make sure that all supervisors have restarted

// Call the terminate command
Artisan::call(TerminateCommand::class);

class SleepJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public function __construct(protected readonly int $sleepDuration)
    {
        // Nothing
    }

    /**
     * Execute the job.
     */
    public function handle(): void
    {
        for ($i = 0; $i < $this->sleepDuration; $i++) {
            sleep(1);
        }

        Log::debug('Job finished.');
    }
}

[driesvints]

Thanks. Seems you already sent in a PR so let's see how it goes.