laravel / ideas

Issues board used for Laravel internals discussions.
938 stars 31 forks source link

Allow pausing batches #2664

Open MatanYadaev opened 2 years ago

MatanYadaev commented 2 years ago

What

An ability to store a batch in the database and control when it should be executed. It can be implemented by pause and continue methods.

Why

I, for example, want to dispatch 5,000 jobs in a batch. Every job contains a User model with many relationships. Fetching these models consumes much memory, so I must split the fetching into a few chunks, and push every chunk to the existing batch. But the problem is that the batch is already running. There's a race condition problem here: the workers might finish the pending jobs before the next chunk is fetched and pushed to the batch - and this will result in completing the batch before it really gets finished. If there was an option to pause the existing batch, and start it just after the whole chunks got stored - it will fix the race condition problem.

Example - Current state

$batch = Bus::batch([])->allowFailures()->dispatch();

User::query()
    ->addManyRelations()
    ->chunkById(1000, static function (Collection $usersChunk) use ($batch): void {
        $jobs = $usersChunk->map(static function (User $user) {
            return new SendNotificationToUserJob($user);
        });

        // The `$batch` might be finished before the second chunk is fetched.
        $batch->add($jobs);
    });

Example - After implementing pausing batches

$batch = Bus::batch([])->allowFailures()->dispatchPaused(); // or `->dispatch()->pause()`

User::query()
    ->addManyRelations()
    ->chunkById(1000, static function (Collection $usersChunk) use ($batch): void {
        $jobs = $usersChunk->map(static function (User $user) {
            return new SendNotificationToUserJob($user);
        });

        $batch->add($jobs);
    });

$batch->continue();