Jobs are slow to be picked up, at least 2 seconds of extra processing time

sts-ryan-holton commented 1 year ago

Horizon Version

5.21.1

Laravel Version

10.28.0

PHP Version

8.2.10

Redis Driver

PhpRedis

Redis Version

redis:6.2-alpine

Database Driver & Version

MySQL 8.0

Description

:wave: Hi,

I have a meaty Laravel job file which is triggered from my controller. I have Carbon set to log the current time before the job is processed, and then afterwards to get the difference. When I add a carbon log instance to the job's handle method, and run it, I'm seeing between 1-2 seconds of extra time (presumably Horizon not picking the job up fast enough)

If I don't use Horizon, and simply call the job's handle method directly from my controller, this delay is gone.

What am I missing to speed up job pick up here?

Steps To Reproduce

Here's my config for Horizon, but the simplest reproduction is to create a job file:

<?php

use Illuminate\Support\Str;

return [

    /*
    |--------------------------------------------------------------------------
    | Horizon Domain
    |--------------------------------------------------------------------------
    |
    | This is the subdomain where Horizon will be accessible from. If this
    | setting is null, Horizon will reside under the same domain as the
    | application. Otherwise, this value will serve as the subdomain.
    |
    */

    'domain' => env('HORIZON_DOMAIN'),

    /*
    |--------------------------------------------------------------------------
    | Horizon Path
    |--------------------------------------------------------------------------
    |
    | This is the URI path where Horizon will be accessible from. Feel free
    | to change this path to anything you like. Note that the URI will not
    | affect the paths of its internal API that aren't exposed to users.
    |
    */

    'path' => env('HORIZON_PATH', 'horizon'),

    /*
    |--------------------------------------------------------------------------
    | Horizon Redis Connection
    |--------------------------------------------------------------------------
    |
    | This is the name of the Redis connection where Horizon will store the
    | meta information required for it to function. It includes the list
    | of supervisors, failed jobs, job metrics, and other information.
    |
    */

    'use' => 'default',

    /*
    |--------------------------------------------------------------------------
    | Horizon Redis Prefix
    |--------------------------------------------------------------------------
    |
    | This prefix will be used when storing all Horizon data in Redis. You
    | may modify the prefix when you are running multiple installations
    | of Horizon on the same server so that they don't have problems.
    |
    */

    'prefix' => env(
        'HORIZON_PREFIX',
        Str::slug(env('APP_NAME', 'laravel'), '_').'_horizon:'
    ),

    /*
    |--------------------------------------------------------------------------
    | Horizon Route Middleware
    |--------------------------------------------------------------------------
    |
    | These middleware will get attached onto each Horizon route, giving you
    | the chance to add your own middleware to this list or change any of
    | the existing middleware. Or, you can simply stick with this list.
    |
    */

    'middleware' => ['web'],

    /*
    |--------------------------------------------------------------------------
    | Queue Wait Time Thresholds
    |--------------------------------------------------------------------------
    |
    | This option allows you to configure when the LongWaitDetected event
    | will be fired. Every connection / queue combination may have its
    | own, unique threshold (in seconds) before this event is fired.
    |
    */

    'waits' => [
        'redis:default' => 60,
    ],

    /*
    |--------------------------------------------------------------------------
    | Job Trimming Times
    |--------------------------------------------------------------------------
    |
    | Here you can configure for how long (in minutes) you desire Horizon to
    | persist the recent and failed jobs. Typically, recent jobs are kept
    | for one hour while all failed jobs are stored for an entire week.
    |
    */

    'trim' => [
        'recent' => 15,
        'pending' => 15,
        'completed' => 15,
        'recent_failed' => 120,
        'failed' => 120,
        'monitored' => 120,
    ],

    /*
    |--------------------------------------------------------------------------
    | Silenced Jobs
    |--------------------------------------------------------------------------
    |
    | Silencing a job will instruct Horizon to not place the job in the list
    | of completed jobs within the Horizon dashboard. This setting may be
    | used to fully remove any noisy jobs from the completed jobs list.
    |
    */

    'silenced' => [
        App\Jobs\ProcessAnalytic::class,
        App\Jobs\StoreApiRequestLog::class,
        App\Jobs\ProcessModelObserver::class,
    ],

    /*
    |--------------------------------------------------------------------------
    | Metrics
    |--------------------------------------------------------------------------
    |
    | Here you can configure how many snapshots should be kept to display in
    | the metrics graph. This will get used in combination with Horizon's
    | `horizon:snapshot` schedule to define how long to retain metrics.
    |
    */

    'metrics' => [
        'trim_snapshots' => [
            'job' => 48,
            'queue' => 48,
        ],
    ],

    /*
    |--------------------------------------------------------------------------
    | Fast Termination
    |--------------------------------------------------------------------------
    |
    | When this option is enabled, Horizon's "terminate" command will not
    | wait on all of the workers to terminate unless the --wait option
    | is provided. Fast termination can shorten deployment delay by
    | allowing a new instance of Horizon to start while the last
    | instance will continue to terminate each of its workers.
    |
    */

    'fast_termination' => false,

    /*
    |--------------------------------------------------------------------------
    | Memory Limit (MB)
    |--------------------------------------------------------------------------
    |
    | This value describes the maximum amount of memory the Horizon master
    | supervisor may consume before it is terminated and restarted. For
    | configuring these limits on your workers, see the next section.
    |
    */

    'memory_limit' => 512,

    /*
    |--------------------------------------------------------------------------
    | Queue Worker Configuration
    |--------------------------------------------------------------------------
    |
    | Here you may define the queue worker settings used by your application
    | in all environments. These supervisors and settings handle all your
    | queued jobs and will be provisioned by Horizon during deployment.
    |
    */

    'defaults' => [

        /**
         * Fast jobs
         *
         * Jobs on this queue are optimised to run as quickly as possible.
         */
        'supervisor-fast-jobs' => [
            'connection' => 'redis-short-running',
            'queue' => ['on-demand-runs-now', 'redirects', 'commissions', 'statistics', 'listeners', 'observers', 'notifications', 'redis-short-running'],
            'balance' => 'auto',
            'maxProcesses' => 1,
            'maxTime' => 60,
            'maxJobs' => 0,
            'memory' => 128,
            'tries' => 1,
            'timeout' => 40,
            'nice' => 0,
        ],

        /**
         * Regular jobs
         *
         * Jobs on this queue should be default.
         */
        'supervisor-regular-jobs' => [
            'connection' => 'redis',
            'queue' => ['on-demand-runs-now', 'logs', 'analytics', 'csv-exports', 'default'],
            'balance' => 'auto',
            'maxProcesses' => 1,
            'maxTime' => 180,
            'maxJobs' => 0,
            'memory' => 128,
            'tries' => 1,
            'timeout' => 120,
            'nice' => 0,
        ],

        /**
         * Slow jobs
         *
         * Jobs on this queue are allowed to run for a longer period of time.
         */
        'supervisor-slow-jobs' => [
            'connection' => 'redis-long-running',
            'queue' => ['on-demand-runs-now', 'applications', 'pingtrees', 'redis-long-running'],
            'balance' => 'auto',
            'maxProcesses' => 1,
            'maxTime' => 700,
            'maxJobs' => 0,
            'memory' => 512,
            'tries' => 1,
            'timeout' => 700,
            'nice' => 0,
        ],
    ],

    'environments' => [
        'production' => [
            'supervisor-fast-jobs' => [
                'minProcesses' => 1,
                'maxProcesses' => 150,
                'balanceMaxShift' => 1,
                'balanceCooldown' => 3,
            ],
            'supervisor-regular-jobs' => [
                'minProcesses' => 1,
                'maxProcesses' => 50,
                'balanceMaxShift' => 1,
                'balanceCooldown' => 3,
            ],
            'supervisor-slow-jobs' => [
                'minProcesses' => 1,
                'maxProcesses' => 250,
                'balanceMaxShift' => 5,
                'balanceCooldown' => 1,
            ],
        ],

        'local' => [
            'supervisor-fast-jobs' => [
                'maxProcesses' => 6,
            ],
            'supervisor-regular-jobs' => [
                'maxProcesses' => 6,
            ],
            'supervisor-slow-jobs' => [
                'maxProcesses' => 6,
            ],
        ],
    ],
];

driesvints commented 1 year ago

Hi there,

Thanks for reporting but it looks like this is a question which can be asked on a support channel. Please only use this issue tracker for reporting bugs with the library itself. If you have a question on how to use functionality provided by this repo you can try one of the following channels:

However, this issue will not be locked and everyone is still free to discuss solutions to your problem!

Thanks.

sts-ryan-holton commented 1 year ago

For reference: https://stackoverflow.com/questions/77326180/laravel-horizon-slow-to-pick-up-job-by-1-2s

laravel / horizon