search

beyondcode / laravel-websockets

Websockets for Laravel. Done right.
https://beyondco.de/docs/laravel-websockets
MIT License
5.07k stars 612 forks source link

Websockets:serve - stops working after some time throwing BroadcastException: Failed to connect to Pusher - supervisor still running, not restarting #792

Closed ven0ms99 closed 7 months ago

ven0ms99 commented 3 years ago

I have this issue for years now on production. The past couple of days I've been really trying to get to the bottom of this.

Problem: After running websockets:serve (via supervisor or manually, doesn't matter) for a few hours, when I use broadcast() my server throws this error: Illuminate\Broadcasting\BroadcastException: Failed to connect to Pusher.. Fact is that after running for a few hours users cannot connect at all to the service or they don't receive messages (because of the given exception) all while the websockets debug dashboard still shows some subscriptions coming in, but at a very slow rate (if I were to restart websockets:serve there would be a lot more going on). However, most of the time it is also not possible to even connect to the debug dashboard ("Channels current state is unavailable"). The command is still running and thus, supervisor is not restarting the process. Running on a powerful state of the art dedicated webserver. I guess concurrent connections at a maximum of around 500 (how can I monitor the number of concurrent connections?).

on laravel-websockets 1. I noticed the behaviour starting when the process gets around 128-140mbit, but that must not necessarily be related. On 2.@dev it usually starts around 300-450mbit.

What I have already tried

My setup

open file limit set in limits.d to 64000.

.env:

BROADCAST_DRIVER=pusher
CACHE_DRIVER=file
SESSION_DRIVER=database
QUEUE_CONNECTION=sync
LARAVEL_WEBSOCKETS_SSL_LOCAL_CERT=(lets encrypt)
LARAVEL_WEBSOCKETS_SSL_LOCAL_PK=(lets encrypt)

composer.json:

{
    "name": "laravel/laravel",
    "description": "The Laravel Framework.",
    "keywords": ["framework", "laravel"],
    "license": "MIT",
    "type": "project",
    "require": {
        "php": "^7.3|^8.0",
        "ext-json": "*",
        "abraham/twitteroauth": "^2.0",
        "barryvdh/laravel-debugbar": "^3.0",
        "beyondcode/laravel-websockets": "2.*@dev",
        "caouecs/laravel-lang": "~3.0",
        "doctrine/dbal": "^2.8",
        "fideloper/proxy": "^4.0",
        "fruitcake/laravel-cors": "^2.0",
        "guzzlehttp/guzzle": "^7.0.1",
        "intervention/image": "^2.5",
        "kreait/laravel-firebase": "^2.1",
        "laravel/framework": "^8.0",
        "laravel/helpers": "^1.4",
        "laravel/tinker": "~2.0",
        "laravel/ui": "^3.0",
        "predis/predis": "^1.1",
        "pusher/pusher-php-server": "~4.0",
        "sentry/sentry-laravel": "^2.3",
        "spatie/laravel-sitemap": "^5.5",
        "stevebauman/location": "^5.0",
        "tgalopin/html-sanitizer": "^1.3"
    },
    "require-dev": {
        "facade/ignition": "^2.5",
        "fzaninotto/faker": "~1.4",
        "mockery/mockery": "0.9.*",
        "nunomaduro/collision": "^5.0",
        "phpunit/phpunit": "^9.0"
    },
    "autoload": {
        "psr-4": {
            "App\\": "app/",
            "Database\\Factories\\": "database/factories/",
            "Database\\Seeders\\": "database/seeders/"
        }
    },
    "autoload-dev": {
        "psr-4": {
            "Tests\\": "tests/"
        },
        "files": ["tests/Utilities/functions.php"]
    },
    "scripts": {
        "post-root-package-install": [
            "php -r \"file_exists('.env') || copy('.env.example', '.env');\""
        ],
        "post-create-project-cmd": [
            "php artisan key:generate"
        ]
    },
    "config": {
        "preferred-install": "dist",
        "sort-packages": true,
        "optimize-autoloader": true
    }
}

broadcasting.php:

<?php

return [

    'default' => env('BROADCAST_DRIVER', 'null'),

    'connections' => [

        'pusher' => [
            'driver' => 'pusher',
            'key' => env('PUSHER_APP_KEY'),
            'secret' => env('PUSHER_APP_SECRET'),
            'app_id' => env('PUSHER_APP_ID'),
            'options' => [
                'cluster' => env('PUSHER_APP_CLUSTER'),
                'host' => '127.0.0.1',
                'port' => 6001,
                'scheme' => 'https',
                'curl_options' => [
                    CURLOPT_SSL_VERIFYHOST => 0,
                ]
            ],
        ],

        'redis' => [
            'driver' => 'redis',
            'connection' => 'default',
        ],

        'log' => [
            'driver' => 'log',
        ],

        'null' => [
            'driver' => 'null',
        ],

    ],

];

websockest.php:

<?php

use BeyondCode\LaravelWebSockets\Dashboard\Http\Middleware\Authorize;

return [

    'dashboard' => [
        'port' => env('LARAVEL_WEBSOCKETS_PORT', 6001),
    ],

    'apps' => [
        [
            'id' => env('PUSHER_APP_ID'),
            'name' => env('APP_NAME'),
            'key' => env('PUSHER_APP_KEY'),
            'secret' => env('PUSHER_APP_SECRET'),
            'path' => env('PUSHER_APP_PATH'),
            'enable_client_messages' => false,
            'enable_statistics' => false,
        ],
    ],

    'app_provider' => BeyondCode\LaravelWebSockets\Apps\ConfigAppProvider::class,

    'allowed_origins' => [
        //
    ],

    /*
     * The maximum request size in kilobytes that is allowed for an incoming WebSocket request.
     */
    'max_request_size_in_kb' => 250,

    /*
     * This path will be used to register the necessary routes for the package.
     */
    'path' => 'websockets',

    'middleware' => [
        'web',
        Authorize::class,
    ],

    'statistics' => [
        'model' => \BeyondCode\LaravelWebSockets\Statistics\Models\WebSocketsStatisticsEntry::class,

        /*
         * Here you can specify the interval in seconds at which statistics should be logged.
         */
        'interval_in_seconds' => 60,

        /*
         * When the clean-command is executed, all recorded statistics older than
         * the number of days specified here will be deleted.
         */
        'delete_statistics_older_than_days' => 60,

        /*
         * Use an DNS resolver to make the requests to the statistics logger
         * default is to resolve everything to 127.0.0.1.
         */
        'perform_dns_lookup' => false,
    ],

    /*
     * Define the optional SSL context for your WebSocket connections.
     * You can see all available options at: http://php.net/manual/en/context.ssl.php
     */
    'ssl' => [
        'local_cert' => env('LARAVEL_WEBSOCKETS_SSL_LOCAL_CERT', null),
        /*
         * Path to local private key file on filesystem in case of separate files for
         * certificate (local_cert) and private key.
         */
        'local_pk' => env('LARAVEL_WEBSOCKETS_SSL_LOCAL_PK', null),
        /*
         * Passphrase for your local_cert file.
         */
        'passphrase' => env('LARAVEL_WEBSOCKETS_SSL_PASSPHRASE', null),
        'verify_peer' => false,
    ],

    'channel_manager' => \BeyondCode\LaravelWebSockets\WebSockets\Channels\ChannelManagers\ArrayChannelManager::class,
];

/config/flare.php:

return [

    #https://github.com/laravel/framework/issues/33952

    /*
    |--------------------------------------------------------------------------
    | Reporting Options
    |--------------------------------------------------------------------------
    |
    | These options determine which information will be transmitted to Flare.
    |
    */

    'reporting' => [
        'maximum_number_of_collected_queries' => 0,
    ],

];

bootstrap.js:

if (window.App.signedIn === true) {
    window.Pusher = require('pusher-js');
    window.Echo = new Echo({
        broadcaster: 'pusher',
        key: 'asjdfasfsjfk3848347',
        wsHost: window.location.hostname,
        wsPort: 6001,
        wssPort: 6001,
        encrypted: true,
        disableStats: true,
        forceTLS: true,
        enabledTransports: ['ws', 'wss'],
        auth: {
            headers: {
                'X-CSRF-TOKEN': window.App.csrfToken,
            },
        },
    })
}

The error my server throws after some time:

Illuminate\Broadcasting\BroadcastException: Failed to connect to Pusher. in /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Broadcasting/Broadcasters/PusherBroadcaster.php:142 Stack trace: #0 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Broadcasting/BroadcastEvent.php(71): Illuminate\Broadcasting\Broadcasters\PusherBroadcaster->broadcast() #1 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Illuminate\Broadcasting\BroadcastEvent->handle() #2 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Container/Util.php(40): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}() #3 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure() #4 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\Container\BoundMethod::callBoundMethod() #5 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Container/Container.php(651): Illuminate\Container\BoundMethod::call() #6 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(128): Illuminate\Container\Container->call() #7 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(128): Illuminate\Bus\Dispatcher->Illuminate\Bus\{closure}() #8 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(103): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #9 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(132): Illuminate\Pipeline\Pipeline->then() #10 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Broadcasting/BroadcastManager.php(116): Illuminate\Bus\Dispatcher->dispatchNow() #11 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php(308): Illuminate\Broadcasting\BroadcastManager->queue() #12 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php(231): Illuminate\Events\Dispatcher->broadcastEvent() #13 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Broadcasting/PendingBroadcast.php(57): Illuminate\Events\Dispatcher->dispatch() #14 /var/www/vhosts/my-site.com/httpdocs/app/Http/Controllers/RepliesController.php(217): Illuminate\Broadcasting\PendingBroadcast->__destruct() #15 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Routing/Controller.php(54): App\Http\Controllers\RepliesController->store() #16 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php(45): Illuminate\Routing\Controller->callAction() #17 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Routing/Route.php(254): Illuminate\Routing\ControllerDispatcher->dispatch() #18 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Routing/Route.php(197): Illuminate\Routing\Route->runController() #19 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Routing/Router.php(695): Illuminate\Routing\Route->run() #20 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(128): Illuminate\Routing\Router->Illuminate\Routing\{closure}() #21 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Auth/Middleware/EnsureEmailIsVerified.php(30): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #22 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Auth\Middleware\EnsureEmailIsVerified->handle() #23 /var/www/vhosts/my-site.com/httpdocs/app/Http/Middleware/BannedUsersCannotAccess.php(31): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #24 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): App\Http\Middleware\BannedUsersCannotAccess->handle() #25 /var/www/vhosts/my-site.com/httpdocs/app/Http/Middleware/UsersNeedToBeSubscribedToTags.php(42): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #26 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): App\Http\Middleware\UsersNeedToBeSubscribedToTags->handle() #27 /var/www/vhosts/my-site.com/httpdocs/app/Http/Middleware/ConfirmedUsersCanAccess.php(32): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #28 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): App\Http\Middleware\ConfirmedUsersCanAccess->handle() #29 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Routing/Middleware/SubstituteBindings.php(50): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #30 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Routing\Middleware\SubstituteBindings->handle() #31 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Routing/Middleware/ThrottleRequests.php(127): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #32 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Routing/Middleware/ThrottleRequests.php(63): Illuminate\Routing\Middleware\ThrottleRequests->handleRequest() #33 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Routing\Middleware\ThrottleRequests->handle() #34 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Auth/Middleware/Authenticate.php(44): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #35 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Auth\Middleware\Authenticate->handle() #36 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/VerifyCsrfToken.php(78): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #37 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Foundation\Http\Middleware\VerifyCsrfToken->handle() #38 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/View/Middleware/ShareErrorsFromSession.php(49): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #39 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\View\Middleware\ShareErrorsFromSession->handle() #40 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(121): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #41 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(64): Illuminate\Session\Middleware\StartSession->handleStatefulRequest() #42 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Session\Middleware\StartSession->handle() #43 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/AddQueuedCookiesToResponse.php(37): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #44 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Cookie\Middleware\AddQueuedCookiesToResponse->handle() #45 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/EncryptCookies.php(67): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #46 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Cookie\Middleware\EncryptCookies->handle() #47 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(103): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #48 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Routing/Router.php(697): Illuminate\Pipeline\Pipeline->then() #49 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Routing/Router.php(672): Illuminate\Routing\Router->runRouteWithinStack() #50 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Routing/Router.php(636): Illuminate\Routing\Router->runRoute() #51 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Routing/Router.php(625): Illuminate\Routing\Router->dispatchToRoute() #52 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(166): Illuminate\Routing\Router->dispatch() #53 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(128): Illuminate\Foundation\Http\Kernel->Illuminate\Foundation\Http\{closure}() #54 /var/www/vhosts/my-site.com/httpdocs/vendor/sentry/sentry-laravel/src/Sentry/Laravel/Http/SetRequestIpMiddleware.php(55): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #55 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Sentry\Laravel\Http\SetRequestIpMiddleware->handle() #56 /var/www/vhosts/my-site.com/httpdocs/vendor/sentry/sentry-laravel/src/Sentry/Laravel/Http/SetRequestMiddleware.php(52): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #57 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Sentry\Laravel\Http\SetRequestMiddleware->handle() #58 /var/www/vhosts/my-site.com/httpdocs/vendor/barryvdh/laravel-debugbar/src/Middleware/InjectDebugbar.php(60): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #59 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Barryvdh\Debugbar\Middleware\InjectDebugbar->handle() #60 /var/www/vhosts/my-site.com/httpdocs/vendor/fruitcake/laravel-cors/src/HandleCors.php(38): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #61 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Fruitcake\Cors\HandleCors->handle() #62 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(21): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #63 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ConvertEmptyStringsToNull.php(31): Illuminate\Foundation\Http\Middleware\TransformsRequest->handle() #64 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Foundation\Http\Middleware\ConvertEmptyStringsToNull->handle() #65 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(21): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #66 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TrimStrings.php(40): Illuminate\Foundation\Http\Middleware\TransformsRequest->handle() #67 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Foundation\Http\Middleware\TrimStrings->handle() #68 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ValidatePostSize.php(27): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #69 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Foundation\Http\Middleware\ValidatePostSize->handle() #70 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/PreventRequestsDuringMaintenance.php(86): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #71 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\Foundation\Http\Middleware\PreventRequestsDuringMaintenance->handle() #72 /var/www/vhosts/my-site.com/httpdocs/vendor/sentry/sentry-laravel/src/Sentry/Laravel/Tracing/Middleware.php(53): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #73 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Sentry\Laravel\Tracing\Middleware->handle() #74 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(103): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #75 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(141): Illuminate\Pipeline\Pipeline->then() #76 /var/www/vhosts/my-site.com/httpdocs/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(110): Illuminate\Foundation\Http\Kernel->sendRequestThroughRouter() #77 /var/www/vhosts/my-site.com/httpdocs/public/index.php(52): Illuminate\Foundation\Http\Kernel->handle() #78 {main}

Each signed in user subscribes to a number of channels for private messages, replies, etc.

The solution I have used for years now, which is restarting websockets:serve every hour is not working for me any longer, because I'd like to implement a chat service and can't really restart the process this often any more without annoying a lot of users. Also, do you have any idea why the process is getting bigger and bigger? Looks like a memory leak to me.

I spent days on this issue now and I have reached a point where I do not know what else to try. I'd appreciate any tips or advise.

edit: might this actually be related to QUEUE_CONNECTION=sync? The thing is: the messages dont slow down slowly over time, but all of a sudden. Also: I have tried a different queue connection and then the messages were delayed heavily in general.

Pes8 commented 3 years ago

I have the same problem.

I restart every 6 hour the process but it's not a real solution and sometimes CPU usage goes really high

simonbuehler commented 3 years ago

not a real answer but does changing to redis channelmanager give the same results? are there free file handles really available, check https://serverfault.com/a/448011 to monitor this, maybe you'l get a hint there

ven0ms99 commented 3 years ago

Some more insights: It is definitely related to the amount of usage. When I have many users using the service it becomes unavailable way faster. Also I went back to stable 1.X, because 2.*@dev didn't help at all actually.

not a real answer but does changing to redis channelmanager give the same results? are there free file handles really available, check https://serverfault.com/a/448011 to monitor this, maybe you'l get a hint there

I'll look into that and get back to you, thanks.

I restart every 6 hour the process but it's not a real solution and sometimes CPU usage goes really high

I used to do that. Now I have a cronjob doing a broadcast() every 10s and whenever I get the error BroadcastException: Failed to connect to Pusher I immediately restart the service. Due to the amount of users I have at the moment it doesn't last 6 hours anymore, sometimes less than 1 hour. But sounds like you do have the same issue indeed.

rmdwirizki commented 3 years ago

Any updates?

@Hillcow Have you solved this problem? I've run into the same issue too, I've already set the max open files limit to 50000 In my test, using ls -1 /proc/{PID}/fd | wc -l command, the opened file growth exponentially as the user increased overtime, it shows 40000 opened files just in an hour! and of course it lead to high cpu usage in a matter of time. The cpu and the opened files count won't ever go down until the worker (in supervisor) is restarted.

I've inspect the opened files every few minutes, and the broadcast will started to fail and return the "Failed to connect to Pusher" error when the opened files reach > 4000