BroadcastException thrown on production server, but works on local machine

[mtanmay]

I've spent my whole day trying to solve this.

I am using redis as queue connection. I have an event TestUpdated which I am broadcasting on test.{id} private channel. The event gets fired and caught by the client properly when I am on a local machine. But on production server (forge), I get BroadcastException when TestUpdated is fired:

Illuminate\Broadcasting\BroadcastException in /home/forge/mydomain.com/vendor/laravel/framework/src/Illuminate/Broadcasting/Broadcasters/PusherBroadcaster.php:117

apps and ssl config in websockets.php:

'apps' => [
    [
        'id' => env('PUSHER_APP_ID'),
        'name' => env('APP_NAME'),
        'key' => env('PUSHER_APP_KEY'),
        'secret' => env('PUSHER_APP_SECRET'),
        'enable_client_messages' => true,
        'enable_statistics' => false,
    ],
],

'ssl' => [
        'local_cert' => env('LOCAL_CERT', null),
        'local_pk' => env('LOCAL_PK', null),
        'passphrase' => null,
        'verify_peer' => false,
    ],

pusher config in broadcasting.php:

'pusher' => [
            'driver' => 'pusher',
            'key' => env('PUSHER_APP_KEY'),
            'secret' => env('PUSHER_APP_SECRET'),
            'app_id' => env('PUSHER_APP_ID'),
            'options' => [
                'cluster' => env('PUSHER_APP_CLUSTER'),
                'encrypted' => true,
                'host' => '127.0.0.1',
                'port' => 6001,
                'scheme' => env('PUSHER_SCHEME'),
                'curl_options' => [
                    CURLOPT_SSL_VERIFYHOST => 0,
                    CURLOPT_SSL_VERIFYPEER => 0,
                ],
            ],
        ],

The relevant environment variables in .env:

PUSHER_APP_ID=someId
PUSHER_APP_KEY=someKey
PUSHER_APP_SECRET=someSecret
PUSHER_APP_CLUSTER=mt1
PUSHER_SCHEME=https

LOCAL_CERT=/etc/nginx/ssl/mydomain.com/123456/server.crt
LOCAL_PK=/etc/nginx/ssl/mydomain.com/123456/server.key

Client-side configuration:

window.Echo = new Echo({
        authEndpoint: 'my/endpoint',
        broadcaster: 'pusher',
        key: 'someKey',
        wsHost: process.env.NODE_ENV == 'development' ? window.location.hostname : 'mydomain.com',
        wsPort: 6001,
        wssPort: 6001,
        disableStats: true,
        encrypted: process.env.NODE_ENV == 'development' ? false : true
    });

Strangely, the presence channel works (could this be related to #149 ?). It's only the broadcasting that seems to be failing.

How do I fix this?

[francislavoie]

To clarify how it works, this lib exposes a WS server and an HTTP server. The HTTP server is used for implementing Pusher's HTTP API (see https://pusher.com/docs/channels/library_auth_reference/rest-api) which includes broadcasting.

This means that whatever service in your stack is broadcasting (in your case, your job queue workers) need to know how to send requests to your laravel-websockets HTTP server.

I don't know enough about Forge to help debug that aspect, but that's what you need to look for. This means you might need to change the 'host' in your broadcasting.php to point to whatever IP/domain your laravel-websockets is running on.

Hopefully when https://github.com/beyondcode/laravel-websockets/pull/140 is merged, you'll be able to use Redis for broadcasting instead and this sort of thing would go away. (This is assuming I'm reading your issue correctly)

[mtanmay]

@francislavoie Thanks a lot man! Changing the host fixes the issue :)

[TheBigK]

@francislavoie and @tanmay-das - I'm running into similar issue and have not been able to figure out a fix. Here's the error I'm getting [specifically after upgrading uBuntu to 18.10] -

[2019-05-05 14:02:43] production.ERROR:  {"exception":"[object] (Illuminate\Broadcasting\BroadcastException(code: 0):  at /var/www/site/vendor/laravel/framework/src/Illuminate/Broadcasting/Broadcasters/PusherBroadcaster.php:119)
[stacktrace]
#0 /var/www/site/vendor/laravel/framework/src/Illuminate/Broadcasting/BroadcastEvent.php(48): Illuminate\Broadcasting\Broadcasters\PusherBroadcaster->broadcast(Array, 'App\\Events\\Conv...', Array)
#1 [internal function]: Illuminate\Broadcasting\BroadcastEvent->handle(Object(Illuminate\Broadcasting\Broadcasters\PusherBroadcaster))
#2 /var/www/site/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(32): call_user_func_array(Array, Array)
#3 /var/www/site/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(90): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
#4 /var/www/site/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(34): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Application), Array, Object(Closure))
#5 /var/www/site/vendor/laravel/framework/src/Illuminate/Container/Container.php(580): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Array, Array, NULL)
#6 /var/www/site/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(94): Illuminate\Container\Container->call(Array)
#7 /var/www/site/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(128): Illuminate\Bus\Dispatcher->Illuminate\Bus\{closure}(Object(Illuminate\Broadcasting\BroadcastEvent))
#8 /var/www/site/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(104): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}(Object(Illuminate\Broadcasting\BroadcastEvent))
#9 /var/www/site/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(98): Illuminate\Pipeline\Pipeline->then(Object(Closure))
#10 /var/www/site/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(49): Illuminate\Bus\Dispatcher->dispatchNow(Object(Illuminate\Broadcasting\BroadcastEvent), false)
#11 /var/www/site/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(86): Illuminate\Queue\CallQueuedHandler->call(Object(Illuminate\Queue\Jobs\SqsJob), Array)
#12 /var/www/site/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(327): Illuminate\Queue\Jobs\Job->fire()
#13 /var/www/site/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(277): Illuminate\Queue\Worker->process('sqs', Object(Illuminate\Queue\Jobs\SqsJob), Object(Illuminate\Queue\WorkerOptions))
#14 /var/www/site/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(118): Illuminate\Queue\Worker->runJob(Object(Illuminate\Queue\Jobs\SqsJob), 'sqs', Object(Illuminate\Queue\WorkerOptions))
#15 /var/www/site/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(102): Illuminate\Queue\Worker->daemon('sqs', 'default', Object(Illuminate\Queue\WorkerOptions))
#16 /var/www/site/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(86): Illuminate\Queue\Console\WorkCommand->runWorker('sqs', 'default')
#17 [internal function]: Illuminate\Queue\Console\WorkCommand->handle()
#18 /var/www/site/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(32): call_user_func_array(Array, Array)
#19 /var/www/site/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(90): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
#20 /var/www/site/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(34): Illuminate\Container\BoundMethod::callBoundMethod(Object(Illuminate\Foundation\Application), Array, Object(Closure))
#21 /var/www/site/vendor/laravel/framework/src/Illuminate/Container/Container.php(580): Illuminate\Container\BoundMethod::call(Object(Illuminate\Foundation\Application), Array, Array, NULL)
#22 /var/www/site/vendor/laravel/framework/src/Illuminate/Console/Command.php(183): Illuminate\Container\Container->call(Array)
#23 /var/www/site/vendor/symfony/console/Command/Command.php(255): Illuminate\Console\Command->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
#24 /var/www/site/vendor/laravel/framework/src/Illuminate/Console/Command.php(170): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Illuminate\Console\OutputStyle))
#25 /var/www/site/vendor/symfony/console/Application.php(908): Illuminate\Console\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#26 /var/www/site/vendor/symfony/console/Application.php(269): Symfony\Component\Console\Application->doRunCommand(Object(Illuminate\Queue\Console\WorkCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#27 /var/www/site/vendor/symfony/console/Application.php(145): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#28 /var/www/site/vendor/laravel/framework/src/Illuminate/Console/Application.php(90): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#29 /var/www/site/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(122): Illuminate\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#30 /var/www/site/artisan(35): Illuminate\Foundation\Console\Kernel->handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#31 {main}

Note that I had everything working perfectly fine before the upgrade. Not sure what went wrong. Would really appreciate if you could point me in the right direction. It's almost 1.5 days I'm banging my head on this.

[francislavoie]

That error means your app can't connect to your websockets server over HTTP. The error message is truncated though so it's impossible to tell what the exception is. I don't know enough about how you set up your app to help, you didn't give nearly enough info.

[TheBigK]

Thank you for the comment, @francislavoie . I've set up the websockets server on 127.0.0.1; the same server which runs nginx.

If I turn on the Statistics dashboard, I do see connection being made to the server. However, it's only when the message is broadcasted, that the error crops up.

Could you please let me know what information could I provide to get better help?

[TheBigK]

Here's how I've setup the app on my production server -

• bootstrap.js:

window.Echo = new Echo({
    broadcaster: 'pusher',
    key: process.env.MIX_PUSHER_APP_KEY,
    cluster: process.env.MIX_PUSHER_APP_CLUSTER,
    encrypted: true,
    wsHost: window.location.hostname,
    wsPort: 6001,
    wssPort: 6001,
    disableStats: true,
    enabledTransports: ['ws', 'wss']
});

config/websockets.php

'local_cert' => '/etc/letsencrypt/live/www.mydomain.com/fullchain.pem',
'local_pk' => '/etc/letsencrypt/live/www.mydomain.com/privkey.pem',

'verify_peer' => true,  
// Have ensured that both these files are accessible by the user

Note about verify_peer => true option : I believe this should be set to true; or commented out. When I set it to false, I get the error pasted in my last post. But if I set it to true, I get following error in Chrome's console -

webSocket connection to 'wss://www.mydomain.com:6001/app/<APP_ID>protocol=7&client=js&version=4.4.0&flash=false' failed: WebSocket opening handshake was canceled

config/broadcasting.php

        'pusher' => [
            'driver' => 'pusher',
            'key' => env('PUSHER_APP_KEY'),
            'secret' => env('PUSHER_APP_SECRET'),
            'app_id' => env('PUSHER_APP_ID'),
            'options' => [
                'cluster' => env('PUSHER_APP_CLUSTER'),
                'encrypted' => true,
                'host'  =>  '127.0.0.1',
                'port'  =>  6001,
                'scheme'    =>  'https',
            ],
        ],

Troubleshooting steps I've tried so far -

1. Tried enabling cacert option in php.ini as follows -

   curl.cainfo="/etc/php/7.3/cacert.pem"
   openssl.cafile="/etc/php/7.3/cacert.pem"

   It makes no difference; and the error persists.

2. Tried setting encrypted option in both bootstrap.js and broadcasting.php to false. Still, the error persists.

3. Tried reinstalling the php7.3-curl extension. No effect on the error.

4. Tried adjusting ntpdate option as suggested in some of the similar errors. No success.

Also, the error I posted above is straight from the log file generated by Laravel. It shows the truncated version and I have no idea how to get further details from it. The error itself doesn't tell me anything about where the problem is.

Note: This error appeared after I ran do-release-upgrade which upgraded my server from ubuntu 18.04 to 18.04.2. Later, I upgraded it to 18.10; just to see if it fixes the issue. I'm now on uBuntu 18.10; but the error's still there.

@francislavoie - Would really appreciate if you could help me identify the issue. I'm on it since last 72 hours and with zero progress. I thank you for your time and look forward to hearing from you.

[Meera9]

I am tired to getting this type error when start websocket server broadcast function give me error like below, please can anyone help me to sort out?

[kirilkirkov]

@Meera9 Did you find a fix?

[DeBelserArne]

I am tired to getting this type error when start websocket server broadcast function give me error like below, please can anyone help me to sort out?

Hey @Meera9 I've had these kind of errors pop up in my dev tools whenever there was something wrong when sending a Notification.

Things that sometimes helped me out:

• Clear browser cache
• Clear the chrome cache in dev tools, press F12, go to Application and press "Clear site data"
• If you have any middleware that protects your routes, make sure you are logged into into your system when triggering websockets.

[loginweb-dev]

I have the same problem, it works local but in a production server with ssl the events are not sent.

[shahriar-b]

I have the same problem, it works local but in a production server with ssl the events are not sent.

same here

[glebk1]

same here

[glebk1]

Please, help

[verenzuela]

any solution...? :'(

[Meera9]

hi

[Meera9]

'pusher' => [
        'driver' => 'pusher',
        'key' => env('PUSHER_APP_KEY'),
        'secret' => env('PUSHER_APP_SECRET'),
        'app_id' => env('PUSHER_APP_ID'),
        'options' => [
            'cluster' => env('PUSHER_APP_CLUSTER'),
            'encrypted' => true,
            'host'  =>  '127.0.0.1',
            'port'  =>  6001,
            'scheme'    =>  'https',
        ],
    ],

did you check this configuration??

[Meera9]

@kirilkirkov did you check abover file setup?

[kirilkirkov]

@Meera9

'pusher' => [ 'driver' => 'pusher', 'key' => env('PUSHER_APP_KEY'), 'secret' => env('PUSHER_APP_SECRET'), 'app_id' => env('PUSHER_APP_ID'), 'options' => [ 'cluster' => env('PUSHER_APP_CLUSTER'), 'encrypted' => true, 'host' => '127.0.0.1', 'port' => 6001, 'scheme' => 'https', 'curl_options' => [ CURLOPT_SSL_VERIFYHOST => 0, CURLOPT_SSL_VERIFYPEER => 0, ] ],

This is the right configuration for me and works fine. Just need to remove host verify from curl :) .

[jimdeveloper89]

I wonder how you guys solved it?? :(

[mohammedjammeh]

I've been stuck on this for a while too. Please let us know if you've managed to solve it. Thanks

[Meera9]

configuration steps you can check here for websockets 1) broadcast.php in config folder

'pusher' => [
            'driver' => 'pusher',
            'key' => env('PUSHER_APP_KEY'),
            'secret' => env('PUSHER_APP_SECRET'),
            'app_id' => env('PUSHER_APP_ID'),
            'options' => [
                'cluster' => 'ap2',
                'encrypted' => false,
                'host' => '127.0.0.1',
                'port' => 6001,
                'scheme' => 'http'
         ],
     ],

2)in bootstrap.js in resources/assests/js folder

import Echo from 'laravel-echo'

window.Pusher = require('pusher-js');

window.Echo = new Echo({
    broadcaster: 'pusher',
    key: "your key here",
    cluster: "ap2",
    wsHost: window.location.hostname,
    wsPort: 6001,
    disableStats: true,
});

3).env file

PUSHER_APP_ID=your app id
PUSHER_APP_KEY=your app key 
PUSHER_APP_SECRET=your secret key of app
PUSHER_APP_CLUSTER=ap2

4)websockets.php in config folder

'apps' => [
        [
            'id' => env('PUSHER_APP_ID'),
            'name' => env('APP_NAME'),
            'key' => env('PUSHER_APP_KEY'),
            'secret' => env('PUSHER_APP_SECRET'),
            'path' => env('PUSHER_APP_PATH'),
            'capacity' => null,
            'enable_client_messages' => true,
            'enable_statistics' => true,
        ],
    ],

Or you check step by step configuration into this link https://laravelframeworknewfeature.blogspot.com/2020/05/laravel-websocket.html

you can create pusher id, key, secret key from here this link https://pusher.com/

I hope this will help you because in my project of chat above configuration was help me

[glrnz]

For those who are still stuck, these links might help.

-link1: https://github.com/beyondcode/laravel-websockets/issues/233#issuecomment-651646969

-link2: https://github.com/beyondcode/laravel-websockets/issues/233#issuecomment-564605696

[mreduar]

Thanks @kirilkirkov this work fine in local.

[stephenjason89]

@kirilkirkov thank you! Curl options worked for me on my prod server.

[Pondake]

(Posting this here as the relevancy of the issue is very alike, if this is a stale issue then I'll create a new one)

Tried multiple variations of configuration above to make it work with nginx so that SSL is allowed (like this).

broadcast.php in config folder

 'pusher' => [
      'driver' => 'pusher',
      'key' => env('PUSHER_APP_KEY'),
      'secret' => env('PUSHER_APP_SECRET'),
      'app_id' => env('PUSHER_APP_ID'),
      'options' => [
          'cluster' => env('PUSHER_APP_CLUSTER'),
          'host' => env('PUSHER_APP_HOST', '127.0.0.1'),
          'port' => env('PUSHER_APP_PORT', 6001),
          'encrypted' => true,
          'scheme' => 'https',
          'debug' => true, // I noticed that pusher 4.1.5 is using debug instead of 'log'
          'curl_options' => [
              CURLOPT_SSL_VERIFYHOST => 0,
              CURLOPT_SSL_VERIFYPEER => 0,
          ],
          'useTLS' => true,
      ],
  ],

websockets.php in config folder

'apps' => [
    [
        'id' => env('PUSHER_APP_ID'),
        'name' => env('APP_NAME'),
        'key' => env('PUSHER_APP_KEY'),
        'secret' => env('PUSHER_APP_SECRET'),
        'path' => env('PUSHER_APP_PATH'),
        'capacity' => null,
        'enable_client_messages' => true,
        'enable_statistics' => true,
        'encrypted' => true,
    ],
],

'ssl' => [
    'local_cert' => env('LARAVEL_WEBSOCKETS_SSL_LOCAL_CERT', null),
    'local_pk' => env('LARAVEL_WEBSOCKETS_SSL_LOCAL_PK', null),
    'passphrase' => env('LARAVEL_WEBSOCKETS_SSL_PASSPHRASE', null),
    'verify_peer' => false,
    'allow_self_signed' => true
],

.env file in root

BROADCAST_DRIVER=pusher
PUSHER_APP_ID=**
PUSHER_APP_KEY=**
PUSHER_APP_SECRET=**
PUSHER_APP_CLUSTER=mt1
PUSHER_APP_HOST="**" // the url websockets is hosted on, verified working on "wss" by checking the dashboard
PUSHER_APP_PORT="443"

MIX_PUSHER_APP_KEY="${PUSHER_APP_KEY}"
MIX_PUSHER_APP_CLUSTER="${PUSHER_APP_CLUSTER}"

Notice that local_cert is not applied here. That's because that's not available through the chosen hosting.

app.js (which usually is bootstrap.js)

window.Echo = new Echo({
      broadcaster: 'pusher',
      key: key, // entry in env
      cluster: cluster, // entry in env
      wsHost: host, // entry in env
      wssHost: host, // entry in env
      wsPort: port, // entry in env
      wssPort: port, // entry in env
      encrypted: true,
      forceTLS: true,
      enabledTransports: ['ws','wss'],
  });

Connecting with the public channel through laravel Echo works fine, but upon sending a message it gives a fat 404 through laravel Horizon.

Locally it works perfect with or without HTTPS using valet. I have spent more than a week at this but I'm starting to get clueless.