Pusher error: Internal server error..

alexsyvolap commented 4 months ago

Reverb Version

1.0.0-beta7

Laravel Version

11.2.0

PHP Version

8.3.6

Description

Hi! Have problem with reverb broadcast in production

.env

BROADCAST_CONNECTION=reverb

REVERB_APP_ID=********
REVERB_APP_KEY=********
REVERB_APP_SECRET=********
REVERB_SERVER_HOST=0.0.0.0
REVERB_SERVER_PORT=8081
REVERB_SCHEME=https

REVERB_HOST=wss.*******.com
REVERB_PORT=443

supervisor

[program:reverb]
command=php /var/www/html/artisan reverb:start --host="0.0.0.0" --port=8081
autostart=true
autorestart=true
user=%(ENV_SUPERVISOR_PHP_USER)s
redirect_stderr=true
stdout_logfile=/var/www/html/storage/logs/reverb.log

Success connection to wss but something goes wrong. Can anyone help?

Steps To Reproduce

.

joedixon commented 4 months ago

Hi @alexsyvolap are you able to get the stack trace for the exception

alexsyvolap commented 4 months ago

Hi @joedixon!

BroadcastException

Illuminate\Broadcasting\BroadcastException:
Pusher error: Internal server error..

  at /var/www/html/vendor/laravel/framework/src/Illuminate/Broadcasting/Broadcasters/PusherBroadcaster.php:165
  at Illuminate\Broadcasting\Broadcasters\PusherBroadcaster->broadcast()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Broadcasting/BroadcastEvent.php:92)
  at Illuminate\Broadcasting\BroadcastEvent->handle()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:36)
  at Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Container/Util.php:41)
  at Illuminate\Container\Util::unwrapIfClosure()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:93)
  at Illuminate\Container\BoundMethod::callBoundMethod()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:35)
  at Illuminate\Container\BoundMethod::call()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Container/Container.php:662)
  at Illuminate\Container\Container->call()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php:128)
  at Illuminate\Bus\Dispatcher->Illuminate\Bus\{closure}()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:144)
  at Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:119)
  at Illuminate\Pipeline\Pipeline->then()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php:132)
  at Illuminate\Bus\Dispatcher->dispatchNow()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Broadcasting/BroadcastManager.php:158)
  at Illuminate\Broadcasting\BroadcastManager->queue()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php:357)
  at Illuminate\Events\Dispatcher->broadcastEvent()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php:280)
  at Illuminate\Events\Dispatcher->invokeListeners()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php:266)
  at Illuminate\Events\Dispatcher->dispatch()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Broadcasting/PendingBroadcast.php:72)
  at Illuminate\Broadcasting\PendingBroadcast->__destruct()
     (/var/www/html/app/Services/Chat/SendMessageService.php:24)
  at App\Services\Chat\SendMessageService->run()
     (/var/www/html/app/Filament/Abstract/ChatAbstract.php:72)
  at App\Filament\Abstract\ChatAbstract->sendMessage()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:36)
  at Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Container/Util.php:41)
  at Illuminate\Container\Util::unwrapIfClosure()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:93)
  at Illuminate\Container\BoundMethod::callBoundMethod()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:35)
  at Illuminate\Container\BoundMethod::call()
     (/var/www/html/vendor/livewire/livewire/src/Wrapped.php:23)
  at Livewire\Wrapped->__call()
     (/var/www/html/vendor/livewire/livewire/src/Mechanisms/HandleComponents/HandleComponents.php:468)
  at Livewire\Mechanisms\HandleComponents\HandleComponents->callMethods()
     (/var/www/html/vendor/livewire/livewire/src/Mechanisms/HandleComponents/HandleComponents.php:100)
  at Livewire\Mechanisms\HandleComponents\HandleComponents->update()
     (/var/www/html/vendor/livewire/livewire/src/LivewireManager.php:96)
  at Livewire\LivewireManager->update()
     (/var/www/html/vendor/livewire/livewire/src/Mechanisms/HandleRequests/HandleRequests.php:89)
  at Livewire\Mechanisms\HandleRequests\HandleRequests->handleUpdate()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php:46)
  at Illuminate\Routing\ControllerDispatcher->dispatch()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Routing/Route.php:260)
  at Illuminate\Routing\Route->runController()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Routing/Route.php:206)
  at Illuminate\Routing\Route->run()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Routing/Router.php:806)
  at Illuminate\Routing\Router->Illuminate\Routing\{closure}()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:144)
  at Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Routing/Middleware/SubstituteBindings.php:50)
  at Illuminate\Routing\Middleware\SubstituteBindings->handle()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:183)
  at Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/VerifyCsrfToken.php:88)
  at Illuminate\Foundation\Http\Middleware\VerifyCsrfToken->handle()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:183)
  at Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/View/Middleware/ShareErrorsFromSession.php:49)
  at Illuminate\View\Middleware\ShareErrorsFromSession->handle()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:183)
  at Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php:121)
  at Illuminate\Session\Middleware\StartSession->handleStatefulRequest()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php:64)
  at Illuminate\Session\Middleware\StartSession->handle()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:183)
  at Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/AddQueuedCookiesToResponse.php:37)
  at Illuminate\Cookie\Middleware\AddQueuedCookiesToResponse->handle()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:183)
  at Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/EncryptCookies.php:75)
  at Illuminate\Cookie\Middleware\EncryptCookies->handle()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:183)
  at Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:119)
  at Illuminate\Pipeline\Pipeline->then()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Routing/Router.php:805)
  at Illuminate\Routing\Router->runRouteWithinStack()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Routing/Router.php:784)
  at Illuminate\Routing\Router->runRoute()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Routing/Router.php:748)
  at Illuminate\Routing\Router->dispatchToRoute()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Routing/Router.php:737)
  at Illuminate\Routing\Router->dispatch()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php:200)
  at Illuminate\Foundation\Http\Kernel->Illuminate\Foundation\Http\{closure}()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:144)
  at Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
     (/var/www/html/vendor/livewire/livewire/src/Features/SupportDisablingBackButtonCache/DisableBackButtonCacheMiddleware.php:19)
  at Livewire\Features\SupportDisablingBackButtonCache\DisableBackButtonCacheMiddleware->handle()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:183)
  at Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ConvertEmptyStringsToNull.php:27)
  at Illuminate\Foundation\Http\Middleware\ConvertEmptyStringsToNull->handle()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:183)
  at Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TrimStrings.php:47)
  at Illuminate\Foundation\Http\Middleware\TrimStrings->handle()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:183)
  at Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Http/Middleware/ValidatePostSize.php:27)
  at Illuminate\Http\Middleware\ValidatePostSize->handle()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:183)
  at Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/PreventRequestsDuringMaintenance.php:110)
  at Illuminate\Foundation\Http\Middleware\PreventRequestsDuringMaintenance->handle()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:183)
  at Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Http/Middleware/HandleCors.php:49)
  at Illuminate\Http\Middleware\HandleCors->handle()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:183)
  at Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Http/Middleware/TrustProxies.php:57)
  at Illuminate\Http\Middleware\TrustProxies->handle()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:183)
  at Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php:119)
  at Illuminate\Pipeline\Pipeline->then()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php:175)
  at Illuminate\Foundation\Http\Kernel->sendRequestThroughRouter()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php:144)
  at Illuminate\Foundation\Http\Kernel->handle()
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Application.php:1168)
  at Illuminate\Foundation\Application->handleRequest()
     (/var/www/html/public/index.php:17)
  at require_once('/var/www/html/public/index.php')
     (/var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/resources/server.php:16)

joedixon commented 4 months ago

If you start the server in debug mode, you should see the cause of the 500 error in the server output.

php artisan reverb:start --debug

alexsyvolap commented 4 months ago

Debug trace

Connection Established .................................. 70363917.405033814
  Message Received ........................................ 70363917.405033814

   1▕ {
   2▕     "event": "pusher:subscribe",
   3▕     "data": {
   4▕         "auth": "-----:-----",
   5▕         "channel": "private-admin.all.chats"...

  Message Handled ......................................... 70363917.405033814
  Message Received ........................................ 70363917.405033814

   1▕ {
   2▕     "event": "pusher:subscribe",
   3▕     "data": {
   4▕         "auth": "-----:-----",
   5▕         "channel": "private-chat.7"
   6▕     }
   7▕ }

  Message Handled ......................................... 70363917.405033814
  Message Received ........................................ 70363917.405033814

   1▕ {
   2▕     "event": "pusher:subscribe",
   3▕     "data": {
   4▕         "auth": "-----:-----",
   5▕         "channel": "private-App.Models.User.1"...

  Message Handled ......................................... 70363917.405033814
  Pruning Stale Connections ..................................................
  Pinging Inactive Connections ...............................................

 [ERROR] Syntax error

Laravel logs

[2024-04-18 12:35:49] production.ERROR: Pusher error: Internal server error.. {"userId":1,"exception":"[object] (Illuminate\\Broadcasting\\BroadcastException(code: 0): Pusher error: Internal server error.. at /var/www/html/vendor/laravel/framework/src/Illuminate/Broadcasting/Broadcasters/PusherBroadcaster.php:165)
[stacktrace]
#0 /var/www/html/vendor/laravel/framework/src/Illuminate/Broadcasting/BroadcastEvent.php(92): Illuminate\\Broadcasting\\Broadcasters\\PusherBroadcaster->broadcast()
#1 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Illuminate\\Broadcasting\\BroadcastEvent->handle()
#2 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#3 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure()
#4 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\\Container\\BoundMethod::callBoundMethod()
#5 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Container.php(662): Illuminate\\Container\\BoundMethod::call()
#6 /var/www/html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(128): Illuminate\\Container\\Container->call()
#7 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(144): Illuminate\\Bus\\Dispatcher->Illuminate\\Bus\\{closure}()
#8 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(119): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#9 /var/www/html/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(132): Illuminate\\Pipeline\\Pipeline->then()
#10 /var/www/html/vendor/laravel/framework/src/Illuminate/Broadcasting/BroadcastManager.php(158): Illuminate\\Bus\\Dispatcher->dispatchNow()
#11 /var/www/html/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php(357): Illuminate\\Broadcasting\\BroadcastManager->queue()
#12 /var/www/html/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php(280): Illuminate\\Events\\Dispatcher->broadcastEvent()
#13 /var/www/html/vendor/laravel/framework/src/Illuminate/Events/Dispatcher.php(266): Illuminate\\Events\\Dispatcher->invokeListeners()
#14 /var/www/html/vendor/laravel/framework/src/Illuminate/Broadcasting/PendingBroadcast.php(72): Illuminate\\Events\\Dispatcher->dispatch()
#15 /var/www/html/app/Services/Chat/SendMessageService.php(24): Illuminate\\Broadcasting\\PendingBroadcast->__destruct()
#16 /var/www/html/app/Filament/Abstract/ChatAbstract.php(72): App\\Services\\Chat\\SendMessageService->run()
#17 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): App\\Filament\\Abstract\\ChatAbstract->sendMessage()
#18 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#19 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure()
#20 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\\Container\\BoundMethod::callBoundMethod()
#21 /var/www/html/vendor/livewire/livewire/src/Wrapped.php(23): Illuminate\\Container\\BoundMethod::call()
#22 /var/www/html/vendor/livewire/livewire/src/Mechanisms/HandleComponents/HandleComponents.php(468): Livewire\\Wrapped->__call()
#23 /var/www/html/vendor/livewire/livewire/src/Mechanisms/HandleComponents/HandleComponents.php(100): Livewire\\Mechanisms\\HandleComponents\\HandleComponents->callMethods()
#24 /var/www/html/vendor/livewire/livewire/src/LivewireManager.php(96): Livewire\\Mechanisms\\HandleComponents\\HandleComponents->update()
#25 /var/www/html/vendor/livewire/livewire/src/Mechanisms/HandleRequests/HandleRequests.php(89): Livewire\\LivewireManager->update()
#26 /var/www/html/vendor/laravel/framework/src/Illuminate/Routing/ControllerDispatcher.php(46): Livewire\\Mechanisms\\HandleRequests\\HandleRequests->handleUpdate()
#27 /var/www/html/vendor/laravel/framework/src/Illuminate/Routing/Route.php(260): Illuminate\\Routing\\ControllerDispatcher->dispatch()
#28 /var/www/html/vendor/laravel/framework/src/Illuminate/Routing/Route.php(206): Illuminate\\Routing\\Route->runController()
#29 /var/www/html/vendor/laravel/framework/src/Illuminate/Routing/Router.php(806): Illuminate\\Routing\\Route->run()
#30 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(144): Illuminate\\Routing\\Router->Illuminate\\Routing\\{closure}()
#31 /var/www/html/vendor/laravel/framework/src/Illuminate/Routing/Middleware/SubstituteBindings.php(50): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#32 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(183): Illuminate\\Routing\\Middleware\\SubstituteBindings->handle()
#33 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/VerifyCsrfToken.php(88): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#34 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(183): Illuminate\\Foundation\\Http\\Middleware\\VerifyCsrfToken->handle()
#35 /var/www/html/vendor/laravel/framework/src/Illuminate/View/Middleware/ShareErrorsFromSession.php(49): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#36 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(183): Illuminate\\View\\Middleware\\ShareErrorsFromSession->handle()
#37 /var/www/html/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(121): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#38 /var/www/html/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(64): Illuminate\\Session\\Middleware\\StartSession->handleStatefulRequest()
#39 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(183): Illuminate\\Session\\Middleware\\StartSession->handle()
#40 /var/www/html/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/AddQueuedCookiesToResponse.php(37): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#41 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(183): Illuminate\\Cookie\\Middleware\\AddQueuedCookiesToResponse->handle()
#42 /var/www/html/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/EncryptCookies.php(75): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#43 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(183): Illuminate\\Cookie\\Middleware\\EncryptCookies->handle()
#44 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(119): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#45 /var/www/html/vendor/laravel/framework/src/Illuminate/Routing/Router.php(805): Illuminate\\Pipeline\\Pipeline->then()
#46 /var/www/html/vendor/laravel/framework/src/Illuminate/Routing/Router.php(784): Illuminate\\Routing\\Router->runRouteWithinStack()
#47 /var/www/html/vendor/laravel/framework/src/Illuminate/Routing/Router.php(748): Illuminate\\Routing\\Router->runRoute()
#48 /var/www/html/vendor/laravel/framework/src/Illuminate/Routing/Router.php(737): Illuminate\\Routing\\Router->dispatchToRoute()
#49 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(200): Illuminate\\Routing\\Router->dispatch()
#50 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(144): Illuminate\\Foundation\\Http\\Kernel->Illuminate\\Foundation\\Http\\{closure}()
#51 /var/www/html/vendor/livewire/livewire/src/Features/SupportDisablingBackButtonCache/DisableBackButtonCacheMiddleware.php(19): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#52 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(183): Livewire\\Features\\SupportDisablingBackButtonCache\\DisableBackButtonCacheMiddleware->handle()
#53 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ConvertEmptyStringsToNull.php(27): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#54 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(183): Illuminate\\Foundation\\Http\\Middleware\\ConvertEmptyStringsToNull->handle()
#55 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TrimStrings.php(47): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#56 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(183): Illuminate\\Foundation\\Http\\Middleware\\TrimStrings->handle()
#57 /var/www/html/vendor/laravel/framework/src/Illuminate/Http/Middleware/ValidatePostSize.php(27): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#58 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(183): Illuminate\\Http\\Middleware\\ValidatePostSize->handle()
#59 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/PreventRequestsDuringMaintenance.php(110): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#60 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(183): Illuminate\\Foundation\\Http\\Middleware\\PreventRequestsDuringMaintenance->handle()
#61 /var/www/html/vendor/laravel/framework/src/Illuminate/Http/Middleware/HandleCors.php(49): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#62 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(183): Illuminate\\Http\\Middleware\\HandleCors->handle()
#63 /var/www/html/vendor/laravel/framework/src/Illuminate/Http/Middleware/TrustProxies.php(57): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#64 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(183): Illuminate\\Http\\Middleware\\TrustProxies->handle()
#65 /var/www/html/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(119): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#66 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(175): Illuminate\\Pipeline\\Pipeline->then()
#67 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(144): Illuminate\\Foundation\\Http\\Kernel->sendRequestThroughRouter()
#68 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(1168): Illuminate\\Foundation\\Http\\Kernel->handle()
#69 /var/www/html/public/index.php(17): Illuminate\\Foundation\\Application->handleRequest()
#70 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/resources/server.php(16): require_once('...')
#71 {main}
"}

joedixon commented 4 months ago

Reverb logs the exception message which in this case is not that helpful ([ERROR] Syntax error).

If you dump out the stack trace here, you will get a better idea what is causing the issue:

https://github.com/laravel/reverb/blob/main/src/Servers/Reverb/Http/Server.php#L63-L66

alexsyvolap commented 4 months ago

Found that. Exception generated here https://github.com/laravel/reverb/blob/main/src/Protocols/Pusher/Http/Controllers/EventsController.php#L28 because request body content is null here https://github.com/laravel/reverb/blob/main/src/Protocols/Pusher/Http/Controllers/Controller.php#L46 but I can't understand why

JsonException

JsonException: Syntax error in /var/www/html/vendor/laravel/reverb/src/Protocols/Pusher/Http/Controllers/EventsController.php:28
Stack trace:
#0 /var/www/html/vendor/laravel/reverb/src/Protocols/Pusher/Http/Controllers/EventsController.php(28): json_decode()
#1 /var/www/html/vendor/laravel/reverb/src/Servers/Reverb/Http/Router.php(69): Laravel\Reverb\Protocols\Pusher\Http\Controllers\EventsController->__invoke()
#2 /var/www/html/vendor/laravel/reverb/src/Servers/Reverb/Http/Server.php(61): Laravel\Reverb\Servers\Reverb\Http\Router->dispatch()
#3 /var/www/html/vendor/laravel/reverb/src/Servers/Reverb/Http/Server.php(106): Laravel\Reverb\Servers\Reverb\Http\Server->handleRequest()
#4 /var/www/html/vendor/evenement/evenement/src/EventEmitterTrait.php(143): Laravel\Reverb\Servers\Reverb\Http\Server->Laravel\Reverb\Servers\Reverb\Http\{closure}()
#5 /var/www/html/vendor/react/stream/src/Util.php(71): Evenement\EventEmitter->emit()
#6 /var/www/html/vendor/evenement/evenement/src/EventEmitterTrait.php(143): React\Stream\Util::React\Stream\{closure}()
#7 /var/www/html/vendor/react/stream/src/DuplexResourceStream.php(196): Evenement\EventEmitter->emit()
#8 /var/www/html/vendor/react/event-loop/src/StreamSelectLoop.php(246): React\Stream\DuplexResourceStream->handleData()
#9 /var/www/html/vendor/react/event-loop/src/StreamSelectLoop.php(213): React\EventLoop\StreamSelectLoop->waitForStreamActivity()
#10 /var/www/html/vendor/laravel/reverb/src/Servers/Reverb/Http/Server.php(42): React\EventLoop\StreamSelectLoop->run()
#11 /var/www/html/vendor/laravel/reverb/src/Servers/Reverb/Console/Commands/StartServer.php(74): Laravel\Reverb\Servers\Reverb\Http\Server->start()
#12 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Laravel\Reverb\Servers\Reverb\Console\Commands\StartServer->handle()
#13 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}()
#14 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure()
#15 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\Container\BoundMethod::callBoundMethod()
#16 /var/www/html/vendor/laravel/framework/src/Illuminate/Container/Container.php(662): Illuminate\Container\BoundMethod::call()
#17 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(212): Illuminate\Container\Container->call()
#18 /var/www/html/vendor/symfony/console/Command/Command.php(279): Illuminate\Console\Command->execute()
#19 /var/www/html/vendor/laravel/framework/src/Illuminate/Console/Command.php(181): Symfony\Component\Console\Command\Command->run()
#20 /var/www/html/vendor/symfony/console/Application.php(1049): Illuminate\Console\Command->run()
#21 /var/www/html/vendor/symfony/console/Application.php(318): Symfony\Component\Console\Application->doRunCommand()
#22 /var/www/html/vendor/symfony/console/Application.php(169): Symfony\Component\Console\Application->doRun()
#23 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(196): Symfony\Component\Console\Application->run()
#24 /var/www/html/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(1183): Illuminate\Foundation\Console\Kernel->handle()
#25 /var/www/html/artisan(13): Illuminate\Foundation\Application->handleCommand()
#26 {main}

joedixon commented 4 months ago

Are you able to provide a repo and detailed steps on how to reproduce?

laravel new bug-report --github="--public"

alexsyvolap commented 4 months ago

Apologies, this project is covered by a NDA. However, I can provide a brief overview of the architecture.

We utilize Docker on the server to deploy the project. The primary load balancer on the server is Traefik.

We expose ports from the backend image as follows:

Port 8080 is used for the main PHP application. Port 8081 is used for the Reverb service. Additionally, Traefik is configured to manage traffic for two domains:

www.**.com, which serves as the main application. Traefik routes all traffic from this domain to Docker port 8080. wss.**.com, which handles WebSocket connections. Traefik routes all traffic from this domain to Docker port 8081.

.env

BROADCAST_CONNECTION=reverb

REVERB_APP_ID=********
REVERB_APP_KEY=********
REVERB_APP_SECRET=********
REVERB_HOST=0.0.0.0
REVERB_PORT=8081
REVERB_SCHEME=https

REVERB_HOST=wss.rebdron.com
REVERB_PORT=443

VITE_REVERB_APP_KEY="${REVERB_APP_KEY}"
VITE_REVERB_HOST="${REVERB_HOST}"
VITE_REVERB_PORT="${REVERB_PORT}"
VITE_REVERB_SCHEME="${REVERB_SCHEME}"

supervisord.conf

[supervisord]
nodaemon=true
user=root
logfile=/var/log/supervisor/supervisord.log
pidfile=/var/run/supervisord.pid
minfds=10000

[program:php]
command=%(ENV_SUPERVISOR_PHP_COMMAND)s
user=%(ENV_SUPERVISOR_PHP_USER)s
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

[program:reverb]
command=php /var/www/html/artisan reverb:start --debug --host="0.0.0.0" --port=8081
autostart=true
autorestart=true
user=%(ENV_SUPERVISOR_PHP_USER)s
redirect_stderr=true
stdout_logfile=/var/www/html/storage/logs/reverb.log

Locally, everything works without any issues. The setup is identical to that on the server.

If you require any further information, please feel free to let me know. I'll try describe that for you.

driesvints commented 4 months ago

@alexsyvolap we need a repo that reproduces the issue otherwise it's going to be hard to debug this. Please run the command Joe provided and reproduce the issue on a fresh Laravel project.

alexsyvolap commented 4 months ago

@driesvints ok, I'll try prepare demo

alexsyvolap commented 4 months ago

Hey, I prepared repo: https://github.com/alexsyvolap/reverb-bug-report

.env

APP_NAME=Laravel
APP_ENV=local
APP_KEY=base64:7FVpnzG/V5xQNNtOYsGlyrjvqoIV6lTjBJe3pVlnej8=
APP_DEBUG=true
APP_TIMEZONE=UTC
APP_URL=http://localhost
APP_ADMIN_PATH=admin

APP_LOCALE=en
APP_FALLBACK_LOCALE=en
APP_FAKER_LOCALE=en_US

APP_MAINTENANCE_DRIVER=file
APP_MAINTENANCE_STORE=database

BCRYPT_ROUNDS=12

LOG_CHANNEL=stack
LOG_STACK=single
LOG_DEPRECATIONS_CHANNEL=null
LOG_LEVEL=debug

DB_CONNECTION=sqlite

SESSION_DRIVER=database
SESSION_LIFETIME=120
SESSION_ENCRYPT=false
SESSION_PATH=/
SESSION_DOMAIN=null

BROADCAST_CONNECTION=reverb
FILESYSTEM_DISK=local
QUEUE_CONNECTION=database

CACHE_STORE=database
CACHE_PREFIX=

MEMCACHED_HOST=127.0.0.1

REDIS_CLIENT=phpredis
REDIS_HOST=127.0.0.1
REDIS_PASSWORD=null
REDIS_PORT=6379

MAIL_MAILER=log
MAIL_HOST=127.0.0.1
MAIL_PORT=2525
MAIL_USERNAME=null
MAIL_PASSWORD=null
MAIL_ENCRYPTION=null
MAIL_FROM_ADDRESS="hello@example.com"
MAIL_FROM_NAME="${APP_NAME}"

AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_DEFAULT_REGION=us-east-1
AWS_BUCKET=
AWS_USE_PATH_STYLE_ENDPOINT=false

VITE_APP_NAME="${APP_NAME}"
VITE_APP_ADMIN_PATH="${APP_ADMIN_PATH}"

REVERB_APP_ID=1
REVERB_APP_KEY=2
REVERB_APP_SECRET=3
REVERB_HOST=localhost
REVERB_PORT=8080
REVERB_SCHEME=http

VITE_REVERB_APP_KEY="${REVERB_APP_KEY}"
VITE_REVERB_HOST="${REVERB_HOST}"
VITE_REVERB_PORT="${REVERB_PORT}"
VITE_REVERB_SCHEME="${REVERB_SCHEME}"

Install:

docker run --rm \
    -u "$(id -u):$(id -g)" \
    -v "$(pwd):/var/www/html" \
    -w /var/www/html \
    laravelsail/php83-composer:latest \
    composer install --ignore-platform-reqs

sail up -d
sail npm i
sail npm run build
sail php artisan migrate
sail php artisan db:seed

Open two browsers and navigate to http://localhost/admin/login. Log in as admin@admin.com in one browser and as test@test.com in the other.

From the test@test.com account, navigate to "Users" and start a "Chat" with the admin user. Ensure that the admin user remains on the Dashboard page.

If you send a message to the admin user from the test@test.com account, you should receive a notification on the admin side.

If everything works correctly, run the following command:

sail php artisan reverb:restart

and repeat the steps to verify the functionality. After running the command, an error should appear.

If everything works correctly again, go to vendor/laravel/reverb/src/Servers/Reverb/Http/Server.php:58 and insert logger()->debug($request->getBody());. Then, run sail php artisan reverb:restart.

In my case, these two steps reproduce the error.

driesvints commented 4 months ago

@alexsyvolap there's way too much code in that repo for us to review. I'm sure that not all of that is needed to demonstrate the issue. Could you please re-create the repo with the most minimum amount of changes needed to demonstrate the issue you're facing?

alexsyvolap commented 4 months ago

@driesvints I provided only a small part of what I have. I don't understand how much I need to shorten it to reproduce the problem. It exists now on this stack.

joedixon commented 4 months ago

@alexsyvolap can you confirm the issue only occurs after running reverb:restart?

alexsyvolap commented 4 months ago

@joedixon the problem arises on the production environment from the beginning of the Docker container launch. Locally, I reproduce it as described above after restarting reverb.

joedixon commented 4 months ago

Hey @alexsyvolap I was only able to recreate the issue when sending a message at the exact time the server was restarting. I was able to mitigate that by wrapping the broadcast in a retry - something like the following:

retry(5, fn () => broadcast(new MessageSent($message))->toOthers(), 500);

On a side note, it looks like your code is binding to the connected event in echo.js and so you will receive multiple events if the client needs to reconnect (such as when the server has been restarted).

RasmusGodske commented 2 months ago

I encountered the same error mentioned in this thread when running my Laravel + Reverb setup in production. In my setup, approximately 80% of the broadcasts would fail, while the remaining 20% succeeded.

Production Setup Details:

Both the Reverb and Laravel applications were configured behind a Cloudflare tunnel, which functioned as a reverse proxy. Here's how the setup was configured:

Reverb Container: https://reverb.example.com:443 (public URL) routes through Cloudflare Tunnel to http://reverb-container:80 (internal network).
Laravel App: https://laravel.example.com:443 (public URL) routes through Cloudflare Tunnel to http://laravel-container:80 (internal network).

Issue Description: Initially, my Laravel application accessed Reverb at its public endpoint (https://reverb.example.com). It was under this configuration that I experienced the frequent broadcast failures as outlined in this issue.

Solution That Worked: I resolved the issue by bypassing the Cloudflare tunnel for internal communications between Laravel and Reverb. Instead of using the public URL, I configured Laravel to access Reverb's internal URL directly (http://reverb-container:80). Following this change, all broadcasts were successful.

This configuration might suggest that there is an issue with how Pusher handles TLS.

laravel / reverb