Illuminate\Broadcasting\BroadcastException: Pusher error: cURL error 7: Failed to connect to localhost port 8080 after 0 ms: Couldn't connect to server (see https://curl.haxx.se/libcurl/c/libcurl-errors.html)

cAstraea commented 2 months ago

Reverb Version

1.0

Laravel Version

10.48

PHP Version

8.2.18

Description

The connections reliably fail in my setup. My setup 2 AWS instances , doing ssl offloading to the load balancer so initially they only had the port 80 server block. The load balancer sends traffic 50 - 50 between the two when the host-header is xxxx.domain.com Reverb was running initially on both instances but because I thought this was the issue I've explored 2 options

use the reverb scaling but this didn't seem to work.

option 2 was adding an alias in route 53 names xxxx-ws.domain.com then on the load balancer assign it 100% to instance A. Stopped reverb on server 2. Added a server block in nginx for xxxx-ws.domain.com which contains for example

server {
listen 80;
listen [::]:80;
server_name xxxxx-ws.domain.com;

# Your WebSocket configuration here
location /app {
    access_log /var/log/nginx/websocket_access_LB.log websocket;
    error_log /var/log/nginx/websocket_error_LB.log;

    proxy_pass http://127.0.0.1:8080;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    # Other necessary proxy settings
}
server_tokens off;
root          /srv/www/xxxxxx-xxxx-api/public;

# Force https on Load Balancer
if ($http_x_forwarded_proto = 'http') {
    return 301 https://$host$request_uri;
}

# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;

add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
add_header X-Frame-Options "SAMEORIGIN";
add_header X-XSS-Protection "1; mode=block";
add_header X-Content-Type-Options "nosniff";

# Uploads
client_max_body_size 20m;
#    client_body_temp_path /tmp/nginx;
 client_body_temp_path /tmp/client_body 1 2;
 # Needed for pulse ? Not sure why, otherwise throws an upstream error
 fastcgi_temp_path /tmp/fastcgi
 ........
 }

Adjusted the frontend and now this instance seems to receive the websocket connections however as seem in the attached video it will reliable fail 70-80% of the times and I would find the failed job in the database failed_jobs table

Illuminate\Broadcasting\BroadcastException: Pusher error: cURL error 7: Failed to connect to localhost port 8080 after 0 ms: Couldn't connect to server (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for http://localhost:8080/apps/428988/events?auth_key=jwrsvunbujlagm3rkhhu&auth_timestamp=1720751739&auth_version=1.0&body_md5=144cd9bf38c99afd80422ea44d59a565&auth_signature=7dda8ad8f673e5c9ee746bb584658d91e907ea5870dd648528a3d6301cf15dc0. in /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Broadcasting/Broadcasters/PusherBroadcaster.php:164 Stack trace: #0 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Broadcasting/BroadcastEvent.php(92): Illuminate\Broadcasting\Broadcasters\PusherBroadcaster->broadcast() #1 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Illuminate\Broadcasting\BroadcastEvent->handle() #2 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}() #3 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure() #4 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\Container\BoundMethod::callBoundMethod() #5 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Container/Container.php(662): Illuminate\Container\BoundMethod::call() #6 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(128): Illuminate\Container\Container->call() #7 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(144): Illuminate\Bus\Dispatcher->Illuminate\Bus\{closure}() #8 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(119): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #9 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(132): Illuminate\Pipeline\Pipeline->then() #10 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(123): Illuminate\Bus\Dispatcher->dispatchNow() #11 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(144): Illuminate\Queue\CallQueuedHandler->Illuminate\Queue\{closure}() #12 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(119): Illuminate\Pipeline\Pipeline->Illuminate\Pipeline\{closure}() #13 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(122): Illuminate\Pipeline\Pipeline->then() #14 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(70): Illuminate\Queue\CallQueuedHandler->dispatchThroughMiddleware() #15 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(102): Illuminate\Queue\CallQueuedHandler->call() #16 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(439): Illuminate\Queue\Jobs\Job->fire() #17 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(389): Illuminate\Queue\Worker->process() #18 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(176): Illuminate\Queue\Worker->runJob() #19 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(137): Illuminate\Queue\Worker->daemon() #20 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(120): Illuminate\Queue\Console\WorkCommand->runWorker() #21 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Illuminate\Queue\Console\WorkCommand->handle() #22 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Container/Util.php(41): Illuminate\Container\BoundMethod::Illuminate\Container\{closure}() #23 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\Container\Util::unwrapIfClosure() #24 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(35): Illuminate\Container\BoundMethod::callBoundMethod() #25 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Container/Container.php(662): Illuminate\Container\BoundMethod::call() #26 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Console/Command.php(211): Illuminate\Container\Container->call() #27 /srv/www/svenson-rsv-api/vendor/symfony/console/Command/Command.php(326): Illuminate\Console\Command->execute() #28 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Console/Command.php(180): Symfony\Component\Console\Command\Command->run() #29 /srv/www/svenson-rsv-api/vendor/symfony/console/Application.php(1096): Illuminate\Console\Command->run() #30 /srv/www/svenson-rsv-api/vendor/symfony/console/Application.php(324): Symfony\Component\Console\Application->doRunCommand() #31 /srv/www/svenson-rsv-api/vendor/symfony/console/Application.php(175): Symfony\Component\Console\Application->doRun() #32 /srv/www/svenson-rsv-api/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(201): Symfony\Component\Console\Application->run() #33 /srv/www/svenson-rsv-api/artisan(35): Illuminate\Foundation\Console\Kernel->handle() #34 {main}

Also looking in it seems reverb or some other internal part would simply refuse to broadcast the event that has ShouldBroadcast, this happens about 30% of the time. I'm running reverb with debug and there is no Broadcasting To ................................... private-calendar-updates and on the websocket connection there is no event received , also no errors associated with the event or leftover jobs

Found the reason for this point. it's when the event is handled by the other instance it doesn't broadcast the event In this image the one on the right. So I'm thinking I need to connect the second reverb instance to the first reverb instance somehow 🤔 Having the REVERB_SCALING_ENABLED=true with the same redis on both instances doesn't seem to fix it.

Video

https://github.com/user-attachments/assets/fb496484-9bf3-49af-b93a-1b157153a2d2

Steps To Reproduce

I couldn't replicate this issue in a single instance setup 🤔 I could replicate it reliably in a multi instance setup

Think I finally cracked it ! But have no idea how to implement this... Server A reverb + queue work Server B reverb + queue work

Sometimes the ShouldBroadcast would activate on Server B but all my frontend clients are connected to Reverb on Server A. Reverb on server B doesn't know what to do with the Event it intercepted. So I would need to implement something like implements ShouldBroadCastOnServerA or have it connect to reverb from server A which I am not sure how to do. I tried pointing the settings of reverb on server B to xxxxx-ws.domain.com but that didn't work.

cAstraea commented 2 months ago

I'll close this, the issue stems from the multiple instance. I managed to kinda hot fix it by changing the broadcasted websocket events to only dispatch on a queue handled on the original instance running reverb. If someone has some suggestions when running in a multi server environment please do chime in :) I'm thinking a separate 3rd instance only dedicated for reverb and its queue worker is needed because they can't communicate between them.

joedixon commented 2 months ago

@cAstraea can you elaborate on why Reverb's scaling didn't work? In a multi-server setup, you will have connections for channels handled across servers, so you need a proxy in the middle to ensure all servers receive the messages for the connections they manage. Reverb uses Redis to achieve this.

cAstraea commented 2 months ago

I think it was not configured correctly based on my understanding? I already use the amazon elastic cache redis on both instances for Cache. On server A -> reverb is running, a queue worker is running also, the cache is set to the aws redis. On server B -> reverb is running , a queue worker is running also, the cache is set to the aws redis. On the frontend -> the WS in browser tab A would hit server A, the nginx would route to reverb instance on server A for example but in certain cases the queue worker from server B would intercept the job "meant" for server A and because server B reverb has no knowledge about the socket id that originated from server A the message is lost into ether. Adding the REVERB_SCALING_ENABLED=true on both instances didn't work and I'm not sure sure how to debug it.

joedixon commented 1 month ago

I don't think it should matter which server processes the job because (and this is assuming the queued job is for broadcasting an event), the job will post the event via HTTP to Reverb and Reverb will distribute it to all listening servers. If you run this configuration and use the --debug option, do you see any events passing through?

laravel / reverb