search

georgeboot / laravel-echo-api-gateway

Use Laravel Echo with API Gateway Websockets. Works with Bref.sh and Laravel Vapor.
MIT License
99 stars 23 forks source link

[Error] AWS HTTP error: Client error: POST resulted in a `410 Gone` response #9

Open mikebronner opened 3 years ago

mikebronner commented 3 years ago

Thank you for this package! I'm using it in conjunction with Laravel Vapor. I'm receiving the following error intermittently, multiple times per day:

Aws\ApiGatewayManagementApi\Exception\ApiGatewayManagementApiException Error executing "PostToConnection" on "https://xxxxx.execute-api.us-east-1.amazonaws.com/yyyyy/@connections/DOvZ1cdTIAMCLtA%3D"; AWS HTTP error: Client error: `POST https://xxxxx.execute-api.us-east-1.amazonaws.com/yyyyy/@connections/DOvZ1cdTIAMCLtA%3D` resulted in a `410 Gone` response Unable to parse error information from response - Error parsing JSON: Syntax error
     vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:195 Aws\WrappedHttpHandler::parseError
     vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:97 Aws\WrappedHttpHandler::Aws\{closure}
     vendor/guzzlehttp/promises/src/Promise.php:204 GuzzleHttp\Promise\Promise::callHandler
     vendor/guzzlehttp/promises/src/Promise.php:169 GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}
     vendor/guzzlehttp/promises/src/RejectedPromise.php:42 GuzzleHttp\Promise\RejectedPromise::GuzzleHttp\Promise\{closure}
     vendor/guzzlehttp/promises/src/TaskQueue.php:48 GuzzleHttp\Promise\TaskQueue::run
     vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:158 GuzzleHttp\Handler\CurlMultiHandler::tick
     vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:183 GuzzleHttp\Handler\CurlMultiHandler::execute
     vendor/guzzlehttp/promises/src/Promise.php:248 GuzzleHttp\Promise\Promise::invokeWaitFn
     vendor/guzzlehttp/promises/src/Promise.php:224 GuzzleHttp\Promise\Promise::waitIfPending
     vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList
     vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending
     vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList
     vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending
     vendor/guzzlehttp/promises/src/Promise.php:62 GuzzleHttp\Promise\Promise::wait
     vendor/aws/aws-sdk-php/src/AwsClientTrait.php:58 Aws\AwsClient::execute
     vendor/aws/aws-sdk-php/src/AwsClientTrait.php:86 Aws\AwsClient::__call
     vendor/georgeboot/laravel-echo-api-gateway/src/ConnectionRepository.php:23 Georgeboot\LaravelEchoApiGateway\ConnectionRepository::sendMessage
     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:140 Georgeboot\LaravelEchoApiGateway\Handler::sendMessage
     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:66 Georgeboot\LaravelEchoApiGateway\Handler::handleMessage
     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:32 Georgeboot\LaravelEchoApiGateway\Handler::handleWebsocket
     vendor/bref/bref/src/Event/ApiGateway/WebsocketHandler.php:19 Bref\Event\ApiGateway\WebsocketHandler::handle
     vendor/georgeboot/laravel-echo-api-gateway/src/Commands/VaporHandle.php:60 Georgeboot\LaravelEchoApiGateway\Commands\VaporHandle::handleWebsocketEvent
     vendor/georgeboot/laravel-echo-api-gateway/src/Commands/VaporHandle.php:52 Georgeboot\LaravelEchoApiGateway\Commands\VaporHandle::handle
     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:36 Illuminate\Container\BoundMethod::Illuminate\Container\{closure}
     vendor/laravel/framework/src/Illuminate/Container/Util.php:40 Illuminate\Container\Util::unwrapIfClosure
     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:93 Illuminate\Container\BoundMethod::callBoundMethod
     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:37 Illuminate\Container\BoundMethod::call
     vendor/laravel/framework/src/Illuminate/Container/Container.php:651 Illuminate\Container\Container::call
     vendor/laravel/framework/src/Illuminate/Console/Command.php:136 Illuminate\Console\Command::execute
     vendor/symfony/console/Command/Command.php:299 Symfony\Component\Console\Command\Command::run
     vendor/laravel/framework/src/Illuminate/Console/Command.php:121 Illuminate\Console\Command::run
     vendor/symfony/console/Application.php:978 Symfony\Component\Console\Application::doRunCommand
     vendor/symfony/console/Application.php:295 Symfony\Component\Console\Application::doRun
     vendor/symfony/console/Application.php:167 Symfony\Component\Console\Application::run
     vendor/laravel/framework/src/Illuminate/Console/Application.php:92 Illuminate\Console\Application::run
     vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:129 Illuminate\Foundation\Console\Kernel::handle
     artisan:43 [main]

Caused by: GuzzleHttp\Exception\ClientException Client error: `POST https://xxxxx.execute-api.us-east-1.amazonaws.com/yyyyy/@connections/DOvZ1cdTIAMCLtA%3D` resulted in a `410 Gone` response
     vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113 GuzzleHttp\Exception\RequestException::create
     vendor/guzzlehttp/guzzle/src/Middleware.php:69 GuzzleHttp\Middleware::GuzzleHttp\{closure}
     vendor/guzzlehttp/promises/src/Promise.php:204 GuzzleHttp\Promise\Promise::callHandler
     vendor/guzzlehttp/promises/src/Promise.php:153 GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}
     vendor/guzzlehttp/promises/src/TaskQueue.php:48 GuzzleHttp\Promise\TaskQueue::run
     vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:158 GuzzleHttp\Handler\CurlMultiHandler::tick
     vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:183 GuzzleHttp\Handler\CurlMultiHandler::execute
     vendor/guzzlehttp/promises/src/Promise.php:248 GuzzleHttp\Promise\Promise::invokeWaitFn
     vendor/guzzlehttp/promises/src/Promise.php:224 GuzzleHttp\Promise\Promise::waitIfPending
     vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList
     vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending
     vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList
     vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending
     vendor/guzzlehttp/promises/src/Promise.php:62 GuzzleHttp\Promise\Promise::wait
     vendor/aws/aws-sdk-php/src/AwsClientTrait.php:58 Aws\AwsClient::execute
     vendor/aws/aws-sdk-php/src/AwsClientTrait.php:86 Aws\AwsClient::__call
     vendor/georgeboot/laravel-echo-api-gateway/src/ConnectionRepository.php:23 Georgeboot\LaravelEchoApiGateway\ConnectionRepository::sendMessage
     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:140 Georgeboot\LaravelEchoApiGateway\Handler::sendMessage
     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:66 Georgeboot\LaravelEchoApiGateway\Handler::handleMessage
     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:32 Georgeboot\LaravelEchoApiGateway\Handler::handleWebsocket
     vendor/bref/bref/src/Event/ApiGateway/WebsocketHandler.php:19 Bref\Event\ApiGateway\WebsocketHandler::handle
     vendor/georgeboot/laravel-echo-api-gateway/src/Commands/VaporHandle.php:60 Georgeboot\LaravelEchoApiGateway\Commands\VaporHandle::handleWebsocketEvent
     vendor/georgeboot/laravel-echo-api-gateway/src/Commands/VaporHandle.php:52 Georgeboot\LaravelEchoApiGateway\Commands\VaporHandle::handle
     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:36 Illuminate\Container\BoundMethod::Illuminate\Container\{closure}
     vendor/laravel/framework/src/Illuminate/Container/Util.php:40 Illuminate\Container\Util::unwrapIfClosure
     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:93 Illuminate\Container\BoundMethod::callBoundMethod
     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:37 Illuminate\Container\BoundMethod::call
     vendor/laravel/framework/src/Illuminate/Container/Container.php:651 Illuminate\Container\Container::call
     vendor/laravel/framework/src/Illuminate/Console/Command.php:136 Illuminate\Console\Command::execute
     vendor/symfony/console/Command/Command.php:299 Symfony\Component\Console\Command\Command::run
     vendor/laravel/framework/src/Illuminate/Console/Command.php:121 Illuminate\Console\Command::run
     vendor/symfony/console/Application.php:978 Symfony\Component\Console\Application::doRunCommand
     vendor/symfony/console/Application.php:295 Symfony\Component\Console\Application::doRun
     vendor/symfony/console/Application.php:167 Symfony\Component\Console\Application::run
     vendor/laravel/framework/src/Illuminate/Console/Application.php:92 Illuminate\Console\Application::run
     vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:129 Illuminate\Foundation\Console\Kernel::handle
     artisan:43 [main]

I believe this has the side effect that the Laravel Echo event will not update the site when this error happens. I'm unclear as to what the cause might be.

georgeboot commented 3 years ago

Hi Mike, thanks for reporting!

I believe this is due to the connection being (unexpectedly) closed by the client. This is currently not handled by Echo.

I will investigate and see if I can fix this for you.

georgeboot commented 3 years ago

@mikebronner can you please test the dev-master to verify this issue has been fixed now (e41a068)? Once I can confirm, I will tag a new version.

mikebronner commented 3 years ago

Hi @georgeboot, thank you so much for the ultra-quick response! :) I will update and monitor over the next few days and report back. Thank you!

mikebronner commented 3 years ago

Hi @georgeboot It appears we are still getting them. These might only be occurring when we are deploying, and not on every deployment. I wonder if this happens when vapor deploys midstream of an Echo event?

Update: may not be related to deployments after all. The latest error happened 5 minutes after the deployment completed.

georgeboot commented 3 years ago

Yeah I think I am catching the exception incorrectly. But it's hard to catch because AWS doesn't document that topic very good (to my knowledge).

I will look further tomorrow.

mikebronner commented 3 years ago

NP, thank you for all your help with this!

Further Update:

This may be related to queued events on vapor. I have several failed queue jobs that triggered this error. I will keep monitoring this.

georgeboot commented 3 years ago

Mind trying the latest master? AWS SDK has a funny way of handling exceptions but I believe it should be correct this time.

mikebronner commented 3 years ago

Absolutely, we'll get it in our next deployment today and keep monitoring. :)

georgeboot commented 3 years ago

@mikebronner is the issue fixed now?

semsphy commented 3 years ago

@mikebronner is the issue fixed now?

Nope. I just switched to dev-master and redeployed, I still got the error message in my log.

ERROR: Error executing "PostToConnection" on "https://xxxxx-api.ap-southeast-1.amazonaws.com/dev/@connections/DfbxWdFeyQ0CGKg%3D"; AWS HTTP error: Client error: `xxxxxx` resulted in a `403 Forbidden` response:
{"message":"Forbidden"}
georgeboot commented 3 years ago

That is a different error. You previously got a 410 Gone, the one you are getting now is a 403 Forbidden.

semsphy commented 3 years ago

That is a different error. You previously got a 410 Gone, the one you are getting now is a 403 Forbidden.

Whoop, let's me re-test and let's you know.

semsphy commented 3 years ago

@georgeboot , confirmed, it is fixed. I got not error now.

mikebronner commented 3 years ago

@georgeboot I installed the update about 2 hrs ago, haven't seen the error come back yet, will continue to monitor.

mikebronner commented 3 years ago

It appears that I might still be getting the error. Here is the full stack trace:

Aws\ApiGatewayManagementApi\Exception\ApiGatewayManagementApiException Error executing "PostToConnection" on "https://wxj1pjq938.execute-api.us-east-1.amazonaws.com/production/@connections/Df14mfUXIAMCIyg%3D"; AWS HTTP error: Client error: `POST https://wxj1pjq938.execute-api.us-east-1.amazonaws.com/production/@connections/Df14mfUXIAMCIyg%3D` resulted in a `410 Gone` response Unable to parse error information from response - Error parsing JSON: Syntax error      vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:195 Aws\WrappedHttpHandler::parseError     vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:97 Aws\WrappedHttpHandler::Aws\{closure}     vendor/guzzlehttp/promises/src/Promise.php:204 GuzzleHttp\Promise\Promise::callHandler     vendor/guzzlehttp/promises/src/Promise.php:169 GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}     vendor/guzzlehttp/promises/src/RejectedPromise.php:42 GuzzleHttp\Promise\RejectedPromise::GuzzleHttp\Promise\{closure}     vendor/guzzlehttp/promises/src/TaskQueue.php:48 GuzzleHttp\Promise\TaskQueue::run     vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:158 GuzzleHttp\Handler\CurlMultiHandler::tick     vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:183 GuzzleHttp\Handler\CurlMultiHandler::execute     vendor/guzzlehttp/promises/src/Promise.php:248 GuzzleHttp\Promise\Promise::invokeWaitFn     vendor/guzzlehttp/promises/src/Promise.php:224 GuzzleHttp\Promise\Promise::waitIfPending     vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList     vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending     vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList     vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending     vendor/guzzlehttp/promises/src/Promise.php:62 GuzzleHttp\Promise\Promise::wait     vendor/aws/aws-sdk-php/src/AwsClientTrait.php:58 Aws\AwsClient::execute     vendor/aws/aws-sdk-php/src/AwsClientTrait.php:86 Aws\AwsClient::__call     vendor/georgeboot/laravel-echo-api-gateway/src/ConnectionRepository.php:27 Georgeboot\LaravelEchoApiGateway\ConnectionRepository::sendMessage     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:139 Georgeboot\LaravelEchoApiGateway\Handler::sendMessage     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:117 Georgeboot\LaravelEchoApiGateway\Handler::subscribe     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:69 Georgeboot\LaravelEchoApiGateway\Handler::handleMessage     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:31 Georgeboot\LaravelEchoApiGateway\Handler::handleWebsocket     vendor/bref/bref/src/Event/ApiGateway/WebsocketHandler.php:19 Bref\Event\ApiGateway\WebsocketHandler::handle     vendor/georgeboot/laravel-echo-api-gateway/src/Commands/VaporHandle.php:60 Georgeboot\LaravelEchoApiGateway\Commands\VaporHandle::handleWebsocketEvent     vendor/georgeboot/laravel-echo-api-gateway/src/Commands/VaporHandle.php:52 Georgeboot\LaravelEchoApiGateway\Commands\VaporHandle::handle     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:36 Illuminate\Container\BoundMethod::Illuminate\Container\{closure}     vendor/laravel/framework/src/Illuminate/Container/Util.php:40 Illuminate\Container\Util::unwrapIfClosure     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:93 Illuminate\Container\BoundMethod::callBoundMethod     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:37 Illuminate\Container\BoundMethod::call     vendor/laravel/framework/src/Illuminate/Container/Container.php:651 Illuminate\Container\Container::call     vendor/laravel/framework/src/Illuminate/Console/Command.php:136 Illuminate\Console\Command::execute     vendor/symfony/console/Command/Command.php:299 Symfony\Component\Console\Command\Command::run     vendor/laravel/framework/src/Illuminate/Console/Command.php:121 Illuminate\Console\Command::run     vendor/symfony/console/Application.php:978 Symfony\Component\Console\Application::doRunCommand     vendor/symfony/console/Application.php:295 Symfony\Component\Console\Application::doRun     vendor/symfony/console/Application.php:167 Symfony\Component\Console\Application::run     vendor/laravel/framework/src/Illuminate/Console/Application.php:92 Illuminate\Console\Application::run     vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:129 Illuminate\Foundation\Console\Kernel::handle     artisan:43 [main] 

Caused by: GuzzleHttp\Exception\ClientException Client error: `POST https://wxj1pjq938.execute-api.us-east-1.amazonaws.com/production/@connections/Df14mfUXIAMCIyg%3D` resulted in a `410 Gone` response      vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113 GuzzleHttp\Exception\RequestException::create     vendor/guzzlehttp/guzzle/src/Middleware.php:69 GuzzleHttp\Middleware::GuzzleHttp\{closure}     vendor/guzzlehttp/promises/src/Promise.php:204 GuzzleHttp\Promise\Promise::callHandler     vendor/guzzlehttp/promises/src/Promise.php:153 GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}     vendor/guzzlehttp/promises/src/TaskQueue.php:48 GuzzleHttp\Promise\TaskQueue::run     vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:158 GuzzleHttp\Handler\CurlMultiHandler::tick     vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:183 GuzzleHttp\Handler\CurlMultiHandler::execute     vendor/guzzlehttp/promises/src/Promise.php:248 GuzzleHttp\Promise\Promise::invokeWaitFn     vendor/guzzlehttp/promises/src/Promise.php:224 GuzzleHttp\Promise\Promise::waitIfPending     vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList     vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending     vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList     vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending     vendor/guzzlehttp/promises/src/Promise.php:62 GuzzleHttp\Promise\Promise::wait     vendor/aws/aws-sdk-php/src/AwsClientTrait.php:58 Aws\AwsClient::execute     vendor/aws/aws-sdk-php/src/AwsClientTrait.php:86 Aws\AwsClient::__call     vendor/georgeboot/laravel-echo-api-gateway/src/ConnectionRepository.php:27 Georgeboot\LaravelEchoApiGateway\ConnectionRepository::sendMessage     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:139 Georgeboot\LaravelEchoApiGateway\Handler::sendMessage     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:117 Georgeboot\LaravelEchoApiGateway\Handler::subscribe     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:69 Georgeboot\LaravelEchoApiGateway\Handler::handleMessage     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:31 Georgeboot\LaravelEchoApiGateway\Handler::handleWebsocket     vendor/bref/bref/src/Event/ApiGateway/WebsocketHandler.php:19 Bref\Event\ApiGateway\WebsocketHandler::handle     vendor/georgeboot/laravel-echo-api-gateway/src/Commands/VaporHandle.php:60 Georgeboot\LaravelEchoApiGateway\Commands\VaporHandle::handleWebsocketEvent     vendor/georgeboot/laravel-echo-api-gateway/src/Commands/VaporHandle.php:52 Georgeboot\LaravelEchoApiGateway\Commands\VaporHandle::handle     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:36 Illuminate\Container\BoundMethod::Illuminate\Container\{closure}     vendor/laravel/framework/src/Illuminate/Container/Util.php:40 Illuminate\Container\Util::unwrapIfClosure     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:93 Illuminate\Container\BoundMethod::callBoundMethod     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:37 Illuminate\Container\BoundMethod::call     vendor/laravel/framework/src/Illuminate/Container/Container.php:651 Illuminate\Container\Container::call     vendor/laravel/framework/src/Illuminate/Console/Command.php:136 Illuminate\Console\Command::execute     vendor/symfony/console/Command/Command.php:299 Symfony\Component\Console\Command\Command::run     vendor/laravel/framework/src/Illuminate/Console/Command.php:121 Illuminate\Console\Command::run     vendor/symfony/console/Application.php:978 Symfony\Component\Console\Application::doRunCommand     vendor/symfony/console/Application.php:295 Symfony\Component\Console\Application::doRun     vendor/symfony/console/Application.php:167 Symfony\Component\Console\Application::run     vendor/laravel/framework/src/Illuminate/Console/Application.php:92 Illuminate\Console\Application::run     vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:129 Illuminate\Foundation\Console\Kernel::handle     artisan:43 [main]
georgeboot commented 3 years ago

Grr. Why does AWS make it so hard to mock errors 🙄

Will dive in tomorrow and hopefully fix this once and for real. Apologies for the hassle!

mikebronner commented 3 years ago

No fret, @georgeboot, thanks for all your work on this! Also no rush, I don't think these are causing issues?

You mentioned the solution is to close the connection and drop the pending message. Is it technically possible to resubmit the message, if it failed, to an open connection, or is the new connection information not available at that point?

georgeboot commented 3 years ago

Is it technically possible to resubmit the message, if it failed, to an open connection, or is the new connection information not available at that point?

Nope. Connections can only be initiated from the client side.

I believe pusher will also just ignore the message.

Once I have implemented client-reconnects (#2), this will no longer be an issue as the old connections details will be purged and Echo will only see the new connection.

semsphy commented 3 years ago

It appears that I might still be getting the error. Here is the full stack trace:

Aws\ApiGatewayManagementApi\Exception\ApiGatewayManagementApiException Error executing "PostToConnection" on "https://wxj1pjq938.execute-api.us-east-1.amazonaws.com/production/@connections/Df14mfUXIAMCIyg%3D"; AWS HTTP error: Client error: `POST https://wxj1pjq938.execute-api.us-east-1.amazonaws.com/production/@connections/Df14mfUXIAMCIyg%3D` resulted in a `410 Gone` response Unable to parse error information from response - Error parsing JSON: Syntax error      vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:195 Aws\WrappedHttpHandler::parseError     vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:97 Aws\WrappedHttpHandler::Aws\{closure}     vendor/guzzlehttp/promises/src/Promise.php:204 GuzzleHttp\Promise\Promise::callHandler     vendor/guzzlehttp/promises/src/Promise.php:169 GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}     vendor/guzzlehttp/promises/src/RejectedPromise.php:42 GuzzleHttp\Promise\RejectedPromise::GuzzleHttp\Promise\{closure}     vendor/guzzlehttp/promises/src/TaskQueue.php:48 GuzzleHttp\Promise\TaskQueue::run     vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:158 GuzzleHttp\Handler\CurlMultiHandler::tick     vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:183 GuzzleHttp\Handler\CurlMultiHandler::execute     vendor/guzzlehttp/promises/src/Promise.php:248 GuzzleHttp\Promise\Promise::invokeWaitFn     vendor/guzzlehttp/promises/src/Promise.php:224 GuzzleHttp\Promise\Promise::waitIfPending     vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList     vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending     vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList     vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending     vendor/guzzlehttp/promises/src/Promise.php:62 GuzzleHttp\Promise\Promise::wait     vendor/aws/aws-sdk-php/src/AwsClientTrait.php:58 Aws\AwsClient::execute     vendor/aws/aws-sdk-php/src/AwsClientTrait.php:86 Aws\AwsClient::__call     vendor/georgeboot/laravel-echo-api-gateway/src/ConnectionRepository.php:27 Georgeboot\LaravelEchoApiGateway\ConnectionRepository::sendMessage     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:139 Georgeboot\LaravelEchoApiGateway\Handler::sendMessage     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:117 Georgeboot\LaravelEchoApiGateway\Handler::subscribe     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:69 Georgeboot\LaravelEchoApiGateway\Handler::handleMessage     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:31 Georgeboot\LaravelEchoApiGateway\Handler::handleWebsocket     vendor/bref/bref/src/Event/ApiGateway/WebsocketHandler.php:19 Bref\Event\ApiGateway\WebsocketHandler::handle     vendor/georgeboot/laravel-echo-api-gateway/src/Commands/VaporHandle.php:60 Georgeboot\LaravelEchoApiGateway\Commands\VaporHandle::handleWebsocketEvent     vendor/georgeboot/laravel-echo-api-gateway/src/Commands/VaporHandle.php:52 Georgeboot\LaravelEchoApiGateway\Commands\VaporHandle::handle     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:36 Illuminate\Container\BoundMethod::Illuminate\Container\{closure}     vendor/laravel/framework/src/Illuminate/Container/Util.php:40 Illuminate\Container\Util::unwrapIfClosure     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:93 Illuminate\Container\BoundMethod::callBoundMethod     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:37 Illuminate\Container\BoundMethod::call     vendor/laravel/framework/src/Illuminate/Container/Container.php:651 Illuminate\Container\Container::call     vendor/laravel/framework/src/Illuminate/Console/Command.php:136 Illuminate\Console\Command::execute     vendor/symfony/console/Command/Command.php:299 Symfony\Component\Console\Command\Command::run     vendor/laravel/framework/src/Illuminate/Console/Command.php:121 Illuminate\Console\Command::run     vendor/symfony/console/Application.php:978 Symfony\Component\Console\Application::doRunCommand     vendor/symfony/console/Application.php:295 Symfony\Component\Console\Application::doRun     vendor/symfony/console/Application.php:167 Symfony\Component\Console\Application::run     vendor/laravel/framework/src/Illuminate/Console/Application.php:92 Illuminate\Console\Application::run     vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:129 Illuminate\Foundation\Console\Kernel::handle     artisan:43 [main] 

Caused by: GuzzleHttp\Exception\ClientException Client error: `POST https://wxj1pjq938.execute-api.us-east-1.amazonaws.com/production/@connections/Df14mfUXIAMCIyg%3D` resulted in a `410 Gone` response      vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113 GuzzleHttp\Exception\RequestException::create     vendor/guzzlehttp/guzzle/src/Middleware.php:69 GuzzleHttp\Middleware::GuzzleHttp\{closure}     vendor/guzzlehttp/promises/src/Promise.php:204 GuzzleHttp\Promise\Promise::callHandler     vendor/guzzlehttp/promises/src/Promise.php:153 GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}     vendor/guzzlehttp/promises/src/TaskQueue.php:48 GuzzleHttp\Promise\TaskQueue::run     vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:158 GuzzleHttp\Handler\CurlMultiHandler::tick     vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php:183 GuzzleHttp\Handler\CurlMultiHandler::execute     vendor/guzzlehttp/promises/src/Promise.php:248 GuzzleHttp\Promise\Promise::invokeWaitFn     vendor/guzzlehttp/promises/src/Promise.php:224 GuzzleHttp\Promise\Promise::waitIfPending     vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList     vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending     vendor/guzzlehttp/promises/src/Promise.php:269 GuzzleHttp\Promise\Promise::invokeWaitList     vendor/guzzlehttp/promises/src/Promise.php:226 GuzzleHttp\Promise\Promise::waitIfPending     vendor/guzzlehttp/promises/src/Promise.php:62 GuzzleHttp\Promise\Promise::wait     vendor/aws/aws-sdk-php/src/AwsClientTrait.php:58 Aws\AwsClient::execute     vendor/aws/aws-sdk-php/src/AwsClientTrait.php:86 Aws\AwsClient::__call     vendor/georgeboot/laravel-echo-api-gateway/src/ConnectionRepository.php:27 Georgeboot\LaravelEchoApiGateway\ConnectionRepository::sendMessage     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:139 Georgeboot\LaravelEchoApiGateway\Handler::sendMessage     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:117 Georgeboot\LaravelEchoApiGateway\Handler::subscribe     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:69 Georgeboot\LaravelEchoApiGateway\Handler::handleMessage     vendor/georgeboot/laravel-echo-api-gateway/src/Handler.php:31 Georgeboot\LaravelEchoApiGateway\Handler::handleWebsocket     vendor/bref/bref/src/Event/ApiGateway/WebsocketHandler.php:19 Bref\Event\ApiGateway\WebsocketHandler::handle     vendor/georgeboot/laravel-echo-api-gateway/src/Commands/VaporHandle.php:60 Georgeboot\LaravelEchoApiGateway\Commands\VaporHandle::handleWebsocketEvent     vendor/georgeboot/laravel-echo-api-gateway/src/Commands/VaporHandle.php:52 Georgeboot\LaravelEchoApiGateway\Commands\VaporHandle::handle     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:36 Illuminate\Container\BoundMethod::Illuminate\Container\{closure}     vendor/laravel/framework/src/Illuminate/Container/Util.php:40 Illuminate\Container\Util::unwrapIfClosure     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:93 Illuminate\Container\BoundMethod::callBoundMethod     vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php:37 Illuminate\Container\BoundMethod::call     vendor/laravel/framework/src/Illuminate/Container/Container.php:651 Illuminate\Container\Container::call     vendor/laravel/framework/src/Illuminate/Console/Command.php:136 Illuminate\Console\Command::execute     vendor/symfony/console/Command/Command.php:299 Symfony\Component\Console\Command\Command::run     vendor/laravel/framework/src/Illuminate/Console/Command.php:121 Illuminate\Console\Command::run     vendor/symfony/console/Application.php:978 Symfony\Component\Console\Application::doRunCommand     vendor/symfony/console/Application.php:295 Symfony\Component\Console\Application::doRun     vendor/symfony/console/Application.php:167 Symfony\Component\Console\Application::run     vendor/laravel/framework/src/Illuminate/Console/Application.php:92 Illuminate\Console\Application::run     vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php:129 Illuminate\Foundation\Console\Kernel::handle     artisan:43 [main]

Hi @mikebronner , I faced this problem when I broadcasted my message. How about you?

mikebronner commented 3 years ago

@semsphy I believe that is the case as well. Is there another scenario this could happen? (Just asking since this package is for event broadcasting, and I didn't know if I maybe missed something to troubleshoot.)

semsphy commented 3 years ago

@mikebronner , indeed, there is only this scenario. I just wanted to confirm your scenario. :)

georgeboot commented 3 years ago

Can you try the dev-master again? Tried something different in 957ce4f

semsphy commented 3 years ago 

[PRODUCTION]: 2021-08-06 12:41:19🚨ERROR: Error executing "PostToConnection" on "https://xxxx.execute-api.ap-southeast-1.amazonaws.com/prod/@connections/DfyBYdK2SQ0CGCA%3D"; AWS HTTP error: Client error: `POST https://xxxxx.execute-api.ap-southeast-1.amazonaws.com/prod/@connections/DfyBYdK2SQ0CGCA%3D` resulted in a `410 Gone` response Unable to parse error information from response - Error parsing JSON: Syntax error 
[{"exception":"[object] (Aws\\ApiGatewayManagementApi\\Exception\\ApiGatewayManagementApiException(code: 0): Error executing \"PostToConnection\" on \"https://xxxxx.execute-api.ap-southeast-1.amazonaws.com/prod/@connections/DfyBYdK2SQ0CGCA%3D\"; AWS HTTP error: Client error: `POST https://xxxxx.execute-api.ap-southeast-1.amazonaws.com/prod/@connections/DfyBYdK2SQ0CGCA%3D` resulted in a `410 Gone` response Unable to parse error information from response - Error parsing JSON: Syntax error at /var/task/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:195)
[previous exception] [object] (GuzzleHttp\\Exception\\ClientException(code: 410): Client error: `POST https://xxxxx.execute-api.ap-southeast-1.amazonaws.com/prod/@connections/DfyBYdK2SQ0CGCA%3D` resulted in a `410 Gone` response at /var/task/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113)"}]```
mikebronner commented 3 years ago

Just pushed the new update, will keep monitoring

georgeboot commented 3 years ago

Yeah so oke I see what is happening. Thanks for testing for me by the way, appreciate it!

Regardless of what type of error the API returns, it already breaks when trying to set-up the ApiGatewayManagementApiException. I thought perhaps we can just catch the GuzzleException but we cant.

~I will open up a issue at the PHP SDK and make sure this gets fixed in there.~ https://github.com/aws/aws-sdk-php/issues/2285

semsphy commented 3 years ago

I saw the creator of bref, he handled different way. 

use Bref\Websocket\SimpleWebsocketClient;

$websocketClient = SimpleWebsocketClient::create(getenv('API_GATEWAY_ID'), getenv('AWS_REGION'), 'prod',);
foreach ($this->connectionStorage->getAllConnections() as $connectionId) {
    $websocketClient->message($connectionId, $message);
}
georgeboot commented 3 years ago

I saw the creator of bref, he handled different way.

use Bref\Websocket\SimpleWebsocketClient;

$websocketClient = SimpleWebsocketClient::create(getenv('API_GATEWAY_ID'), getenv('AWS_REGION'), 'prod',);
foreach ($this->connectionStorage->getAllConnections() as $connectionId) {
    $websocketClient->message($connectionId, $message);
}

yeah he uses https://github.com/async-aws/aws. Might do that as well, as the main AWS SDK can be a bit of a pain in the ass sometimes.

mikebronner commented 3 years ago

Hi guys, I just wanted to check in on this. I know there is the issue opened with AWS. @georgeboot have you been able to make any progress implementing async-aws? This is becoming critical for us, we are seeing around 14000 errors due to this each month, impacting our user-experience significantly.

Thanks again for all your hard work on this package. :)

georgeboot commented 3 years ago

@mikebronner can you perhaps help with Sam's request in https://github.com/aws/aws-sdk-php/issues/2285#issuecomment-904865573? I'm just too busy to do all the debugging work now.

As soon as AWS SDK fix the issue, this package should have the fix as well. Moving the async-aws is an option, but likely more work.

mikebronner commented 3 years ago

@georgeboot sure thing, I'll give it a shot.

mikebronner commented 3 years ago

@georgeboot quick question: can you point me where to set debug => true, per Sam's request? Is there a way I can do it so that composer won't overwrite it on update?

georgeboot commented 3 years ago

I believe what Sam wants is the following:

  1. Create a websocket API
  2. Establish a connection and write down the connectionId
  3. Close the connection
  4. Send a message to the closed connection using a client with debug = true and capture the full output

Setting debug=true needs to be set here: https://github.com/georgeboot/laravel-echo-api-gateway/blob/957ce4f0bd83754f7be92ffe8ec144b6e8cb71ef/src/ConnectionRepository.php#L17-L20

As you already have the package deployed, you could inspect the DynamoDB table to obtain a connectionId. Wait till it disconnects and send the message from your local computer (using live key ID + secret). Thats the easiest I can think of.

Thanks for helping!

mikebronner commented 3 years ago

@georgeboot Thanks for the hints. We have not been able to replicate the problem, we always get 403 Forbidden when trying to access a closed connection.

georgeboot commented 3 years ago

@mikebronner are you able to post to a valid connection? 403 indicates unauthorised, which is a different error.

mikebronner commented 3 years ago

@georgeboot Yes, but the connection is already gone. We were looking through our logs, and the code for the package, and think we might have an idea. Our AWS guy is thinking that this is due to ConsistentReads defaulting to false. What if you did this:

    public function getConnectionIdsForChannel(string ...$channels): Collection
    {
        $promises = collect($channels)->map(fn($channel) => $this->dynamoDb->queryAsync([
            "ConsistentRead" => true,
            'TableName' => $this->table,
            'IndexName' => 'lookup-by-channel',
            'KeyConditionExpression' => 'channel = :channel',
            'ExpressionAttributeValues' => [
                ':channel' => ['S' => $channel],
            ],
        ]))->toArray();

        $responses = Utils::all($promises)->wait();

        return collect($responses)
            ->flatmap(fn($result) => $result['Items'])
            ->map(fn($item) => $item['connectionId']['S'])
            ->unique();
    }

And then also further down where you run ->query(). That will make sure the DynamoDB instances are synced up and all have the same connections configured.

This is just an observation, since we cannot reproduce it under set conditions, and our thinking is that it is due to the asynchronous nature of DynamoDB instances, and getting that to stay in sync with our app.

I'll submit this in a PR and test.

georgeboot commented 3 years ago

Could be, but sounds like a race condition to me. Happens a bit too much for that, but perhaps you're right.

If happily accept your PR if it fixes the issue 😊

mikebronner commented 3 years ago

I put the PR patch in my composer and will be testing, I'll let you know. :)

jdavidbakr commented 2 years ago

I was getting this same issue when I was trying to implement presence channels. I think this change to src/ConnectionRepository.php should fix this issue - it seems there may be multiple ways that the exception is being thrown.

https://github.com/georgeboot/laravel-echo-api-gateway/pull/25/files