search

statamic / cms

The core Laravel CMS Composer package
https://statamic.com
Other
4.06k stars 530 forks source link

Failing Statamic Outpost responses resulting in 500 Errors on Licensed Statamic sites #9359

Closed sjclark closed 9 months ago

sjclark commented 9 months ago

Bug description

The Statamic Control Panels of all my licensed Statamic 3 & 4 sites are currently down as of ~5pm AEST. Root cause: HTTP 500 - Internal Server Error Incident started at 2024-01-19 05:47:38 Client error: POST https://outpost.statamic.com/v3/query resulted in a 403 Forbidden response

Seems to be others with the same issue in the Statamic Discord.

How to reproduce

Try use the Control Panel on any live site - results in a 500 Error - as per logs

[2024-01-19 05:51:32] production.ERROR: Client error: `POST https://outpost.statamic.com/v3/query` resulted in a `403 Forbidden` response:
<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title><meta http-equiv="Content-Type" content="text/htm (truncated...)
 {"exception":"[object] (GuzzleHttp\\Exception\\ClientException(code: 403): Client error: `POST https://outpost.statamic.com/v3/query` resulted in a `403 Forbidden` response:
<!DOCTYPE html><html lang=\"en-US\"><head><title>Just a moment...</title><meta http-equiv=\"Content-Type\" content=\"text/htm (truncated...)
 at /home/ploi/site.co.nz/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113)
[stacktrace]

Logs

[2024-01-19 05:51:32] production.ERROR: Client error: `POST https://outpost.statamic.com/v3/query` resulted in a `403 Forbidden` response:
<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title><meta http-equiv="Content-Type" content="text/htm (truncated...)
 {"exception":"[object] (GuzzleHttp\\Exception\\ClientException(code: 403): Client error: `POST https://outpost.statamic.com/v3/query` resulted in a `403 Forbidden` response:
<!DOCTYPE html><html lang=\"en-US\"><head><title>Just a moment...</title><meta http-equiv=\"Content-Type\" content=\"text/htm (truncated...)
 at /home/ploi/site.co.nz/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113)
[stacktrace]
#0 /home/ploi/site.co.nz/vendor/guzzlehttp/guzzle/src/Middleware.php(72): GuzzleHttp\\Exception\\RequestException::create()
#1 /home/ploi/site.co.nz/vendor/guzzlehttp/promises/src/Promise.php(209): GuzzleHttp\\Middleware::GuzzleHttp\\{closure}()
#2 /home/ploi/site.co.nz/vendor/guzzlehttp/promises/src/Promise.php(158): GuzzleHttp\\Promise\\Promise::callHandler()
#3 /home/ploi/site.co.nz/vendor/guzzlehttp/promises/src/TaskQueue.php(52): GuzzleHttp\\Promise\\Promise::GuzzleHttp\\Promise\\{closure}()
#4 /home/ploi/site.co.nz/vendor/guzzlehttp/promises/src/Promise.php(251): GuzzleHttp\\Promise\\TaskQueue->run()
#5 /home/ploi/site.co.nz/vendor/guzzlehttp/promises/src/Promise.php(227): GuzzleHttp\\Promise\\Promise->invokeWaitFn()
#6 /home/ploi/site.co.nz/vendor/guzzlehttp/promises/src/Promise.php(272): GuzzleHttp\\Promise\\Promise->waitIfPending()
#7 /home/ploi/site.co.nz/vendor/guzzlehttp/promises/src/Promise.php(229): GuzzleHttp\\Promise\\Promise->invokeWaitList()
#8 /home/ploi/site.co.nz/vendor/guzzlehttp/promises/src/Promise.php(69): GuzzleHttp\\Promise\\Promise->waitIfPending()
#9 /home/ploi/site.co.nz/vendor/guzzlehttp/guzzle/src/Client.php(189): GuzzleHttp\\Promise\\Promise->wait()
#10 /home/ploi/site.co.nz/vendor/statamic/cms/src/Licensing/Outpost.php(63): GuzzleHttp\\Client->request()
#11 /home/ploi/site.co.nz/vendor/statamic/cms/src/Licensing/Outpost.php(58): Statamic\\Licensing\\Outpost->performRequest()
#12 /home/ploi/site.co.nz/vendor/statamic/cms/src/Licensing/Outpost.php(48): Statamic\\Licensing\\Outpost->performAndCacheRequest()
#13 /home/ploi/site.co.nz/vendor/statamic/cms/src/Licensing/Outpost.php(38): Statamic\\Licensing\\Outpost->request()
#14 /home/ploi/site.co.nz/vendor/statamic/cms/src/Licensing/Outpost.php(33): Statamic\\Licensing\\Outpost->response()
#15 /home/ploi/site.co.nz/vendor/statamic/cms/src/Http/Middleware/CP/ContactOutpost.php(19): Statamic\\Licensing\\Outpost->radio()
#16 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Statamic\\Http\\Middleware\\CP\\ContactOutpost->handle()
#17 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Routing/Middleware/SubstituteBindings.php(50): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#18 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\\Routing\\Middleware\\SubstituteBindings->handle()
#19 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/VerifyCsrfToken.php(78): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#20 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\\Foundation\\Http\\Middleware\\VerifyCsrfToken->handle()
#21 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/View/Middleware/ShareErrorsFromSession.php(49): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#22 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\\View\\Middleware\\ShareErrorsFromSession->handle()
#23 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(121): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#24 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(64): Illuminate\\Session\\Middleware\\StartSession->handleStatefulRequest()
#25 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\\Session\\Middleware\\StartSession->handle()
#26 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/AddQueuedCookiesToResponse.php(37): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#27 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\\Cookie\\Middleware\\AddQueuedCookiesToResponse->handle()
#28 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/EncryptCookies.php(67): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#29 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\\Cookie\\Middleware\\EncryptCookies->handle()
#30 /home/ploi/site.co.nz/vendor/statamic/cms/src/Http/Middleware/SwapExceptionHandler.php(19): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#31 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Statamic\\Http\\Middleware\\SwapExceptionHandler->handle()
#32 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(103): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#33 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Routing/Router.php(719): Illuminate\\Pipeline\\Pipeline->then()
#34 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Routing/Router.php(698): Illuminate\\Routing\\Router->runRouteWithinStack()
#35 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Routing/Router.php(662): Illuminate\\Routing\\Router->runRoute()
#36 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Routing/Router.php(651): Illuminate\\Routing\\Router->dispatchToRoute()
#37 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(167): Illuminate\\Routing\\Router->dispatch()
#38 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(128): Illuminate\\Foundation\\Http\\Kernel->Illuminate\\Foundation\\Http\\{closure}()
#39 /home/ploi/site.co.nz/vendor/statamic/cms/src/Http/Middleware/DisableFloc.php(18): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#40 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Statamic\\Http\\Middleware\\DisableFloc->handle()
#41 /home/ploi/site.co.nz/vendor/statamic/cms/src/Http/Middleware/CheckMultisite.php(14): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#42 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Statamic\\Http\\Middleware\\CheckMultisite->handle()
#43 /home/ploi/site.co.nz/vendor/statamic/cms/src/Http/Middleware/CheckComposerJsonScripts.php(14): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#44 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Statamic\\Http\\Middleware\\CheckComposerJsonScripts->handle()
#45 /home/ploi/site.co.nz/vendor/statamic/cms/src/Http/Middleware/PoweredByHeader.php(19): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#46 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Statamic\\Http\\Middleware\\PoweredByHeader->handle()
#47 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(21): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#48 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ConvertEmptyStringsToNull.php(31): Illuminate\\Foundation\\Http\\Middleware\\TransformsRequest->handle()
#49 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\\Foundation\\Http\\Middleware\\ConvertEmptyStringsToNull->handle()
#50 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(21): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#51 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TrimStrings.php(40): Illuminate\\Foundation\\Http\\Middleware\\TransformsRequest->handle()
#52 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\\Foundation\\Http\\Middleware\\TrimStrings->handle()
#53 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ValidatePostSize.php(27): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#54 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\\Foundation\\Http\\Middleware\\ValidatePostSize->handle()
#55 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/PreventRequestsDuringMaintenance.php(86): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#56 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Illuminate\\Foundation\\Http\\Middleware\\PreventRequestsDuringMaintenance->handle()
#57 /home/ploi/site.co.nz/vendor/fruitcake/laravel-cors/src/HandleCors.php(38): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#58 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Fruitcake\\Cors\\HandleCors->handle()
#59 /home/ploi/site.co.nz/vendor/fideloper/proxy/src/TrustProxies.php(57): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#60 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(167): Fideloper\\Proxy\\TrustProxies->handle()
#61 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(103): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#62 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(142): Illuminate\\Pipeline\\Pipeline->then()
#63 /home/ploi/site.co.nz/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(111): Illuminate\\Foundation\\Http\\Kernel->sendRequestThroughRouter()
#64 /home/ploi/site.co.nz/public/index.php(51): Illuminate\\Foundation\\Http\\Kernel->handle()
#65 {main}
"}

Environment

All licensed V3 and V4 sites
Some reports unlicensed and "staging" enviroment sites work fine

Installation

Fresh statamic/statamic site via CLI

Antlers Parser

Runtime (default)

Additional details

Sites are hosted on Ploi, php8.1 & 8.2, predominantly Lets Encrypt SSL. Some sites behind Cloudflare. Front-end of sites (if using Static Caching) works fine.

Occuring on sites with both current and expired licenses.

I've noticed the Statamic site itself seems to currently be behind interstitial Cloudflare checkers - may be related to that. Might need to whitelist/blacklist from Cloudflare.

Seems fairly fragile that the license server failing would pull down all sites? (but maybe I'm just 5pm friday grumpy) 😂

bruceontheloose commented 9 months ago

Thanks for reporting it; we're experiencing the same thing.

dadaxr commented 9 months ago

same here

comankey commented 9 months ago

experiencing the same on a few of our sites

modrictin commented 9 months ago

Same 😬

isarphilipp commented 9 months ago

100% this is Cloudflare protection. You can see the "Just a moment..." message

isarphilipp commented 9 months ago
image
duncanmcclean commented 9 months ago

We're currently looking into this, sorry!

duncanmcclean commented 9 months ago

This should be resolved now.

jasonvarga commented 9 months ago

Should be resolved now.

But while testing it, I didn't get a 500 error, the CP continued to work. I just got a "could not contact the outpost" banner, as expected. Odd.

We check for a RequestException like you're seeing. I'm not sure why it's not being caught for you. 🤔

jasonvarga commented 9 months ago

What versions of Statamic were you running?

gassius commented 9 months ago

@jasonvarga not OP, but we have a couple of sites running. FWIW the one on 4.44.0 got the outpost errror in the log as described but did not break. Other at 3.3.48 did fail (CP returning 500) with the error

comankey commented 9 months ago

This should be resolved now.

does indeed. thank you very much for the quick fix

duncanmcclean commented 9 months ago

Did anyone have sites >= v4.35.0 that went down?

sjclark commented 9 months ago

Hiyo cheers, resolved for me Resolved at 2024-01-19 07:32:40 - away from computer currently - but don't think any of my sites would be beyond 4.35.0 atm

duncanmcclean commented 9 months ago

I've managed to figure out why some people were experiencing server errors, while others were seeing the expected "could not contact the outpost" message in the Control Panel.

From testing, it seems like sites running v4.12.0 and above didn't experience any server errors due to #8475 being included in v4.12.0 which made some improvements around Outpost error handling.

To avoid any potential statamic.com downtime causing server errors in the Control Panel, I'd recommend updating to v4.12.0 or higher.

sebszocinski commented 9 months ago

To avoid any potential statamic.com downtime causing server errors in the Control Panel, I'd recommend updating to v4.12.0 or higher.

I'm sorry but I don't think this is a very good response. We have many sites that although deemed "legacy", are still operational and used by customers on a daily basis that don't necessarily NEED to be updated. Having the whole control panel crash because of an outpost issue seems a bit crazy. If older versions of Statamic are no longer being supported why not just remove this check?

jasonvarga commented 9 months ago

Having the whole control panel crash because of an outpost issue seems a bit crazy.

This was never an intentional behavior, and was fixed once we discovered it happened.

I've just backported that same fix to the 3.4 line and have tagged 3.4.16 if updating all the way to 4.x is not an option for you.