Closed edalzell closed 1 year ago
It's always the same "key"
It looks like this could happen when you use full measure caching: when you clear the regular cache, but not the static cache. Then the page (with the replacement placeholders) would still be cached in a file, but the actual replacements per region would be lost, as they are stored in the regular cache. That would lead to the error in this issue, where the Session
doesn't have the requested region.
@edalzell are you using full measure caching on this site?
@edalzell are you using full measure caching on this site?
No
Cloudflare?
Cloudflare?
Yes
What kind of static caching are you using, if at all?
What kind of static caching are you using, if at all?
1/2 measure.
@edalzell do you get this issue locally?
@edalzell do you get this issue locally?
Nope, but I normally don't do static caching locally. But it is enable on this site and I don't see it in my logs. But the live site gets waaaaaaay more traffic of course.
Are there any workarounds (apart disabling static caching) yet?
Are there any workarounds (apart disabling static caching) yet?
Currently no. So you are having the same issue? Do you have any additional info that might help?
This is still a mystery. Can somebody share a full log file for this issue? Maybe there are more, possibly related errors in it to go on.
[2022-12-08 12:00:39] production.ERROR: Undefined array key "1b042a2794ea6ce863dab6f73d6abc456e010963" {"exception":"[object] (ErrorException(code: 0): Undefined array key \"1b042a2794ea6ce863dab6f73d6abc456e010963\" at /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Collections/Collection.php:1679)
[stacktrace]
#0 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php(259): Illuminate\\Foundation\\Bootstrap\\HandleExceptions->handleError()
#1 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Collections/Collection.php(1679): Illuminate\\Foundation\\Bootstrap\\HandleExceptions->Illuminate\\Foundation\\Bootstrap\\{closure}()
#2 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/NoCache/Session.php(49): Illuminate\\Support\\Collection->offsetGet()
#3 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/Replacers/NoCacheReplacer.php(66): Statamic\\StaticCaching\\NoCache\\Session->region()
#4 [internal function]: Statamic\\StaticCaching\\Replacers\\NoCacheReplacer->Statamic\\StaticCaching\\Replacers\\{closure}()
#5 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/Replacers/NoCacheReplacer.php(61): preg_replace_callback()
#6 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/Replacers/NoCacheReplacer.php(53): Statamic\\StaticCaching\\Replacers\\NoCacheReplacer->performReplacement()
#7 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/Replacers/NoCacheReplacer.php(47): Statamic\\StaticCaching\\Replacers\\NoCacheReplacer->replace()
#8 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/Replacers/NoCacheReplacer.php(32): Statamic\\StaticCaching\\Replacers\\NoCacheReplacer->replaceInResponse()
#9 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/Middleware/Cache.php(71): Statamic\\StaticCaching\\Replacers\\NoCacheReplacer->replaceInCachedResponse()
#10 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Collections/Traits/EnumeratesValues.php(262): Statamic\\StaticCaching\\Middleware\\Cache->Statamic\\StaticCaching\\Middleware\\{closure}()
#11 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/Middleware/Cache.php(71): Illuminate\\Support\\Collection->each()
#12 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/Middleware/Cache.php(42): Statamic\\StaticCaching\\Middleware\\Cache->makeReplacements()
#13 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Statamic\\StaticCaching\\Middleware\\Cache->handle()
#14 /home/forge/alaskapac.org/vendor/statamic/cms/src/Http/Middleware/AuthGuard.php(14): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#15 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Statamic\\Http\\Middleware\\AuthGuard->handle()
#16 /home/forge/alaskapac.org/vendor/statamic/cms/src/Http/Middleware/AddViewPaths.php(20): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#17 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Statamic\\Http\\Middleware\\AddViewPaths->handle()
#18 /home/forge/alaskapac.org/vendor/statamic/cms/src/Http/Middleware/Localize.php(37): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#19 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Statamic\\Http\\Middleware\\Localize->handle()
#20 /home/forge/alaskapac.org/vendor/statamic/cms/src/Http/Middleware/HandleToken.php(13): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#21 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Statamic\\Http\\Middleware\\HandleToken->handle()
#22 /home/forge/alaskapac.org/vendor/statamic/cms/src/Http/Middleware/StacheLock.php(29): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#23 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Statamic\\Http\\Middleware\\StacheLock->handle()
#24 /home/forge/alaskapac.org/vendor/transformstudios/takeover-redirects/src/Http/Middleware/RedirectIfScheduled.php(17): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#25 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): TransformStudios\\TakeoverRedirects\\Http\\Middleware\\RedirectIfScheduled->handle()
#26 /home/forge/alaskapac.org/vendor/transformstudios/gated/src/Http/Middleware/HandleGate.php(18): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#27 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): TransformStudios\\Gated\\Http\\Middleware\\HandleGate->handle()
#28 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Routing/Middleware/SubstituteBindings.php(50): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#29 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Routing\\Middleware\\SubstituteBindings->handle()
#30 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/VerifyCsrfToken.php(78): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#31 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Foundation\\Http\\Middleware\\VerifyCsrfToken->handle()
#32 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/View/Middleware/ShareErrorsFromSession.php(49): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#33 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\View\\Middleware\\ShareErrorsFromSession->handle()
#34 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(121): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#35 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(64): Illuminate\\Session\\Middleware\\StartSession->handleStatefulRequest()
#36 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Session\\Middleware\\StartSession->handle()
#37 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/AddQueuedCookiesToResponse.php(37): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#38 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Cookie\\Middleware\\AddQueuedCookiesToResponse->handle()
#39 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/EncryptCookies.php(67): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#40 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Cookie\\Middleware\\EncryptCookies->handle()
#41 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(116): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#42 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Routing/Router.php(728): Illuminate\\Pipeline\\Pipeline->then()
#43 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Routing/Router.php(705): Illuminate\\Routing\\Router->runRouteWithinStack()
#44 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Routing/Router.php(669): Illuminate\\Routing\\Router->runRoute()
#45 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Routing/Router.php(658): Illuminate\\Routing\\Router->dispatchToRoute()
#46 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(190): Illuminate\\Routing\\Router->dispatch()
#47 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(141): Illuminate\\Foundation\\Http\\Kernel->Illuminate\\Foundation\\Http\\{closure}()
#48 /home/forge/alaskapac.org/vendor/statamic/cms/src/Http/Middleware/DisableFloc.php(18): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#49 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Statamic\\Http\\Middleware\\DisableFloc->handle()
#50 /home/forge/alaskapac.org/vendor/statamic/cms/src/Http/Middleware/CheckMultisite.php(14): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#51 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Statamic\\Http\\Middleware\\CheckMultisite->handle()
#52 /home/forge/alaskapac.org/vendor/statamic/cms/src/Http/Middleware/CheckComposerJsonScripts.php(14): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#53 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Statamic\\Http\\Middleware\\CheckComposerJsonScripts->handle()
#54 /home/forge/alaskapac.org/vendor/statamic/cms/src/Http/Middleware/PoweredByHeader.php(19): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#55 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Statamic\\Http\\Middleware\\PoweredByHeader->handle()
#56 /home/forge/alaskapac.org/vendor/livewire/livewire/src/DisableBrowserCache.php(19): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#57 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Livewire\\DisableBrowserCache->handle()
#58 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(21): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#59 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ConvertEmptyStringsToNull.php(31): Illuminate\\Foundation\\Http\\Middleware\\TransformsRequest->handle()
#60 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Foundation\\Http\\Middleware\\ConvertEmptyStringsToNull->handle()
#61 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(21): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#62 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TrimStrings.php(40): Illuminate\\Foundation\\Http\\Middleware\\TransformsRequest->handle()
#63 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Foundation\\Http\\Middleware\\TrimStrings->handle()
#64 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ValidatePostSize.php(27): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#65 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Foundation\\Http\\Middleware\\ValidatePostSize->handle()
#66 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/PreventRequestsDuringMaintenance.php(86): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#67 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Foundation\\Http\\Middleware\\PreventRequestsDuringMaintenance->handle()
#68 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Http/Middleware/HandleCors.php(49): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#69 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Http\\Middleware\\HandleCors->handle()
#70 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Http/Middleware/TrustProxies.php(39): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#71 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Http\\Middleware\\TrustProxies->handle()
#72 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(116): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#73 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(165): Illuminate\\Pipeline\\Pipeline->then()
#74 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(134): Illuminate\\Foundation\\Http\\Kernel->sendRequestThroughRouter()
#75 /home/forge/alaskapac.org/public/index.php(52): Illuminate\\Foundation\\Http\\Kernel->handle()
#76 {main}
@arthurperton
Getting this too. Client sent me this: https://flareapp.io/share/KPglNBZ7#F75
@edalzell Are you using multisite for this one?
@edalzell Are you using multisite for this one?
Yes
That's interesting. Can you show your sites.php
and static_caching.php
configs here please?
'sites' => [
'default' => [
'name' => 'zakat.org (EN)',
'locale' => 'en_US',
'url' => env('APP_URL'),
],
'ar' => [
'name' => 'zakat.org (AR)',
'locale' => 'ar_DZ',
'url' => env('APP_URL').'/ar/',
],
'zfi' => [
'name' => 'zfinstitute.com (ZFI)',
'locale' => 'en_US',
'url' => env('ZFI_URL'),
'attributes' => [
'disable_pre_footer' => true,
'footer_below_wrapper' => true,
],
],
],
<?php
return [
/*
|--------------------------------------------------------------------------
| Active Static Caching Strategy
|--------------------------------------------------------------------------
|
| To enable Static Caching, you should choose a strategy from the ones
| you have defined below. Leave this null to disable static caching.
|
*/
'strategy' => env('STATAMIC_STATIC_CACHING_STRATEGY', null),
/*
|--------------------------------------------------------------------------
| Caching Strategies
|--------------------------------------------------------------------------
|
| Here you may define all of the static caching strategies for your
| application as well as their drivers.
|
| Supported drivers: "application", "file"
|
*/
'strategies' => [
'half' => [
'driver' => 'application',
'expiry' => null,
],
'full' => [
'driver' => 'file',
'path' => public_path('static'),
'lock_hold_length' => 0,
],
],
/*
|--------------------------------------------------------------------------
| Exclusions
|--------------------------------------------------------------------------
|
| Here you may define a list of URLs to be excluded from static
| caching. You may want to exclude URLs containing dynamic
| elements like contact forms, or shopping carts.
|
*/
'exclude' => [
'/get-involved/ramadan-2021*',
'/orphan-update',
'/resource-center/zakat-calculator',
'/sponsor-an-orphan*',
],
/*
|--------------------------------------------------------------------------
| Invalidation Rules
|--------------------------------------------------------------------------
|
| Here you may define the rules that trigger when and how content would be
| flushed from the static cache. See the documentation for more details.
| If a custom class is not defined, the default invalidator is used.
|
| https://statamic.dev/static-caching
|
*/
'invalidation' => [
'rules' => 'all',
],
];
I am experiencing the same error and it seems we are making progress in resolving the issue.
The error occurs specifically when the multi-site feature is enabled.
However, I was able to resolve the issue by commenting out all the multi-site configurations in sites.php
, except for one.
This allowed me to get everything working properly and confirm suggestion about error in multi-site configuration.
Unfortunately, changing the configuration as suggested in the https://statamic.dev/static-caching#multisite documentation did not help to kick off multi-site
That's super interesting, the only site this appeared on for us IS multi-site.
Yep mine multisite too.
That's super interesting, the only site this appeared on for us IS multi-site.
Isn’t this exactly what zzzaaa was saying?
The error occurs specifically when the multi-site feature is enabled.
I‘m also experiencing issues with nocache on a multisite.
We are also experiencing this on a website, together with #7039
This site is NOT a multisite, but it IS using Livewire.
We were using half
measure caching, and we didn't clear the cache very often (once a day).
We are experiencing this primarily on a page with forms (with livewire). We also have another dynamic livewire page which has been excluded from all caching, this page is working fine.
@edalzell are you also using livewire?
Can confirm I was using Livewire.
Yes
Curious.
@zzzaaa and @j3ll3yfi5h were you also using Livewire?
@jasonvarga Sorry, missed your comment: I don't use Livewire.
Hey!
So I was also having this problem, similar to others we're using Livewire here, no issues on local or in dev, just once it got into live. We're also not a multisite.
It's not a fix by any means, but to stop the 500 error, I've done the following:
vendor/statamic/cms/src/StaticCaching/NoCache/Session.php - Line 49:
return $this->regions[$key] ?? '';
vendor/statamic/cms/src/StaticCaching/Replacers/NoCacheReplacer.php - Line 59:
private function performReplacement(string $content)
{
return preg_replace_callback(self::PATTERN, function ($matches) {
if (! $region = $matches[1] ?? null) {
return '';
}
$tempRegion = $this->session->region($region);
if(empty($tempRegion)) {
return '';
}
return $tempRegion->render();
}, $content);
}
At a guess, I imagine it's just leaving some parts of the site missing (or re-generating it?), but for us it's better than the 500 error.
Hopefully that's useful to someone in the meantime - let me know if you'd like me to send over any logs etc!
Okay, so on from the above, I’ve found it does just return empty - so I’ve updated the code to behave slightly differently, that from what I can tell, seems to be okay.
vendor/statamic/cms/src/StaticCaching/Replacers/NoCacheReplacer.php - Line 59:
private function performReplacement(string $content)
{
return preg_replace_callback(self::PATTERN, function ($matches) use ($content) {
if (! $region = $matches[1] ?? null) {
return ‘’;
}
$tempRegion = $this->session->region($region);
if(empty($tempRegion)) {
return ‘<span>MISSING NOCACHE</span>‘;
}
return $tempRegion->render();
}, $content);
}
vendor/statamic/cms/src/StaticCaching/Middleware/Cache.php - Line 43:
public function handle($request, Closure $next)
{
$lock = $this->createLock($request);
while (! $lock->acquire()) {
sleep(1);
}
$isValid = true;
if ($this->canBeCached($request) && $this->cacher->hasCachedPage($request)) {
$response = response($this->cacher->getCachedPage($request));
$this->makeReplacements($response);
if(strpos($response->content(), ‘<span>MISSING NOCACHE</span>‘) > -1) {
$isValid = false;
}
if($isValid) { // If not, continue, cache again??
return $response;
}
}
$response = $next($request);
if ($this->shouldBeCached($request, $response) || !$isValid) {
$lock->acquire(true);
$this->makeReplacementsAndCacheResponse($request, $response);
$this->nocache->write();
} elseif (! $response->isRedirect()) {
$this->makeReplacements($response);
}
return $response;
}
Obviously, this is by no means a fix, but it does seem to be working okay for us at the minute as a workaround.
We see this issue on a increasing number of our statamic instances. We introduced more nocache Tags with the latest release. We are on half mesure, NO multisite, NO livewire, Redis for cache
We have also reported this months ago to support@statamic.com, but they did not reply so far. I really hope this is something that is going to be addressed as it makes the cache pretty unusable on some instances.
I'm a little frightened because now we use nocache around everything related to posts. This affects nearly every instance we were running.
i reproduced it locally with file cache driver, no multisite, no livewire.
I didn't figure out how this situation could arise, in my production case perhaps due to redis key expiry. But there must be many ways as seen in this ticket here.
I think cache should always be self-healing and should not depend on existence of specific files or cache keys.
Would you be able to try out #8449 using a Composer patch?
This solution will end up serving the uncached page if it runs into that error. (Similar to @Alt-Ben 's revision) It doesn't solve the reason the region is missing from the cache to begin with.
Sounds like @mmodler is onto something with maybe Redis auto-expiring, but @edalzell's original support:details output shows he is using the regular statamic/file cache driver. So, maybe it's a race condition somehow? No idea really.
maybe it's a race condition somehow?
The site this was happening on the most is our busiest site, and we don't pre-generate the static cache, so a race condition is a definite possibility.
I don't think its related to redis "forgetting" keys. We have default TTL of two month and if redis would forget keys randomly, we will see lots of different cache errors.
Here some examples from yesterdays log, the memory errors point to a race condition?
@edalzell
Do you see the same on your busy instance with file cache driver?
I tried to use the composer patch which failed :) Digging deeper i found that there was a race fix with statamic 3.4 (https://github.com/statamic/cms/pull/7722). Our platform is still on v3.3.68, so i think what we see is the locking issue?
So just as a side note - we've tried all of the various caching methods in the live environment, from what I remember, the file cache didn't have this issue, but it was really really slow, like it wasn't actually doing the caching in the first place. When I swapped it to database (at the time), it returned to the speeds we'd expect, but then started getting the array key issue.
We're also on 4.11.0 now, I've also just applied the patch, it normally takes a little while for the error to kick in, so I'll let you know whether we start to see it again over the next couple of days or not!
I will try 3.4.11 + composer patch now.
Just as an update, I haven't seen the issue return with the patch - so looks like it mitigates it for now?
I will try 3.4.11 + composer patch now.
We don't see the exception in the logs since we patched.
was very happy to see no more cache errors in the log files for the last few days.
Today a colleague pointed out to me that our Redis instances are suddenly using 5gb of memory instead of 500mb. Since we only have a portion of the instances live already, this is a scary increase.
I already have the following findings:
=> The fix floods Redis for me, with file cache driver probably thousands and thousands of files would be created.
@Alt-Ben Can you please check if you get flooded with response / nocache cache entries, too?
@mmodler, so we're using memcached as the cache driver currently, it is basically maxing out the server swap memory (~2GB), but I don't think I've ever seen it not doing that whilst we've had memcached on. We also don't have anything that monitors it properly to be honest, so I couldn't be sure. But we've also got other issues around the volume of assets we have on this particular site that might distort any resource usage data anyway.
After much degugging it turned out the problem had nothing to do with nocache. I was just too fixated on the nocache entries :)
On one instance there were heaps of old urls with query parameters called, probably from a search engine bot. This led to the mass of response keys, the resulting nocache keys were then "expected behavior".
As a quick fix I set ignore_query_strings
globally to true, and change the value in my service provider to false if a query parameter from my whitelist is included in the url.
So @mmodler are you saying the fix is okay? Your memory issue was unrelated to it?
@jasonvarga Atm its ok.
Bug description
We see hundreds of these entries in our logs, any ideas what the cause could be, or how to debug?
How to reproduce
Take a look at the logs on one of our sites (can provide if you want)
Logs
Environment