edalzell commented 2 years ago

Bug description

We see hundreds of these entries in our logs, any ideas what the cause could be, or how to debug?

How to reproduce

Take a look at the logs on one of our sites (can provide if you want)

Logs

[2022-09-24 10:55:15] production.ERROR: Undefined array key "b78dd0dbcef234639b15d6854a7bfa39980cda7a" {"exception":"[object] (ErrorException(code: 0): Undefined array key \"b78dd0dbcef234639b15d6854a7bfa39980cda7a\" at /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Collections/Collection.php:1679)
[stacktrace]
#0 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php(259): Illuminate\\Foundation\\Bootstrap\\HandleExceptions->handleError()
#1 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Collections/Collection.php(1679): Illuminate\\Foundation\\Bootstrap\\HandleExceptions->Illuminate\\Foundation\\Bootstrap\\{closure}()
#2 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/NoCache/Session.php(49): Illuminate\\Support\\Collection->offsetGet()
#3 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/Replacers/NoCacheReplacer.php(66): Statamic\\StaticCaching\\NoCache\\Session->region()
#4 [internal function]: Statamic\\StaticCaching\\Replacers\\NoCacheReplacer->Statamic\\StaticCaching\\Replacers\\{closure}()
#5 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/Replacers/NoCacheReplacer.php(61): preg_replace_callback()
#6 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/Replacers/NoCacheReplacer.php(53): Statamic\\StaticCaching\\Replacers\\NoCacheReplacer->performReplacement()
#7 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/Replacers/NoCacheReplacer.php(47): Statamic\\StaticCaching\\Replacers\\NoCacheReplacer->replace()
#8 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/Replacers/NoCacheReplacer.php(32): Statamic\\StaticCaching\\Replacers\\NoCacheReplacer->replaceInResponse()
#9 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/Middleware/Cache.php(42): Statamic\\StaticCaching\\Replacers\\NoCacheReplacer->replaceInCachedResponse()
#10 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Collections/Traits/EnumeratesValues.php(262): Statamic\\StaticCaching\\Middleware\\Cache->Statamic\\StaticCaching\\Middleware\\{closure}()
#11 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/Middleware/Cache.php(42): Illuminate\\Support\\Collection->each()
#12 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Statamic\\StaticCaching\\Middleware\\Cache->handle()

Environment

Environment
Application Name: AlaskaPAC
Laravel Version: 9.30.1
PHP Version: 8.1.6
Composer Version: 2.4.2
Environment: production
Debug Mode: OFF
URL: alaskapac.org
Maintenance Mode: OFF

Cache
Config: NOT CACHED
Events: NOT CACHED
Routes: NOT CACHED
Views: CACHED

Drivers
Broadcasting: log
Cache: statamic
Database: mysql
Logs: stack / daily, front
Mail: smtp
Queue: sync
Session: file

Statamic
Addons: 18
Antlers: runtime
Version: 3.3.36 PRO

Statamic Addons
aryehraber/statamic-captcha: 1.8.0
aryehraber/statamic-uuid: 2.1.0
doublethreedigital/duplicator: 2.3.2
edalzell/forma: 1.2
jacksleight/statamic-bard-mutator: 1.1.2
jacksleight/statamic-bard-texstyle: 1.0.3
jonassiewertsen/statamic-external-link: 1.6.0
jonassiewertsen/statamic-livewire: 2.9.0
statamic/seo-pro: 3.1.0
swiftmade/statamic-clear-assets: 1.1.0
transformstudios/events: 4.0.6
transformstudios/front: 1.10
transformstudios/gated: dev-main
transformstudios/github: dev-main
transformstudios/magic-link: dev-main
transformstudios/review: 3.5.0
transformstudios/simple: 2.6.0
transformstudios/takeover-redirects: dev-main



### Installation

Fresh statamic/statamic site via CLI

### Antlers Parser

runtime (new)

### Additional details

_No response_

edalzell commented 2 years ago

It's always the same "key"

arthurperton commented 2 years ago

It looks like this could happen when you use full measure caching: when you clear the regular cache, but not the static cache. Then the page (with the replacement placeholders) would still be cached in a file, but the actual replacements per region would be lost, as they are stored in the regular cache. That would lead to the error in this issue, where the Session doesn't have the requested region.

arthurperton commented 2 years ago

@edalzell are you using full measure caching on this site?

edalzell commented 2 years ago

@edalzell are you using full measure caching on this site?

No

jasonvarga commented 2 years ago

Cloudflare?

edalzell commented 2 years ago

Cloudflare?

Yes

arthurperton commented 2 years ago

What kind of static caching are you using, if at all?

edalzell commented 2 years ago

What kind of static caching are you using, if at all?

1/2 measure.

jasonvarga commented 2 years ago

@edalzell do you get this issue locally?

edalzell commented 2 years ago

@edalzell do you get this issue locally?

Nope, but I normally don't do static caching locally. But it is enable on this site and I don't see it in my logs. But the live site gets waaaaaaay more traffic of course.

j3ll3yfi5h commented 1 year ago

Are there any workarounds (apart disabling static caching) yet?

arthurperton commented 1 year ago

Are there any workarounds (apart disabling static caching) yet?

Currently no. So you are having the same issue? Do you have any additional info that might help?

jasonvarga commented 1 year ago

7039 is likely very related.

arthurperton commented 1 year ago

This is still a mystery. Can somebody share a full log file for this issue? Maybe there are more, possibly related errors in it to go on.

edalzell commented 1 year ago

[2022-12-08 12:00:39] production.ERROR: Undefined array key "1b042a2794ea6ce863dab6f73d6abc456e010963" {"exception":"[object] (ErrorException(code: 0): Undefined array key \"1b042a2794ea6ce863dab6f73d6abc456e010963\" at /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Collections/Collection.php:1679)
[stacktrace]
#0 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php(259): Illuminate\\Foundation\\Bootstrap\\HandleExceptions->handleError()
#1 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Collections/Collection.php(1679): Illuminate\\Foundation\\Bootstrap\\HandleExceptions->Illuminate\\Foundation\\Bootstrap\\{closure}()
#2 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/NoCache/Session.php(49): Illuminate\\Support\\Collection->offsetGet()
#3 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/Replacers/NoCacheReplacer.php(66): Statamic\\StaticCaching\\NoCache\\Session->region()
#4 [internal function]: Statamic\\StaticCaching\\Replacers\\NoCacheReplacer->Statamic\\StaticCaching\\Replacers\\{closure}()
#5 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/Replacers/NoCacheReplacer.php(61): preg_replace_callback()
#6 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/Replacers/NoCacheReplacer.php(53): Statamic\\StaticCaching\\Replacers\\NoCacheReplacer->performReplacement()
#7 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/Replacers/NoCacheReplacer.php(47): Statamic\\StaticCaching\\Replacers\\NoCacheReplacer->replace()
#8 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/Replacers/NoCacheReplacer.php(32): Statamic\\StaticCaching\\Replacers\\NoCacheReplacer->replaceInResponse()
#9 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/Middleware/Cache.php(71): Statamic\\StaticCaching\\Replacers\\NoCacheReplacer->replaceInCachedResponse()
#10 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Collections/Traits/EnumeratesValues.php(262): Statamic\\StaticCaching\\Middleware\\Cache->Statamic\\StaticCaching\\Middleware\\{closure}()
#11 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/Middleware/Cache.php(71): Illuminate\\Support\\Collection->each()
#12 /home/forge/alaskapac.org/vendor/statamic/cms/src/StaticCaching/Middleware/Cache.php(42): Statamic\\StaticCaching\\Middleware\\Cache->makeReplacements()
#13 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Statamic\\StaticCaching\\Middleware\\Cache->handle()
#14 /home/forge/alaskapac.org/vendor/statamic/cms/src/Http/Middleware/AuthGuard.php(14): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#15 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Statamic\\Http\\Middleware\\AuthGuard->handle()
#16 /home/forge/alaskapac.org/vendor/statamic/cms/src/Http/Middleware/AddViewPaths.php(20): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#17 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Statamic\\Http\\Middleware\\AddViewPaths->handle()
#18 /home/forge/alaskapac.org/vendor/statamic/cms/src/Http/Middleware/Localize.php(37): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#19 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Statamic\\Http\\Middleware\\Localize->handle()
#20 /home/forge/alaskapac.org/vendor/statamic/cms/src/Http/Middleware/HandleToken.php(13): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#21 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Statamic\\Http\\Middleware\\HandleToken->handle()
#22 /home/forge/alaskapac.org/vendor/statamic/cms/src/Http/Middleware/StacheLock.php(29): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#23 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Statamic\\Http\\Middleware\\StacheLock->handle()
#24 /home/forge/alaskapac.org/vendor/transformstudios/takeover-redirects/src/Http/Middleware/RedirectIfScheduled.php(17): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#25 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): TransformStudios\\TakeoverRedirects\\Http\\Middleware\\RedirectIfScheduled->handle()
#26 /home/forge/alaskapac.org/vendor/transformstudios/gated/src/Http/Middleware/HandleGate.php(18): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#27 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): TransformStudios\\Gated\\Http\\Middleware\\HandleGate->handle()
#28 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Routing/Middleware/SubstituteBindings.php(50): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#29 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Routing\\Middleware\\SubstituteBindings->handle()
#30 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/VerifyCsrfToken.php(78): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#31 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Foundation\\Http\\Middleware\\VerifyCsrfToken->handle()
#32 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/View/Middleware/ShareErrorsFromSession.php(49): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#33 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\View\\Middleware\\ShareErrorsFromSession->handle()
#34 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(121): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#35 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Session/Middleware/StartSession.php(64): Illuminate\\Session\\Middleware\\StartSession->handleStatefulRequest()
#36 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Session\\Middleware\\StartSession->handle()
#37 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/AddQueuedCookiesToResponse.php(37): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#38 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Cookie\\Middleware\\AddQueuedCookiesToResponse->handle()
#39 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Cookie/Middleware/EncryptCookies.php(67): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#40 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Cookie\\Middleware\\EncryptCookies->handle()
#41 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(116): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#42 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Routing/Router.php(728): Illuminate\\Pipeline\\Pipeline->then()
#43 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Routing/Router.php(705): Illuminate\\Routing\\Router->runRouteWithinStack()
#44 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Routing/Router.php(669): Illuminate\\Routing\\Router->runRoute()
#45 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Routing/Router.php(658): Illuminate\\Routing\\Router->dispatchToRoute()
#46 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(190): Illuminate\\Routing\\Router->dispatch()
#47 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(141): Illuminate\\Foundation\\Http\\Kernel->Illuminate\\Foundation\\Http\\{closure}()
#48 /home/forge/alaskapac.org/vendor/statamic/cms/src/Http/Middleware/DisableFloc.php(18): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#49 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Statamic\\Http\\Middleware\\DisableFloc->handle()
#50 /home/forge/alaskapac.org/vendor/statamic/cms/src/Http/Middleware/CheckMultisite.php(14): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#51 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Statamic\\Http\\Middleware\\CheckMultisite->handle()
#52 /home/forge/alaskapac.org/vendor/statamic/cms/src/Http/Middleware/CheckComposerJsonScripts.php(14): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#53 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Statamic\\Http\\Middleware\\CheckComposerJsonScripts->handle()
#54 /home/forge/alaskapac.org/vendor/statamic/cms/src/Http/Middleware/PoweredByHeader.php(19): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#55 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Statamic\\Http\\Middleware\\PoweredByHeader->handle()
#56 /home/forge/alaskapac.org/vendor/livewire/livewire/src/DisableBrowserCache.php(19): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#57 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Livewire\\DisableBrowserCache->handle()
#58 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(21): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#59 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ConvertEmptyStringsToNull.php(31): Illuminate\\Foundation\\Http\\Middleware\\TransformsRequest->handle()
#60 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Foundation\\Http\\Middleware\\ConvertEmptyStringsToNull->handle()
#61 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TransformsRequest.php(21): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#62 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/TrimStrings.php(40): Illuminate\\Foundation\\Http\\Middleware\\TransformsRequest->handle()
#63 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Foundation\\Http\\Middleware\\TrimStrings->handle()
#64 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/ValidatePostSize.php(27): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#65 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Foundation\\Http\\Middleware\\ValidatePostSize->handle()
#66 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Foundation/Http/Middleware/PreventRequestsDuringMaintenance.php(86): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#67 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Foundation\\Http\\Middleware\\PreventRequestsDuringMaintenance->handle()
#68 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Http/Middleware/HandleCors.php(49): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#69 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Http\\Middleware\\HandleCors->handle()
#70 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Http/Middleware/TrustProxies.php(39): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#71 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(180): Illuminate\\Http\\Middleware\\TrustProxies->handle()
#72 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(116): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#73 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(165): Illuminate\\Pipeline\\Pipeline->then()
#74 /home/forge/alaskapac.org/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(134): Illuminate\\Foundation\\Http\\Kernel->sendRequestThroughRouter()
#75 /home/forge/alaskapac.org/public/index.php(52): Illuminate\\Foundation\\Http\\Kernel->handle()
#76 {main}

@arthurperton

johncarter- commented 1 year ago

Getting this too. Client sent me this: https://flareapp.io/share/KPglNBZ7#F75

arthurperton commented 1 year ago

@edalzell Are you using multisite for this one?

edalzell commented 1 year ago

@edalzell Are you using multisite for this one?

Yes

arthurperton commented 1 year ago

That's interesting. Can you show your sites.php and static_caching.php configs here please?

edalzell commented 1 year ago

    'sites' => [

        'default' => [
            'name' => 'zakat.org (EN)',
            'locale' => 'en_US',
            'url' => env('APP_URL'),
        ],

        'ar' => [
            'name' => 'zakat.org (AR)',
            'locale' => 'ar_DZ',
            'url' => env('APP_URL').'/ar/',
        ],

        'zfi' => [
            'name' => 'zfinstitute.com (ZFI)',
            'locale' => 'en_US',
            'url' => env('ZFI_URL'),
            'attributes' => [
                'disable_pre_footer' => true,
                'footer_below_wrapper' => true,
            ],
        ],

    ],

<?php

return [

    /*
    |--------------------------------------------------------------------------
    | Active Static Caching Strategy
    |--------------------------------------------------------------------------
    |
    | To enable Static Caching, you should choose a strategy from the ones
    | you have defined below. Leave this null to disable static caching.
    |
    */

    'strategy' => env('STATAMIC_STATIC_CACHING_STRATEGY', null),

    /*
    |--------------------------------------------------------------------------
    | Caching Strategies
    |--------------------------------------------------------------------------
    |
    | Here you may define all of the static caching strategies for your
    | application as well as their drivers.
    |
    | Supported drivers: "application", "file"
    |
    */

    'strategies' => [

        'half' => [
            'driver' => 'application',
            'expiry' => null,
        ],

        'full' => [
            'driver' => 'file',
            'path' => public_path('static'),
            'lock_hold_length' => 0,
        ],

    ],

    /*
    |--------------------------------------------------------------------------
    | Exclusions
    |--------------------------------------------------------------------------
    |
    | Here you may define a list of URLs to be excluded from static
    | caching. You may want to exclude URLs containing dynamic
    | elements like contact forms, or shopping carts.
    |
    */

    'exclude' => [
        '/get-involved/ramadan-2021*',
        '/orphan-update',
        '/resource-center/zakat-calculator',
        '/sponsor-an-orphan*',
    ],

    /*
    |--------------------------------------------------------------------------
    | Invalidation Rules
    |--------------------------------------------------------------------------
    |
    | Here you may define the rules that trigger when and how content would be
    | flushed from the static cache. See the documentation for more details.
    | If a custom class is not defined, the default invalidator is used.
    |
    | https://statamic.dev/static-caching
    |
    */

    'invalidation' => [
        'rules' => 'all',
    ],
];

zzzaaa commented 1 year ago

I am experiencing the same error and it seems we are making progress in resolving the issue. The error occurs specifically when the multi-site feature is enabled. However, I was able to resolve the issue by commenting out all the multi-site configurations in sites.php, except for one. This allowed me to get everything working properly and confirm suggestion about error in multi-site configuration.

Unfortunately, changing the configuration as suggested in the https://statamic.dev/static-caching#multisite documentation did not help to kick off multi-site

edalzell commented 1 year ago

That's super interesting, the only site this appeared on for us IS multi-site.

johncarter- commented 1 year ago

Yep mine multisite too.

marijoo commented 1 year ago

That's super interesting, the only site this appeared on for us IS multi-site.

Isn’t this exactly what zzzaaa was saying?

The error occurs specifically when the multi-site feature is enabled.

I‘m also experiencing issues with nocache on a multisite.

jeroenimpres commented 1 year ago

We are also experiencing this on a website, together with #7039 This site is NOT a multisite, but it IS using Livewire. We were using half measure caching, and we didn't clear the cache very often (once a day).

We are experiencing this primarily on a page with forms (with livewire). We also have another dynamic livewire page which has been excluded from all caching, this page is working fine.

@edalzell are you also using livewire?

johncarter- commented 1 year ago

Can confirm I was using Livewire.

edalzell commented 1 year ago

Yes

jasonvarga commented 1 year ago

Curious.

@zzzaaa and @j3ll3yfi5h were you also using Livewire?

j3ll3yfi5h commented 1 year ago

@jasonvarga Sorry, missed your comment: I don't use Livewire.

Alt-Ben commented 1 year ago

Hey!

So I was also having this problem, similar to others we're using Livewire here, no issues on local or in dev, just once it got into live. We're also not a multisite.

It's not a fix by any means, but to stop the 500 error, I've done the following:

vendor/statamic/cms/src/StaticCaching/NoCache/Session.php - Line 49: return $this->regions[$key] ?? '';

vendor/statamic/cms/src/StaticCaching/Replacers/NoCacheReplacer.php - Line 59:

private function performReplacement(string $content)
{
    return preg_replace_callback(self::PATTERN, function ($matches) {
        if (! $region = $matches[1] ?? null) {
            return '';
        }

        $tempRegion = $this->session->region($region);

        if(empty($tempRegion)) {
            return '';
        }

        return $tempRegion->render();
    }, $content);
}

At a guess, I imagine it's just leaving some parts of the site missing (or re-generating it?), but for us it's better than the 500 error.

Hopefully that's useful to someone in the meantime - let me know if you'd like me to send over any logs etc!

Alt-Ben commented 1 year ago

Okay, so on from the above, I’ve found it does just return empty - so I’ve updated the code to behave slightly differently, that from what I can tell, seems to be okay.

vendor/statamic/cms/src/StaticCaching/Replacers/NoCacheReplacer.php - Line 59:

private function performReplacement(string $content)
{
    return preg_replace_callback(self::PATTERN, function ($matches) use ($content) {
        if (! $region = $matches[1] ?? null) {
            return ‘’;
        }
        $tempRegion = $this->session->region($region);
        if(empty($tempRegion)) {
            return ‘<span>MISSING NOCACHE</span>‘;
        }
        return $tempRegion->render();
    }, $content);
}

vendor/statamic/cms/src/StaticCaching/Middleware/Cache.php - Line 43:

public function handle($request, Closure $next)
{
    $lock = $this->createLock($request);
    while (! $lock->acquire()) {
        sleep(1);
    }
    $isValid = true;
    if ($this->canBeCached($request) && $this->cacher->hasCachedPage($request)) {
        $response = response($this->cacher->getCachedPage($request));
        $this->makeReplacements($response);
        if(strpos($response->content(), ‘<span>MISSING NOCACHE</span>‘) > -1) {
            $isValid = false;
        }
        if($isValid) { // If not, continue, cache again??
            return $response;
        }
    }
    $response = $next($request);
    if ($this->shouldBeCached($request, $response) || !$isValid) {
        $lock->acquire(true);
        $this->makeReplacementsAndCacheResponse($request, $response);
        $this->nocache->write();
    } elseif (! $response->isRedirect()) {
        $this->makeReplacements($response);
    }
    return $response;
}

Obviously, this is by no means a fix, but it does seem to be working okay for us at the minute as a workaround.

mmodler commented 1 year ago

We see this issue on a increasing number of our statamic instances. We introduced more nocache Tags with the latest release. We are on half mesure, NO multisite, NO livewire, Redis for cache

jeroenimpres commented 1 year ago

We have also reported this months ago to support@statamic.com, but they did not reply so far. I really hope this is something that is going to be addressed as it makes the cache pretty unusable on some instances.

mmodler commented 1 year ago

I'm a little frightened because now we use nocache around everything related to posts. This affects nearly every instance we were running.

mmodler commented 1 year ago

i reproduced it locally with file cache driver, no multisite, no livewire.

Add a no_cache tag to a page
Load the page with half measure
Look which file under /storage/framework/cache/data which file contains your cached page. I searched for "<span class="nocache" data-nocache="
Delete all files in /storage/framework/cache/data except the half-measure-cachefile
Reload the page => Undefined array key "62ada2d29f20296fb463ea3838a77665378034cf"

I didn't figure out how this situation could arise, in my production case perhaps due to redis key expiry. But there must be many ways as seen in this ticket here.

I think cache should always be self-healing and should not depend on existence of specific files or cache keys.

jasonvarga commented 1 year ago

Would you be able to try out #8449 using a Composer patch?

This solution will end up serving the uncached page if it runs into that error. (Similar to @Alt-Ben 's revision) It doesn't solve the reason the region is missing from the cache to begin with.

Sounds like @mmodler is onto something with maybe Redis auto-expiring, but @edalzell's original support:details output shows he is using the regular statamic/file cache driver. So, maybe it's a race condition somehow? No idea really.

edalzell commented 1 year ago

maybe it's a race condition somehow?

The site this was happening on the most is our busiest site, and we don't pre-generate the static cache, so a race condition is a definite possibility.

mmodler commented 1 year ago

I don't think its related to redis "forgetting" keys. We have default TTL of two month and if redis would forget keys randomly, we will see lots of different cache errors.

Here some examples from yesterdays log, the memory errors point to a race condition?

grafik

mmodler commented 1 year ago

@edalzell

Do you see the same on your busy instance with file cache driver?

mmodler commented 1 year ago

I tried to use the composer patch which failed :) Digging deeper i found that there was a race fix with statamic 3.4 (https://github.com/statamic/cms/pull/7722). Our platform is still on v3.3.68, so i think what we see is the locking issue?

Alt-Ben commented 1 year ago

So just as a side note - we've tried all of the various caching methods in the live environment, from what I remember, the file cache didn't have this issue, but it was really really slow, like it wasn't actually doing the caching in the first place. When I swapped it to database (at the time), it returned to the speeds we'd expect, but then started getting the array key issue.

We're also on 4.11.0 now, I've also just applied the patch, it normally takes a little while for the error to kick in, so I'll let you know whether we start to see it again over the next couple of days or not!

mmodler commented 1 year ago

I will try 3.4.11 + composer patch now.

Alt-Ben commented 1 year ago

Just as an update, I haven't seen the issue return with the patch - so looks like it mitigates it for now?

mmodler commented 1 year ago

I will try 3.4.11 + composer patch now.

We don't see the exception in the logs since we patched.

mmodler commented 1 year ago

was very happy to see no more cache errors in the log files for the last few days.

Today a colleague pointed out to me that our Redis instances are suddenly using 5gb of memory instead of 500mb. Since we only have a portion of the instances live already, this is a scary increase.

grafik

I already have the following findings:

The instances with the high number of cache keys are live, but have less than 100 pages and only local, moderate traffic.
The pages have one or more nocache tags on the home page and/or the news page.
The pages are hardly updated, I guess by an editor about once a day and then by expiring deadlines in the midnight.
I have compared two responses keys of the start page, the some difference is the data-nocache ID

grafik

At first glance at the nocache keys the HTML is NOT different, but apparently the rest of the serialized data is. I'm not sure why so much extra data is serialized here, and not just the HTML. I guess I will unserialize the two keys and find out the difference.

=> The fix floods Redis for me, with file cache driver probably thousands and thousands of files would be created.

mmodler commented 1 year ago

@Alt-Ben Can you please check if you get flooded with response / nocache cache entries, too?

Alt-Ben commented 1 year ago

@mmodler, so we're using memcached as the cache driver currently, it is basically maxing out the server swap memory (~2GB), but I don't think I've ever seen it not doing that whilst we've had memcached on. We also don't have anything that monitors it properly to be honest, so I couldn't be sure. But we've also got other issues around the volume of assets we have on this particular site that might distort any resource usage data anyway.

mmodler commented 1 year ago

After much degugging it turned out the problem had nothing to do with nocache. I was just too fixated on the nocache entries :)

On one instance there were heaps of old urls with query parameters called, probably from a search engine bot. This led to the mass of response keys, the resulting nocache keys were then "expected behavior".

As a quick fix I set ignore_query_strings globally to true, and change the value in my service provider to false if a query parameter from my whitelist is included in the url.

jasonvarga commented 1 year ago

So @mmodler are you saying the fix is okay? Your memory issue was unrelated to it?

mmodler commented 1 year ago

@jasonvarga Atm its ok.

statamic / cms

Static Cache/`nocache` tag error #6781

Bug description

How to reproduce

Logs

Environment

7039 is likely very related.