Static Caching invalidator not working when running from the queue

[jameswtc]

Bug Description

The static cache file not removed when the queue is run through php artisan queue:work, (or using database queue).

After hours of inspection, the is due to the caching of full URL in the cache:

the problem is in /cms/src/StaticCaching/Cachers/FileCacher.php line 36

$this->getUrl($request)

in /cms/src/StaticCaching/StaticCacheManager.php line 49

'base_url' => $this->app['request']->root(),

The request object is not available when running the command from Laravel artisan, and the root url resolved to localhost and the scheme is set to http instead of https.

This caused the $this->getUrl($request) not able to get the cached version of the url with the actual host.

How to Reproduce

In the .env file, set

CACHE_STRATEGY=full
...
QUEUE_CONNECTION=database

Setup the queue worker to process the queue.

Modify a collection entry and save it.

Inspect the static folder, the cached file still presents, i.e., not removed by invalidator.

Environment

Statamic version: 3.0.39

PHP version: 7.4

FIX:

I managed to make it work by reading the base_url from the .env file instead of resolving it from the request object.

      return array_merge($config, [
          'exclude' => $this->app['config']['statamic.static_caching.exclude'] ?? [],
          'ignore_query_strings' => $this->app['config']['statamic.static_caching.ignore_query_strings'] ?? false,
          'base_url' => env('BASE_URL'), // $this->app['request']->root(),
          'locale' => Site::current()->handle(),
      ]);

[edalzell]

We are running half-measure static caching and running into something similar. We are using the queue (Redis) cuz we have the git integration turned on.

[edalzell]

I tried 'base_url' => URL::getSiteUrl(),, but that did not fix this issue, for me.

[jameswtc]

I tried 'base_url' => URL::getSiteUrl(),, but that did not fix this issue, for me.

For this to work, the URL must also be cached in your chosen caching driver too. If for some reason you clear your cache, then you have to manually remove it too.

The StaticCacheManager first check for the URL in the cache to retrieve the file path.

[edalzell]

I tried 'base_url' => URL::getSiteUrl(),, but that did not fix this issue, for me.

For this to work, the URL must also be cached in your chosen caching driver too. If for some reason you clear your cache, then you have to manually remove it too.

The StaticCacheManager first check for the URL in the cache to retrieve the file path.

@jameswtc I don't quite understand this, can you explain what you mean? I'd like to make a PR, as this still affects us on every site we deploy.

[andjsch]

I dug a bit into the problem, why URLs, that should be invalidated with the entry (listed inside config/statamic/static_caching.php) won't be invalidated.

I could see that the URLs that have been saved as a static file, aren't cached [with the cacher]. This is why they won't be invalidated, as the invalidateUrl() function exits early. I adjusted the function to look like this to make it work temporarily:

[andjsch]

Background info: I don't know why, but invalidation on this specific site has worked for a very long time with the database queue connection enabled. I think it has stopped working as soon as they started to translate some content.

I would be curious if the above fix that worked for me, works for one of you as well. Because if it does, I am happy to make a PR out of it.

[jakubjo]

Thanks @edalzell for your workaround you described in #3305!

[edalzell]

Thank @edalzell for your workaround you described in #3305!

You're welcome

[jakubjo]

I've fixed this by adding base_url to config/statamic/static_caching.php:

    /*
    |--------------------------------------------------------------------------
    | Caching Strategies
    |--------------------------------------------------------------------------
    |
    | Here you may define all of the static caching strategies for your
    | application as well as their drivers.
    |
    | Supported drivers: "application", "file"
    |
    */

    'strategies' => [
        'half' => [
            'driver' => 'application',
            'expiry' => null,
            'base_url' => config('app.url'), // <-- added this line
        ],

        'full' => [
            'driver' => 'file',
            'path' => public_path('static'),
            'lock_hold_length' => 0,
            'base_url' => config('app.url'), // <-- added this line
        ],
    ],

Everything seems fine. I'll watch the issue and report here if anything will not work as expected.

[edalzell]

I've fixed this by adding base_url to config/statamic/static_caching.php:

OH nice find, thanks!

[andrew-ireland]

adding base_url to config/statamic/static_caching.php: didn't fix my issues with static cache invalidaton, when using queues — anyone else had any luck?

Using QUEUE_CONNECTION=redis the static cache will not invalidate, even with this in place. If I use the standard QUEUE_CONNECTION=sync though, the static cache is cleared, and the invalidation rules are respected.

[dniccum]

We have experienced this item on our website as well. The work around proposed by @andrew-ireland does seem to fix the issue.

[K3CK]

Seems that the provided fix doesn't work for me, when using the half cache and QUEUE_CONNECTION=redis. The current entry, that is saved, is not invalidated. This is especially a problem when using it together with the "Protecting content" feature. An entry might still be available when it is fetched from the cache.

[jasonvarga]

@K3CK please provide your static_caching.php and sites.php config files.

[K3CK]

Here you go: config_files.zip

I'm also wondering how clearing the cache should work, when using an async entry for QUEUE_CONNECTION like "redis". Should it clear the entry immediately or is put on the queue to be cleared by a worker?

[edalzell]

Here you go: config_files.zip

I'm also wondering how clearing the cache should work, when using an async entry for QUEUE_CONNECTION like "redis". Should it clear the entry immediately or is put on the queue to be cleared by a worker?

use env('APP_URL') in your sites and have that set to the full and proper domain.

[AndrewHaine]

Hi all,

I'm still experiencing an issue here, the same as above, invalidation works when using the sync driver but not when invalidating URLs from a queue job. I've done a bit of debugging and found that the $this->getUrls($domain) call in FileCacher.php is returning an empty collection, indicating that the URLs don't exist in the cache which Statamic has generated (even though I've confirmed that the files do exist).

I've also attempted setting the static_caching.strategies.full.base_url config option to my app's URL.

If it helps, I'm using Horizon to monitor the queue and the redis cache driver, here are my support details:

Statamic 3.4.1 Pro Laravel 8.83.27 PHP 8.1.15 Stache Watcher Enabled Static Caching full withcandour/aardvark-seo 2.0.30 withcandour/statamic-anonymous-uploads 0.0.3 withcandour/statamic-blog-helpers 0.1.3 withcandour/statamic-imgix 0.1.5 withcandour/statamic-webpack 0.2.0

[duncanmcclean]

We're running into this issue as well. I can reproduce locally using both the sync and redis queue drivers. I've tried setting my site URL to config('app.url') which hasn't worked for me.

(I've submitted a support request with a video showing it not working)

[andrew-ireland]

We're still observing this issue with the most recent version of Statamic

If we set QUEUE_CONNECTION=sync our invalidation rules work as expected, but if QUEUE_CONNECTION=redis is set I can see the invalidation job completes in Laravel Horizon, but no changes update on the frontend (existing static cache remains in place).

I can leave CACHE_DRIVER=redis without any negative effect to invalidation.

This is on our production server managed by Laravel Forge. This is not a multisite config, though I've seen the same issue noted elsewhere for users with that approach in place. Happy to provide further details as requested — just ask.

[freshface]

Same for me, does not work.

[dominikfoeger]

I have the same problem and even though I know that's just a workaround, maybe it helps someone who is not thinking about this and needs a quickfix: You don't have to set QUEUE_CONNECTION=redis globally if all you want to do is queue some backend jobs. I changed QUEUE_CONNECTION=redis back to sync and inside my job (for async uploading videos to a streaming service):

public function __construct(String $asset_id)
    {
        $this->onConnection('redis');
    }

So the cache invalidation will be sync, but my backend job will use redis. Hope it helps someone until this is fixed. ✌️

[freshface]

Wondering if there was any solution for this?

[morphsteve]

Another workaround that may be of use to some. We needed an async queue for the Git integration and are using the 'half' strategy.

With thanks to @duncanmcclean and his post at https://steadfastcollective.com/articles/handling-statamic-static-cache-invalidation-on-large-sites.

Create a new event listener for the EntrySaved event:

<?php

namespace App\Listeners;

use Statamic\Entries\Entry;
use Statamic\Events\EntrySaved;
use Statamic\StaticCaching\Cacher;
use Statamic\StaticCaching\Cachers\ApplicationCacher;

class ForceEntryInvalidation
{
    /**
     * @param ApplicationCacher $cacher
     */
    public function __construct(
        private Cacher $cacher,
    ) {
    }

    public function handle(EntrySaved $event): void
    {
        /** @var Entry $entry */
        $entry = $event->entry;

        $this->cacher->invalidateUrl($entry->url(), config('app.url'));
    }
}

Then register this at app/Providers/EventServiceProvider.php:

<?php

namespace App\Providers;

use App\Listeners\ForceEntryInvalidation;
use Illuminate\Auth\Events\Registered;
use Illuminate\Auth\Listeners\SendEmailVerificationNotification;
use Illuminate\Foundation\Support\Providers\EventServiceProvider as ServiceProvider;
use Statamic\Events\EntrySaved;

class EventServiceProvider extends ServiceProvider
{
    /**
     * The event listener mappings for the application.
     *
     * @var array<class-string, array<int, class-string>>
     */
    protected $listen = [
        EntrySaved::class => [
            ForceEntryInvalidation::class,
        ],

        ...
    ];

    ...
}

This solution will synchronously invalidate the static cache for that entry URL. For more complex invalidation needs we will advise users to clear the static cache via Utilities > Cache Manager in the control panel, however the event listener could be adapted to perform more complex actions.

Similarly, the listener may require some adaptation if using the 'full' static caching strategy. Duncan's post linked above gives some ideas on that.

In my testing, making the listener queueable (by implementing ShouldQueue) resulted in the same behaviour as Statamic's own Invalidate listener, i.e. Horizon reports that the job runs, but it has no effect.

[mscruse]

I am seeing this on a new site I'm about to launch when QUEUE_CONNECTION is set to redis.

Horizon is reporting the Invalidate job running successfully, but the static cache files are not being cleared.

Changing QUEUE_CONNECTION to sync invalidates the cache correctly on entry save.

Environment
Laravel Version: 10.28.0
PHP Version: 8.2.12
Composer Version: 2.6.5
Environment: production
Debug Mode: OFF
Maintenance Mode: OFF

Cache
Config: NOT CACHED
Events: NOT CACHED
Routes: NOT CACHED
Views: CACHED

Drivers
Broadcasting: log
Cache: statamic
Database: mysql
Logs: stack / single
Mail: postmark
Queue: sync
Session: file

[edalzell]

I am seeing this on a new site I'm about to launch when QUEUE_CONNECTION is set to redis.

Yup, that's we set up all our multi-sites, sync is default queue, then set the queue on the git stuff and manually set the queue on any jobs we need.

[jasonvarga]

We looked into this some more and believe it's probably resolved if you use a proper url in your sites config rather than the default relative url.

i.e. in config/statamic/sites.php:

'sites' => [
    'default' => [
        'name' => 'My Site',
        'locale' => 'en_US',
-       'url' => '/',
+       'url' => 'https://mysite.com/', // or config('app.url')
    ],
],

If you are still running into this issue once updating the urls in your site config, let us know and we can reopen this.

[JorisOrangeStudio]

If you are still running into this issue once updating the urls in your site config, let us know and we can reopen this.

I'm using STATAMIC_STATIC_CACHING_STRATEGY=half and after changing the sites.php to proper url's it is working in some cases, but not always. For example when disabling STATAMIC_STACHE_WATCHER on production or changing to full strategy it does not invalidate properly. The Statamic\StaticCaching\Invalidate jobs are executed and returning DONE. However the pages are not invalidated properly, because it still shows the old content. I feel like it is still not 100% stable. We are also using redis as queue and multisites.

[donatasp94]

@jasonvarga Still an issue. Updating the url didn't fix it. Only switching from redis to sync queue fixes it

[johncarter-]

Can confirm.

[eiiot]

Still happening on 5.22.1 w/ redis or database as a queue, should this be reopened?

[chopperoon]

I'm also seeing this issue using half measure caching and database as a queue on a multisite project.