Using Glide with a CDN doesn't appear to use cached results

michaelr0 commented 2 years ago

Bug description

I'm trying to use Glide, Statamic Tag's and a CDN on a project.

However what I'm finding is that the use of Glide with a CDN, seems that it does not use the cache and always does a request to the CDN to see if the image exists.

I'm using the Statamic::tag() helper to access the needed glide images, like so.

Statamic::tag('glide')->params($params)->fetch()

I can see that files are generated in storage/framework/cache/glide, which seem to contain the correct url, however the response times, measured with Blackfire seems to suggest that these files are never used, as the response time for an api that just returns the url of a single image, is 4.5 seconds.

Looking at https://github.com/statamic/cms/blob/3.3/src/Imaging/ImageGenerator.php I can see that the ImageGenerator uses a Glide::cacheStore method, so If I replicate the usage of that, then the response is much much faster, after generating a cache (which doesn't match the cache that is used by the glide tag)

Glide::cacheStore()->rememberForever('path::'.$path.'::'.md5(json_encode($params)), function () use ($path, $params) {
    return Statamic::tag('glide')->params(array_merge(['src' => $path], $params))->fetch();
});

How to reproduce

Configure Glide to use S3 for storage
Try using Statamic::tag('glide')

Logs

No response

Versions

Statamic 3.3.11 Pro Laravel 9.12.2 PHP 8.1.6

Installation

Fresh statamic/statamic site via CLI

Antlers Parser

runtime (new)

Additional details

No response

jasonvarga commented 2 years ago

Can you show your config/statamic/assets.php and config/filesystems.php files?

michaelr0 commented 2 years ago

Sure thing @jasonvarga

assets.php

<?php

return [
    'image_manipulation' => [
        /*
        |--------------------------------------------------------------------------
        | Route Prefix
        |--------------------------------------------------------------------------
        |
        | The route prefix for serving HTTP based manipulated images through Glide.
        | If using the cached option, this should be the URL of the cached path.
        |
        */

        'route' => 'img',

        /*
        |--------------------------------------------------------------------------
        | Require Glide security token
        |--------------------------------------------------------------------------
        |
        | With this option enabled, you are protecting your website from mass image
        | resize attacks. You will need to generate tokens using the Glide tag
        | but may want to disable this while in development to tinker.
        |
        */

        'secure' => true,

        /*
        |--------------------------------------------------------------------------
        | Image Manipulation Driver
        |--------------------------------------------------------------------------
        |
        | The driver that will be used under the hood for image manipulation.
        | Supported: "gd" or "imagick" (if installed on your server)
        |
        */

        'driver' => 'gd',

        /*
        |--------------------------------------------------------------------------
        | Save Cached Images
        |--------------------------------------------------------------------------
        |
        | Enabling this will make Glide save publicly accessible images. It will
        | increase performance at the cost of the dynamic nature of HTTP based
        | image manipulation. You will need to invalidate images manually.
        |
        */

        'cache' => 'wasabi-glide',

        'cache_path' => public_path('img'),

        /*
        |--------------------------------------------------------------------------
        | Image Manipulation Presets
        |--------------------------------------------------------------------------
        |
        | Rather than specifying your manipulation params in your templates with
        | the glide tag, you may define them here and reference their handles.
        | They will also be automatically generated when you upload assets.
        |
        */

        'presets' => [
            // 'small' => ['w' => 200, 'h' => 200, 'q' => 75, 'fit' => 'crop'],
        ],
    ],

    /*
    |--------------------------------------------------------------------------
    | Auto-Crop Assets
    |--------------------------------------------------------------------------
    |
    | Enabling this will make Glide automatically crop assets at their focal
    | point (at at the center if no focal point is defined). Otherwise,
    | you will need to manually add any crop related parameters.
    |
    */

    'auto_crop' => true,

    /*
    |--------------------------------------------------------------------------
    | Control Panel Thumbnail Restrictions
    |--------------------------------------------------------------------------
    |
    | Thumbnails will not be generated for any assets any larger (in either
    | axis) than the values listed below. This helps prevent memory usage
    | issues out of the box. You may increase or decrease as necessary.
    |
    */

    'thumbnails' => [
        'max_width' => 10000,
        'max_height' => 10000,
    ],

    /*
    |--------------------------------------------------------------------------
    | File Previews with Google Docs
    |--------------------------------------------------------------------------
    |
    | Filetypes that cannot be rendered with HTML5 can opt into the Google Docs
    | Viewer. Google will get temporary access to these files so keep that in
    | mind for any privacy implications: https://policies.google.com/privacy
    |
    */

    'google_docs_viewer' => false,

    /*
    |--------------------------------------------------------------------------
    | Cache Metadata
    |--------------------------------------------------------------------------
    |
    | Asset metadata (filesize, dimensions, custom data, etc) will get cached
    | to optimize performance, so that it will not need to be constantly
    | re-evaluated from disk. You may disable this option if you are
    | planning to continually modify the same asset repeatedly.
    |
    */

    'cache_meta' => true,

    /*
    |--------------------------------------------------------------------------
    | Focal Point Editor
    |--------------------------------------------------------------------------
    |
    | When editing images in the Control Panel, there is an option to choose
    | a focal point. When working with third-party image providers such as
    | Cloudinary it can be useful to disable Statamic's built-in editor.
    |
    */

    'focal_point_editor' => true,
];

filesystems.php

<?php

$links = [];

if ('local' === env('APP_ENV', 'production')) {
    $links[public_path('vendor/buildamic')] = base_path('vendor/handmadeweb/buildamic/public');
}

return [
    /*
    |--------------------------------------------------------------------------
    | Default Filesystem Disk
    |--------------------------------------------------------------------------
    |
    | Here you may specify the default filesystem disk that should be used
    | by the framework. The "local" disk, as well as a variety of cloud
    | based disks are available to your application. Just store away!
    |
    */

    'default' => env('FILESYSTEM_DRIVER', 'local'),

    /*
    |--------------------------------------------------------------------------
    | Default Cloud Filesystem Disk
    |--------------------------------------------------------------------------
    |
    | Many applications store files both locally and in the cloud. For this
    | reason, you may specify a default "cloud" driver here. This driver
    | will be bound as the Cloud disk implementation in the container.
    |
    */

    'cloud' => env('FILESYSTEM_CLOUD', 's3'),

    /*
    |--------------------------------------------------------------------------
    | Filesystem Disks
    |--------------------------------------------------------------------------
    |
    | Here you may configure as many filesystem "disks" as you wish, and you
    | may even configure multiple disks of the same driver. Defaults have
    | been setup for each driver as an example of the required options.
    |
    | Supported Drivers: "local", "ftp", "sftp", "s3"
    |
    */

    'disks' => [
        'local' => [
            'driver' => 'local',
            'root' => storage_path('app'),
        ],

        'public' => [
            'driver' => 'local',
            'root' => storage_path('app/public'),
            'url' => env('APP_URL').'/storage',
            'visibility' => 'public',
        ],

        // 's3' => [
        //     'driver' => 's3',
        //     'key' => env('AWS_ACCESS_KEY_ID'),
        //     'secret' => env('AWS_SECRET_ACCESS_KEY'),
        //     'region' => env('AWS_DEFAULT_REGION'),
        //     'bucket' => env('AWS_BUCKET'),
        //     'url' => env('AWS_URL'),
        //     'endpoint' => env('AWS_ENDPOINT'),
        //     // 'visibility' => 'public', // https://statamic.dev/assets#visibility
        // ],

        'wasabi-glide' => [
            'driver' => 's3',
            'key' => env('WASABI_ACCESS_KEY'),
            'secret' => env('WASABI_SECRET_KEY'),
            'region' => env('WASABI_DEFAULT_REGION'),
            'bucket' =>  env('WASABI_BUCKET'),
            'url' =>  env('WASABI_BUCKET_URL'),
            'endpoint' => env('WASABI_ENDPOINT'),
            'visibility' => 'public', // https://statamic.dev/assets#visibility
        ],

        'assets' => [
            'driver' => 'local',
            'root' => public_path('assets'),
            'url' => '/assets',
            'visibility' => 'public',
        ],
    ],

    /*
    |--------------------------------------------------------------------------
    | Symbolic Links
    |--------------------------------------------------------------------------
    |
    | Here you may configure the symbolic links that will be created when the
    | `storage:link` Artisan command is executed. The array keys should be
    | the locations of the links and the values should be their targets.
    |
    */

    'links' => array_merge([
        public_path('storage') => storage_path('app/public'),
    ], $links),
];

michaelr0 commented 2 years ago

So far I have found that by modifying https://github.com/statamic/cms/blob/3.3/src/Tags/Glide.php#L170

    private function generateGlideUrl($item)
    {
        try {
            $url = $this->isResizable($item) ? $this->getManipulator($item)->build() : $this->normalizeItem($item);
        } catch (\Exception $e) {
            \Log::error($e->getMessage());

            return;
        }

        $url = ($this->params->bool('absolute', $this->useAbsoluteUrls())) ? URL::makeAbsolute($url) : URL::makeRelative($url);

        return $url;
    }

to the following:

    private function generateGlideUrl($item)
    {
        $cache = config('statamic.assets.image_manipulation.cache');

        if (is_string($cache)) {
            $params = $this->params->except(['src', 'id', 'path']);
            $cacheKey = 'asset::'.$item.'::'.md5(json_encode($params));

            if ($url = GlideFacade::cacheStore()->get($cacheKey)) {
                return Storage::disk($cache)->url($url);
               // Or maybe return GlideFacade::url().'/'.$url;
            }
        }

        try {
            $url = $this->isResizable($item) ? $this->getManipulator($item)->build() : $this->normalizeItem($item);
        } catch (\Exception $e) {
            \Log::error($e->getMessage());

            return;
        }

        $url = ($this->params->bool('absolute', $this->useAbsoluteUrls())) ? URL::makeAbsolute($url) : URL::makeRelative($url);

        return $url;
    }

I have been able to get the response times of an api call that uses Statamic::tag('glide') to request 1 image, down from 5.58s to 304ms and 12.1 MB peak memory to 5.41 MB.

There would need to be some other considerations, as I'm unsure at this stage if this has impacted anything else, but it appears to be working how I'd expect it.

jasonvarga commented 2 years ago

Do you have the Stache watcher enabled in config/statamic/stache.php?

Do your load times decrease if turning that off? (Without your changes to Glide.php)

michaelr0 commented 2 years ago

Hi @jasonvarga,

I can confirm that the load times are better with stache watcher disabled, memory usage remains about the same as it was (not the reduction seen in the changes made)

imacrayon commented 1 year ago

So far I have found that by modifying https://github.com/statamic/cms/blob/3.3/src/Tags/Glide.php#L170

This change significantly sped up my page loads as well, however, I had to change this line:

$params = $this->params->except(['src', 'id', 'path']);

To this:

$params = $this->getGlideParams($item);

Otherwise using Statamic's special params like preset and crop_focal messed up the cache key.

imacrayon commented 1 year ago

I made a patch for the Glide tag, it works for paired and single tags:

https://gist.github.com/imacrayon/17c5bc47c2d49e19768de260cf0a895c

grantholle commented 1 year ago

I'm not sure how I just discovered this issue only now, but definitely experiencing this. Locally, my load times are 40s+. On the server, because it's close to the CDN, the load times are ok enough, but can definitely tell the glide cache is being ignored.

I posted a message in the Discord, but no one seemed to know.

Could that patch be added as a PR? Any reason it hasn't been?

grantholle commented 1 year ago

I've done a little more debugging and trying to understand how it's all working.

I've noticed that sometimes on line 168, $item is already an instance of Statamic\Contracts\Assets\Asset (I think it's when passing the asset itself to src in Antlers, rather than just the path?). When calling Asset::find($item), it rehydrates data about the file. In this case, to the external object storage.

Adding a check on the type of $item prevents one extra call to pull that data:

private function generateImage($item)
{
    $item = $this->normalizeItem($item);
    $params = $this->getGlideParams($item);

    if (is_string($item) && Str::isUrl($item)) {
        $image = Str::startsWith($item, ['http://', 'https://'])
            ? $this->getGenerator()->generateByUrl($item, $params)
            : $this->getGenerator()->generateByPath($item, $params);

        return $image;
    }

    if (!$item instanceof AssetContract) {
        $item = Asset::find($item);
    }

    return $this->getGenerator()->generateByAsset($item, $params);
}

However, I'm noticing that the underlying issue probably isn't Glide outside of this instance, but how the asset itself is retrieved and what is retrieved about it. Still investigating, but wanted to report my findings here. If someone else has more knowledge it would be really helpful, as this is destroying my performance.

imacrayon commented 1 year ago

I think some of the initial issues I was encountering have been ironed out. However, I can confirm there's still some slowness when using the Glide tag pair; I think has to do with this section of code: https://github.com/statamic/cms/blob/1c287bc5dd89169b19cc550bf2fd94f8aab25969/src/Imaging/Attributes.php#L20-L26 Here the $source disk would be my CDN and $this->cacheDisk() is a local image attribute cache in storage/statamic/attributes-cache

The code looks to be blowing out the image attribute cache every time image attributes are read from the Glide tag.

Here's the calling line within the Glide tag: https://github.com/statamic/cms/blob/1c287bc5dd89169b19cc550bf2fd94f8aab25969/src/Tags/Glide.php#L139 The GlideManager::cacheDisk() passed into from is my CDN disk.

statamic / cms