Improve performance with cloud-storage/CDNs

[camflan]

I flipped off create_images_on_demand and actually saw a performance decrease in my usage. I think this is caused (again) by leveraging a cloud-storage/cdn for my images.

When using create_images_on_demand, the URLs are cached, so retrieving URLs only have to hit Redis. However, with create_images_on_demand, the URLs are requested from my cloud-storage provider each time - which is obviously way slower than Redis.

This is at least 1 of the places we need to update in order to resolve this issue. https://github.com/respondcreate/django-versatileimagefield/blob/d46f05626bce1327886bb9673e533ba8206d8da9/versatileimagefield/datastructures/filteredimage.py#L146

I think if we move the cache.get() call outside the self.create_on_demand check, then we can also cache.set in the imagewarmer.

My thinking is, we cache the URLs when they are warmed and then we can bypass the create check - because either way (prewarming vs on-demand) we already have the URL and don't need to do anymore work.

I plan on working on this over the next couple of days, but if anyone has ideas or input - please let me know!

cc @respondcreate

[camflan]

Closing, this is actually a bug in django-storages. I'll work on the underlying bug there instead :)

I'm still up for working on the above proposal if it will improve this library as well 👍

[bunnyfly]

Hey camflan - did you make any progress on this or have any good workaround/solutions?

I'm using this with django-s3-storage. I also set create_images_on_demand False to speed things up because on-demand creation was slow. But now URLs aren't being cached and generating URLs every API call is very slow.

According to my profiling, about 500ms+ is added to each query of a small list of objects with versatileimagefields.

I switched from django-s3-storage to django-storages like you mention here, but still no quick URLs. It's a little faster but not much - still about 500ms spent in: /site-packages/storages/backends/s3boto3.py:609(url) /site-packages/botocore/signers.py:535(generate_presigned_url)

My URLs don't have auth or anything...just plain links so there's no reason to generate them every time. I have set: AWS_DEFAULT_ACL = 'public-read' AWS_QUERYSTRING_AUTH = False AWS_S3_BUCKET_AUTH = False

[camflan]

@bunnyfly I haven't worked on this patch yet, though I think it would be beneficial. I'm also using a storage class wrapper around GoogleStorage with Django-storages to bypass the roundtrip to the CDN for getting URLs.

class GoogleCloudMediaStorage(GoogleCloudStorage):
    """GoogleCloudStorage suitable for Django's Media files."""

    def __init__(self, *args, **kwargs):
        if not settings.MEDIA_URL:
            raise Exception("MEDIA_URL has not been configured")
        kwargs["bucket_name"] = setting("GS_MEDIA_BUCKET_NAME")
        super(GoogleCloudMediaStorage, self).__init__(*args, **kwargs)

    def url(self, name):
        """.url that doesn't call Google."""
        return urljoin(settings.MEDIA_URL, name)

With this, I have the CDN base URL set as MEDIA_URL and then my URLs are created entirely in Django without having to reach across the network

[ilikerobots]

@camflan : Huge thanks for that ^^. I'm using ImageKit with VersatileImageField with an AWS S3 backend and spent a great deal of time trying to "fix" ImageKit's caching. Your comment illuminated the real issue.

[camflan]

@camflan : Huge thanks for that ^^. I'm using ImageKit with VersatileImageField with an AWS S3 backend and spent a great deal of time trying to "fix" ImageKit's caching. Your comment illuminated the real issue.

Hey @ilikerobots – that's awesome, glad it helped someone else 🤠