Warm from image instance

[vesterbaek]

I'm using S3 as my storage backend. Right now when getting an image upload, I download the original image and fix orientation based on exif and then save the image. I then need to warm my image cache for this image field. Right now this causes the image to be fetched from S3 once per thumbnail -- this is rather time consuming.

It would be nice to be able to warm thumbnails for a specific field given an already in-memory instance of the source image. How would I best achieve this?

[vesterbaek]

Seems this will not be straight forward to do given the way the code is currently structured. I made a hacky workaround by patching up the storage instance to always return a cached file. It's not pretty, but it gets the job done and has good impact on time spent.

class CachedImageFieldWarmer(object):
    def __init__(self, instance, rendition_key_set, image_attr, source_file):
        self.image_field = reduce(getattr, image_attr.split("."), instance)
        if isinstance(rendition_key_set, six.string_types):
            rendition_key_set = get_rendition_key_set(rendition_key_set)
        self.size_key_list = [size_key for key, size_key in validate_versatileimagefield_sizekey_list(rendition_key_set)]
        self.fixed_source_file = source_file

    def _prewarm_versatileimagefield(self, size_key):
        create_on_demand = self.image_field.create_on_demand
        self.image_field.create_on_demand = True
        storage = self.image_field.storage
        existing_open = storage.open

        def open_fixed(self, path, mode, fixed_source_file):
            fixed_source_file.seek(0)
            return fixed_source_file
        storage.open = types.MethodType(partial(open_fixed, fixed_source_file=self.fixed_source_file), storage)

        try:
            url = get_url_from_image_key(self.image_field, size_key)
        finally:
            self.image_field.create_on_demand = create_on_demand
            storage.open = existing_open
        return url

    def warm(self):
        for size_key in self.size_key_list:
            self._prewarm_versatileimagefield(size_key)

I use it as the normal warmer via a wrapper function that will use the cached version if we have the file and the normal warmer if not. It only works on a single instance and not queryset. I also changed the error reporting to be exception based instead.

def prepare_image_field_cache(instance, image_attr, rendition_key_set, source_file=None):
    if source_file:
        CachedImageFieldWarmer(instance, rendition_key_set, image_attr, source_file).warm()
    else:
        warmer = VersatileImageFieldWarmer(instance, rendition_key_set, image_attr)
        num_images_pre_warmed, failed_to_create_image_path_list = warmer.warm()
        if failed_to_create_image_path_list:
            raise Exception("Failed to warm images %s" % failed_to_create_image_path_list)

Input on how to solve this properly is welcome

[vesterbaek]

I think it would be good to restructure the code such that url generation for a key is separated from actual image file generation. Having to override image_field.create_on_demand = True in the current code is a good indication that this could be structured better. Would also make it easier to hook into the cache generation on a lower level to do stuff like I need above.