This PR includes two improvements to the thumbnails code when using remote media storage (eg. S3).
1. Allow absolute path for THUMBNAILS_DIR_NAME
Allow the THUMBNAILS_DIR_NAME setting to specify an absolute path. In this case, THUMBNAILS_DIR_NAME will be created relative to the MEDIA_ROOT directory, rather than relative to each image's directory. Each image's path is recreated below the root THUMBNAILS_DIR_NAME directory to avoid name collisions. Default behavior is not changed, so existing installations are not affected.
THUMBNAILS_DIR_NAME = "/.thumbnails"
Using this feature offers improvements to listdir() on eg. a Boto3 storage backend, which is constrained to pulling all objects under a particular directory (ie. recursively) due to the nature S3-like object storage systems. With many thumbnail images, this can be a major performance hit, for example when browsing the Media Library in admin. By keeping this directory at the root level, outside of the location of media library files (/uploads), thumbnails do not show up in the Media Library at all, mitigating the problem.
2. Use default_storage for thumbnail creation rather than os
This PR includes two improvements to the thumbnails code when using remote media storage (eg. S3).
1. Allow absolute path for THUMBNAILS_DIR_NAME
Allow the
THUMBNAILS_DIR_NAME
setting to specify an absolute path. In this case,THUMBNAILS_DIR_NAME
will be created relative to theMEDIA_ROOT
directory, rather than relative to each image's directory. Each image's path is recreated below the rootTHUMBNAILS_DIR_NAME
directory to avoid name collisions. Default behavior is not changed, so existing installations are not affected.Using this feature offers improvements to
listdir()
on eg. a Boto3 storage backend, which is constrained to pulling all objects under a particular directory (ie. recursively) due to the nature S3-like object storage systems. With many thumbnail images, this can be a major performance hit, for example when browsing the Media Library in admin. By keeping this directory at the root level, outside of the location of media library files (/uploads
), thumbnails do not show up in the Media Library at all, mitigating the problem.2. Use default_storage for thumbnail creation rather than os
Implementation of the fix described here: https://groups.google.com/forum/#!msg/mezzanine-users/WGVaNhD5vRs/7Ktw7-UkosgJ. Thumbnails are now created solely with
default_storage
rather thanos
functions, which of course does not work when using S3 for media storage.Note that this fix also appears to be implemented in a couple of other PRs: https://github.com/stephenmcd/mezzanine/pull/1951 and https://github.com/stephenmcd/mezzanine/pull/1922