stephenmcd / mezzanine

CMS framework for Django
http://mezzanine.jupo.org
BSD 2-Clause "Simplified" License
4.76k stars 1.65k forks source link

Improvements to thumbnails code for using remote media storage #1952

Open cjh79 opened 4 years ago

cjh79 commented 4 years ago

This PR includes two improvements to the thumbnails code when using remote media storage (eg. S3).

1. Allow absolute path for THUMBNAILS_DIR_NAME

Allow the THUMBNAILS_DIR_NAME setting to specify an absolute path. In this case, THUMBNAILS_DIR_NAME will be created relative to the MEDIA_ROOT directory, rather than relative to each image's directory. Each image's path is recreated below the root THUMBNAILS_DIR_NAME directory to avoid name collisions. Default behavior is not changed, so existing installations are not affected.

THUMBNAILS_DIR_NAME = "/.thumbnails"

Using this feature offers improvements to listdir() on eg. a Boto3 storage backend, which is constrained to pulling all objects under a particular directory (ie. recursively) due to the nature S3-like object storage systems. With many thumbnail images, this can be a major performance hit, for example when browsing the Media Library in admin. By keeping this directory at the root level, outside of the location of media library files (/uploads), thumbnails do not show up in the Media Library at all, mitigating the problem.

2. Use default_storage for thumbnail creation rather than os

Implementation of the fix described here: https://groups.google.com/forum/#!msg/mezzanine-users/WGVaNhD5vRs/7Ktw7-UkosgJ. Thumbnails are now created solely with default_storage rather than os functions, which of course does not work when using S3 for media storage.

Note that this fix also appears to be implemented in a couple of other PRs: https://github.com/stephenmcd/mezzanine/pull/1951 and https://github.com/stephenmcd/mezzanine/pull/1922

cjh79 commented 4 years ago

Note that I have created a PR in the filebrowser-safe project which corresponds to this change.

https://github.com/stephenmcd/filebrowser-safe/pull/132