ThrawnCA
commented
3 years ago Our fork at https://github.com/qld-gov-au/ckanext-s3filestore recently added support for serving resources via CloudFront (currently on the 'develop' branch). It has two parts:
- A configuration option that allows you to substitute a different hostname in the signed URL, so that you can use your own (CloudFront) domain and forward requests to S3 from there instead of sending them to the application;
- Caching of signed URLs in Redis (default 30 minutes).
Note that we did attempt to use the time-freezing approach described at https://advancedweb.hu/how-to-use-s3-signed-urls-with-cloudfront/ to do our caching, by truncating the timestamp to the last half-hour before signing. However, this turned out to be unreliable; monkey-patching the system time functions like that resulted in race conditions where other code obtained a reference to the fake time function and then encountered errors calling it later.
Also note that our fork includes a variety of other features that may or may not interest you.
Cloudfront - AWS's CDN, improves the user experience.
It can also be deployed to help control bandwidth cost and stop scrapers/bots from abusing downloads.
By using CloudFront Signed URLs, we can add code to render signed URLs that expire after a given time (e.g. every hour).
This way, users can still download files without requiring a login and have the additional benefit of low-latency access, while minimizing scraper/bot abuse.