Add Cloudfront signed URL support for S3

[jqnatividad]

Since boto is already included (#1), consider including Cloudfront signed URL support.

Using boto's create_signed_url, the publisher can ensure that users visit the data portal to download the file, rather than just saving the S3 link:

• to ensure web analytics are current
• to give ability to specify policies in the future, like limiting IP addresses from which files can be downloaded
  • added bonus of increasing download performance by using AWS's CDN

[TkTech]

Hello @jqnatividad I might be missing something here, but ckanext-cloudstorage has supported secure urls for AWS and Azure since the first commit.

[MrkGrgsn]

@TkTech has a point but I think the key difference in what @jqnatividad is suggesting is using Cloudfront rather than S3, but whether that is actually a bonus will depend on a repository's download volumes and the geographic distribution of downloads.

[jqnatividad]

Hi @TkTech, the context of this request is that we have a user with a lot of large high-value files that they plan to distribute on their CKAN site as @MrkGrgsn suspected.

They don't want to be penalized for success when they release these files in terms of bandwidth overages, so we've formulated a multi-prong strategy to give them a better handle on their bandwidth costs. This includes:

• taking advantage of S3's native torrent support, so they can distribute these files in collaboration with their academic partners
• taking advantage of S3's Requester Pays option, to give commercial users, a "fast-lane" option (#9)
• and with this GH issue - taking advantage of Cloudfront's signed URL feature to better manage tracking/oversight of data downloads, e.g.:
  • higher quality web analytics as users don't just download directly from S3 URLs (signed URLs are only good for a few minutes, and are automatically updated on the CKAN interface)
  • minimize high-volume user abuse, running up bandwidth cost by creating download automations
  • transparent automatic compression of files saving bandwidth
  • giving future ability to specify custom policies when deploying CKAN as an internal data-sharing platform as well, e.g.:
  • specify valid IP addresses from which downloads can happen on a per CKAN organization basis
  • ability for commercial user to selectively "micropay" for "fast-lane" access of a very large file without having to use S3 Requester Pay buckets
  • manage when data files "expire", so users don't download "old" files unnecessarily/accidentally

[TkTech]

Hello @jqnatividad,

f you want to expose this through CloudFront instead you can subclass the Storage class to add your custom get_url_from_filename() logic after setting up your Distribution on the AWS CloudFront console. [configuration, client-specific]

If you want to expose torrent links instead you just need to update your template (ex: <a href="{{ storage.get_url_from_filename(...) }}.torrent">...</a>) since just adding .torrent to the URL will give you a torrent link. [template]

9 is a handled via one-time configuration of your S3 bucket from the AWS console, not ckanext-cloudstorage. If you were thinking of having it replicated to two buckets (one internal, one requester-pays) this is best handled by a custom background job, not the uploader. [configuration, client-specific]

--

Just to summarize, everything you want to do is best done through your client-specific extension and the AWS console, not directly in ckanext-cloudstorage.

ckanext-cloudstorage is intended to support basic CRUD functionality across a significant number of providers in a portable manner more than exposing all possible S3/AWS features, which is best left to your client extension.

[jqnatividad]

Thanks for your expert guidance @TkTech!

Will be sure to implement it per your comment above and share with you and the community our findings with our client-specific extension.