search

TkTech / ckanext-cloudstorage

Implements support for resource storage against multiple popular providers via apache-libcloud (S3, Azure Storage, etc...)
MIT License
35 stars 55 forks source link

S3 BitTorrent downloads #33

Open jqnatividad opened 5 years ago

jqnatividad commented 5 years ago

"Any object in Amazon S3 that can be read anonymously can also be downloaded via BitTorrent."

https://docs.aws.amazon.com/AmazonS3/latest/dev/S3TorrentRetrieve.html

"Retrieving a .torrent file for any publicly available object is easy. Simply add a "?torrent" query string parameter at the end of the REST GET request for the object."

It'd be nice if there's an INI option to expose a torrent download link. This will help CKAN instances not be penalized for success when publishing large popular datasets - in terms of availability and bandwidth overages.

TkTech commented 5 years ago

Any ideas on how to display it? There's no real way in the UI to currently handle multiple download links for a single resource, and the templates used to list the resources are practically always customized by other sites, so any custom template we provided would either conflict or not be used at all.

jqnatividad commented 5 years ago

Perhaps, using something like a DownloadTorrent resource extras flag to indicate that a resource should be available as a torrent link? If its not present, its just a normal download, if its TorrentOnly, its only available as a Torrent link, and if its TorrentAlso, its available as both a normal download and a torrent link?

Implementing this in a reference template should help guide sites that want to take advantage of this torrents. And if they don't, it will still behave as before...

jqnatividad commented 5 years ago

BTW, this idea was borne out by one user whose datasets are very popular with academia. Oftentimes, universities have very big, practically "free" pipes, and giving this option allows them to have de-facto peering with them.

This also helps if there's another government shutdown similar to 2013 - https://opendata.stackexchange.com/questions/1199/is-data-gov-down-due-to-the-government-shutdown-or-is-it-down-for-good

which inspired the data mirror work you did with UC3 last year - http://uc3.cdlib.org/2017/02/09/government-data-at-risk/

cc @philipashlock @JJediny