Add an S3 storage plugin

[ThiefMaster]

(Brainstorming - comments welcome!)

This storage plugin (storage_s3) would use boto3 (or any other well-maintained AWS/S3 client) to implement a storage backend (s3:...) for attachments etc. backed by S3.

Some functionality the plugin should have:

• Configurable bucket name (restricted charset, i.e. something like ^[a-z0-9-]{3,63}$)
• Dynamic bucket names - so you can have one bucket per year/month/week (day might be overkill). either via placeholder in the bucket name or separate option (similar to datestamp in the eos/xrootd backend)
• Maintenance task (celery): applies to all s3 STORAGE_BACKENDS that use dynamic buckets. create next bucket early enough (e.g. one period before it is needed, e.g. a february bucket would be created on 1st of january, a week 2 bucket would be created beginning of week 1, for yearly it might be enough to do it the month before as well (ie never more than one month before it's needed)). The maintenance task needs to know what ACL etc to set on the bucket. Specifying an ACL would probably be ugly, let's see if we can copy an existing "template-like" bucket for this?
• S3 credentials should be loadable from a file (maybe using the default format uses for aws credentials) in addition to putting them directly in the config string. we could go one step further and add general support for loading settings from a config file, e.g. something like s3:config=/path/to/config.file,rotate=weekly to keep the indico config cleaner when using multiple s3 backends with almost identical configs
• Backup is out of the scope of this plugin; that would be done separately if needed
• Downloading files (for users) is done by redirecting to a pre-signed URL; by default we might want to use the 'virtual' addressing_style, but being able to override this probably makes sense. To discuss: always specify content-disposition/content-type with what we have in our database or just specify it during save and assume it remains correct? TTL of the signed links?

While we will most likely use Ceph's S3 API at CERN - also for testing/development - we should make sure the backend works fine with the "real' Amazon S3 as well. The free trial account might do the job, since it includes S3 storage as well.

Useful docs link: http://boto3.readthedocs.io/en/latest/guide/s3.html

[ThiefMaster]

@mvidalgarcia suggested libcloud which looks VERY nice in terms of its Python API - when using that the plugin would be a generic cloud storage plugin instead of being S3-specific.

The only real drawback I see is that it seems to lack an equivalent of boto3's generate_presigned_url() method that generates a temporary public URL to download a specific file. While it might not be too hard to implement the signature generation, it would be problematic when used with something that isn't S3...

cloudstorage looks good as well, but unfornately it supports only Python 3. Besides that, it has hard dependencies against all the various cloud storage client libraries which isn't amazing since it pulls in a ton of new packages (especially for Azure). BTW, their docs indicate that libcloud does indeed lack support for generating signed URLs:

Generate temporary signed URLs for downloading and uploading files.