elifesciences / elife-download

Tools and details about how to download eLife article data
MIT License
0 stars 0 forks source link

Create S3 bucket(s) to store article zip files for the public #2

Open gnott opened 6 years ago

gnott commented 6 years ago

The goal is to create public readable bucket(s) to store article zip files for anyone to download.

Some initial ideas are not finalised yet, for example

@giorgiosironi perhaps you may be able to help here if the buckets should be defined in infrastructure definitions.

gnott commented 6 years ago

Looking back at some notes, we were also thinking about a separate folder for PoA and VoR articles.

Also, the folder / path structure is not decided. The zip file name format is also not decided.

giorgiosironi commented 6 years ago

In general, the fewer buckets an application has, the easier to maintain as the paths are kept within the application rather than being exported to infrastructure definitions in other project. For example, a single bucket with poa/ and vor/ prefixes (or other prefixes, no preferences expressed here) is functionally identical to two different buckets but preferable to multiple ones because it's a single infrastructure definition that will change less often than multiple buckets. S3 has filters by prefix so listing the files (the only bucket-wide operation) can still be performed.

On that note, we have prod-elife-published which is publicly readable and also has a CDN on top. The CDN could also be useful to deliver these files so they could be pushed there as zip/poa/.../zip/vor/... (again, just sample prefixes, no preferences).

gnott commented 6 years ago

Super suggestions Giorgio, thanks. Reusing the existing bucket should be fine. It sounds like we want to keep poa and vor content separate, so the zip/poa/ and zip/vor/ folders are perfect.