Long-term solution to large file storage

eddietejeda commented 6 years ago

User Story

As a site owner, I want my Federalist site to host large files (images / PDFs) without concern of running out of build container memory.

Background

There's no good reason that Federalist needs to load large files into memory. They just need to get dropped into S3 somehow, and our current method is inefficient and wasteful.

https://gsa-tts.slack.com/archives/C04823NMA/p1487884812002420

Acceptance Criteria

[ ] Change is deployed via deploy of Lambda + s3 supporting GitLFS

Level of effort - High

Implementation outline (if higher than "low" effort):

[x] Talk to will about backstory
[x] Investigate whitelisting files or directories other options?
[x] Finalize decision of our approach going forward
[x] Discuss w/ ISSO allowing clients access to s3 bucket via AWS cli to a lfs path
[ ] Investigate Git LFS using Lambda + s3

eddietejeda commented 5 years ago

Follow up with ONNR to discuss their build process in more detail.

eddietejeda commented 5 years ago

This issue is going into icebox for a while. We're about to make changes to our infrastructure and need to determine where this fits in.

eddietejeda commented 5 years ago

Can we evaluate this: https://aws.amazon.com/athena/

davemcorwin commented 5 years ago

@eddietejeda you may want to create a new issue for looking into Athena. This issue looks to be about large static resources, such as images and pdfs.
It might be useful to come up with a use case for this. It looks like its focus is reading large datasets that already exist in S3, such as logging, so would our users want a feature to upload json files of data to be queried?

cloud-gov / pages-core