[Epic][AP] CMS Public S3 file asset storage

timcosgrove commented 1 year ago

Background

CMS-managed file assets - images and PDFs primarily, that are added by CMS editors as part of their content - are stored in a file store that is within the VA network, like the CMS itself is. This means that when CMS content data that references these files, the URLs are to locations that are not accessible by regular users.

Content Build solves this problem by downloading every file asset used in VA.gov pages with every single content release. This is time-consuming.

To solve this problem for Next Build, we have set up file sync from the CMS file stores to public S3 buckets that can operate as web servers. Files are synced automatically as they are added to the CMS; the sync typically takes a few minutes.

As pages are built by Next Build, we need to substitute the URL to a given file as the CMS understands it with a URL to the public S3 bucket location. This will need to be done on all text where file references could existn.

Assumptions

In all cases here we are speaking of file assets that are managed by the CMS and delivered with a page - images and PDFs primarily.

Acceptance Criteria

[ ] Files that are saved in the production or staging CMS are stored in a publicly-reachable S3 bucket acting as a web server.
[ ] All existing CMS-managed files are migrated to these same S3 buckets
[ ] Next Build is updated to make use of these new locations.
[ ] Optionally: Content Build is also updated to make use of these locations and the old download functionality is removed.

### Tasks
- [ ] https://github.com/department-of-veterans-affairs/va.gov-cms/issues/18437

### Completed Tasks
- [ ] https://github.com/department-of-veterans-affairs/va.gov-cms/issues/16929
- [ ] https://github.com/department-of-veterans-affairs/va.gov-cms/issues/16930
- [ ] https://github.com/department-of-veterans-affairs/va.gov-cms/issues/18071
- [ ] https://github.com/department-of-veterans-affairs/va.gov-cms/issues/18070
- [ ] https://github.com/department-of-veterans-affairs/va.gov-cms/issues/16925
- [ ] https://github.com/department-of-veterans-affairs/va.gov-cms/issues/16926
- [ ] https://github.com/department-of-veterans-affairs/va.gov-cms/issues/16928

timcosgrove commented 1 year ago

See #10894 for anything useful.

timcosgrove commented 11 months ago

Note that this epic was written with the assumption of a more managed solution to files. We have also proposed 'dumb sync' where anything written by the CMS to its asset file system automatically syncs to S3.

We need to investigate that proposal, write it up, put out an RFC

department-of-veterans-affairs / va.gov-cms