harvard-lil / perma

Indelible links
408 stars 72 forks source link

Add a secondary storage to support wacz experiments. #3520

Closed rebeccacremona closed 1 month ago

rebeccacremona commented 1 month ago

We've been working on plans to start capturing and playing back WACZ-formatted archives, either in addition to or instead of WARC-formatted. We've also been experimenting with converting our existing archives to WACZ.

To experiment more safely, we'd like to have another S3 bucket available, one that we can feel comfortable messing around with, deleting artifacts from, etc... entirely separate from where we keep the WARCs for Perma's actual collection, which, appropriately, has very strict rules.

This PR adds config for an additional "secondary" storage, and locally wires it up to minio.

It should be safe to deploy without changing stage or prod config, and without creating any additional cloud storage: we can do that in the future, if we have any code that actually uses the secondary storage that we want to try out in stage or prod.

codecov[bot] commented 1 month ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 68.60%. Comparing base (42c3d5d) to head (7a8092d). Report is 92 commits behind head on develop.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## develop #3520 +/- ## ======================================== Coverage 68.60% 68.60% ======================================== Files 48 48 Lines 6794 6795 +1 ======================================== + Hits 4661 4662 +1 Misses 2133 2133 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.