make a plan about perma -> wacz

[matteocargnelutti]

Discussed today - https://hlslil.slack.com/archives/C07URASMC/p1659730375523229

Summary:

• Using .waczs, which are meant to be range-streamed and are cached differently than .warc.gzs by <replay-web-page>, would help alleviate the limitations of client-side storage we're dealing with.
• We could start a new collection of .wacz files by converting .warc.gz on the fly as they are requested, and storing the resulting artifact.
• We could alternatively convert everything at once, with the cut off being Perma's transition to a capture engine generating .waczs by default.
• Aspects to consider:
  • storage and computation costs
  • whether it matters or not if the entire collection gets converted to waczs
  • signing
  • error detection and automation

---

Would running a small scale experiment locally on a batch of X archives first, with automated checks to identify edge cases, be a good first step?

[matteocargnelutti]

Related to #3038

[rebeccacremona]

Throwing a thought in this ticket: if we continue to send our stuff to Internet Archive after this migration, we will probably want to continue sending them WARCs instead of WACZs. I think they will prefer to derive their own CDX lines and store them in their own format like they do currently.