Create archives of raw responses for Data Sources automatically

[josh-chamberlain]

Context

A prototype, we are making automatic archives via internet archive in this repository.

Long term strategy

For long-term stability, we should use our own archival solution, distributed storage, etc. This repository currently relies on an API which is documented via google docs.

Great writing, if a bit out of date, about why this stability is important: https://github.com/pdapm/intro

Potential next steps

• PDAP will need its own storage holdings, which are not currently in the budget. We don't want to end up paying more and more for something we rely on, anyway.
  • People have offered to donate cloud and local storage
  • We may be able to get a grant.
• PDAP could maintain its own storage archive of select Data Sources (or all of them if we're masochistic)
• PDAP could crawl deeper into Data Sources and archive any records which have been linked
• We could help people use Archivebox to store particular Data Sources locally

[josh-chamberlain]

I reached out to https://wiki.archiveteam.org/. Their response:

So, a lot of this seems like it would be better as pulling into the wayback? We have space for datasets and files, of course, but webpages work great with the wayback machine.

So, unless we are talking terabytes, uploading sets of items to the archive into a collection named after you is simple. And if we're talking terabytes, I can arrange that too.

jason@textfiles.com can be reached for next steps if needed.