Create archives of raw responses for Data Sources automatically

[josh-chamberlain]

The prototype: https://github.com/Police-Data-Accessibility-Project/automatic-archives

Great writing about why this is important: https://github.com/pdapm/intro

This issue is about the long term strategy:

• PDAP will need its own storage holdings, which are not currently in the budget. We don't want to end up paying more and more for something we rely on, anyway.
  • People have offered to donate cloud and local storage
  • We may be able to get a grant.
• PDAP could maintain its own storage archive of select Data Sources (or all of them if we're masochistic)
• PDAP could crawl deeper into Data Sources and archive any records which have been linked
• We could help people use Archivebox to store particular Data Sources locally

[josh-chamberlain]

I reached out to https://wiki.archiveteam.org/. Their response:

So, a lot of this seems like it would be better as pulling into the wayback? We have space for datasets and files, of course, but webpages work great with the wayback machine.

So, unless we are talking terabytes, uploading sets of items to the archive into a collection named after you is simple. And if we're talking terabytes, I can arrange that too.

jason@textfiles.com can be reached for next steps if needed.

[josh-chamberlain]

We're using an API which is currently documented here: https://docs.google.com/document/d/1Nsv52MvSjbLb2PCpHlat0gkzw0EvtSgpKHu4mk0MnrA/edit#heading=h.1gmodju1d6p0

It makes sense to me that we would use our own archival solution, distributed storage, etc