feat: initial API, round out flow for local proof-of-concept

[b5]

Opening this PR now so others can track progress & add comments.

One finished, this PR should be an initial implementation of #16, and allow the following flow:

1. Run a crawl, recording details to a local directory of CBOR records (walk start)
2. Do another crawl, recording to a different local directory of CBOR records (walk start again, with a different target dir)
3. spin up an API that reads from both local directories, and serves an API that allows time/url parsing across (walk server)

Once that's possible, we merge, party, and move on to filing lots of bugs.

I'd like to break this PR into two parts: the first to land just a proof-of-concept workflow so others can play with it, and then a follow up PR to incorporate feedback. This second PR should be where a lot of bug closing happens. and we get out of proof-of-concept land and into usable code land. By doing it this way I'm hoping a brave soul or two will be willing to use this buggy code to help write some initial documentation.

After that, only three things stand in our way of an initial staging server:

• Adapters for a cloud-based object store (Amazon S3 probs)
• Containerize, configure
• Deploy to k8s

[Frijol]

I'd like to break this PR into two parts: the first to land just a proof-of-concept workflow so others can play with it, and then a follow up PR to incorporate feedback. This second PR should be where a lot of bug closing happens. and we get out of proof-of-concept land and into usable code land. By doing it this way I'm hoping a brave soul or two will be willing to use this buggy code to help write some initial documentation.

Love this.

[Mr0grog]

From review on a call:

• Should print port when running walk server
• Need 404 for bad URLs
• /captures/resolved/now behaves differently than /captures/resolved/now/
• Chokes on having protocol in URL
  • This: /captures/resolved/now/https://youtube.com redirects to /captures/resolved/now/https:/youtube.com (note one slash after the protocol); this should probably either work fine or redirect to a version with the protocol removed
  • The redirect above is a 500 error (should probably be a 404; see also the note about /captures/resolved/now/ above)
• Would be nice to record the headers in the metadata
  • Possibly replay this in the headers for raw captures (Wayback does this by prefixing original, replayed headers with X-Archive-Orig-)
• Will need paging in short order
• Ideally would have some other querystring-based filters, like time range (for the collection index)

[b5]

Oooook I think this is ready for review. Lots to do, but from our latest round of feedback:

• [x] Should print port when running walk server
• [x] Need 404 for bad URLs (needs confirmation tho)
• [x] /captures/resolved/now behaves differently than /captures/resolved/now/
• [x] Chokes on having protocol in URL
  • This: /captures/resolved/now/https://youtube.com redirects to /captures/resolved/now/https:/youtube.com (note one slash after the protocol); this should probably either work fine or redirect to a version with the protocol removed
  • The redirect above is a 500 error (should probably be a 404; see also the note about /captures/resolved/now/ above) haven't managed to fix the 500 response code, but the rest is good
• [x] Would be nice to record the headers in the metadata can be accomplished with config.Worker.RecordResponseHeaders: true
  • [ ] Possibly replay this in the headers for raw captures (Wayback does this by prefixing original, replayed headers with X-Archive-Orig-) (I vote we punt on this until we need it, It'll take maybe 2 hrs)
• [x] Will need paging in short order (all done!)
• [ ] Ideally would have some other querystring-based filters, like time range (for the collection index) (would be nice, maybe let's open an issue to refine this?)

I've also added a feature I think is important for this phase: the capacity to fetch seeds from a file or URL. a new string configuratino property: config.Coordinator.SeedsPath lets you supply a string that's either a URL or a filepath (relative to pwd, or absolute) of a newline-delimited list of urls to seed. I've been using this to point at the raw text from @Mr0grog's gist: https://gist.github.com/Mr0grog/40cdcd56b048d7f00b0d47d3aca70be0/raw/c6ad8f6f55c93ab46b033d4486033b249c8b65db/webmonitoring_active_urls.txt

The biggest thing I'd like to tackle next:

• handling of redirects. currently not being recorded properly
• starting & stopping a crawl with the same local target leads to weird "not found" urls. I'd like to dig in on this & see if we need any stop/resume adjustments to our interfaces.

but all in all I think we should merge this & start into documentation & testing for the web monitoring use-case. Would love to hear if others agree

[b5]

Alright, after today's team call I'm re-fired up to take a run at saving this project. I'm going to merge this b/c we're already waaaaaay past master, and I can't write a proper "getting started" readme without doing some of the work we agreed needs doing first in today's call.