Coordinator doesn’t always automatically end when all work is done

Currently, running a crawl requires explicitly stopping walk when the crawl has completed by sending SIGINT with ctrl+c. It would be a huge improvement if the coordinator could reliably detect when all the items in its queue have been successfully processed by a worker, then stop.

The coordinator actually already has a structure for this: https://github.com/qri-io/walk/blob/5e8f3fe5e9da69d249b2c58a3a788c6722452dc5/lib/coordinator.go#L102-L128

(The right flags on a request in the RequestStore get triggered in the Completed() callback.)

But it has a few issues:

You have to explicitly configure DoneScanMilli otherwise it won’t run.
If the URL that was queued isn’t normalized, it won’t ever get marked as done. See this discussion item in #2 for more on that.
- There are a variety of ways to handle this, including normalize URLs on their way into the RequestStore.
- This also came up briefly in #13, but we decided to punt on it there and address it separately.
I’m not sure if we’ve done our full due diligence around making sure there aren’t other things we need to wait for.

We also need tests to cover this better :)

qri-io / walk

Coordinator doesn’t always automatically end when all work is done #14