Support sparse hypercore (sync + indexing)

[hackergrrl]

Right now the mapeo stack doesn't handle sparse data well, which can occur in the case of a sync that is prematurely terminated. What happens is

1. sync no longer finishes (this is worked around right now using a ~20s timeout + no data seen on the sync stream).
2. multifeed-index stops indexing a feed if it hits a set of N sequence numbers that have missing blocks. It will resume if/when that data is available, but it works sequentially right now, so if, say block 0 were missing, no indexing on that feed would happen, even if later sequence numbers were available.

I'm thinking of some fixes for this:

• [ ] Introduce an opportunistic sync mode in mapeo-core (or maybe multifeed?), which uses peer.remoteBitfield on the remote hypercore(s) to figure out what entries each remote feed has, and request only those. Done in sparse mode, this will let sync properly terminate.
• [ ] Update multifeed-index to maintain a bitfield (not unlike hypercore does) of what's been indexed, so that subsequent runs of the indexer will know where it left off. This will let the indexer actually index what's available and handle holes in the data ok.

[okdistribute]

I think some of this is what @telamon is attempting to address with https://github.com/decentstack/.

[okdistribute]

see: https://github.com/decentstack/decentstack/blob/master/docs/middleware_interface.md

[telamon]

Wow @okdistribute , yes! Im a bit surprised you dug up the prerelease docs! But yes i hope to provide an extra layer of control to applications :)

[okdistribute]

Pulling this over from #96, by @noffle

As a future reference for what a more complete fix might look like:

• [x] backport https://github.com/kappa-db/multifeed/pull/46 to the version of multifeed we use (pre 5.x.x I think?)
• [ ] pass down opts.sparse = true all the way down the stack to hypercore (kappa-osm -> kappa-core -> multifeed -> hypercore)
• [ ] figure out how to know when the multifeed handshake is done & we have all feed refs (I added this to 5.x.x, we'll need to backport)
• [ ] figure out what blocks the remote feeds have
• [ ] modify mapeo-core to issue core.download() requests for only those blocks
• [ ] remove heartbeat ;)