Expose information about replication progress

[hackergrrl]

We (Digital Democracy, in this context) work with partners who run Mapeo on older/commodity phones and laptops, and will sync hypercore feeds (via kappa-core) using hypercore's replication API ({live:false}).

Sometimes there is a lot of data to be synced between devices, and fresh syncs can take over ten minutes. Right now we show a spinner, but it would be very useful if we could expose overall progress (N/M uploads done; X/Y uploads done). This would let us potentially show a rough time estimate, and, more importantly, make it clear to the users that the app has not frozen, or that the sync has stalled.

Right now I've hacked something on top of kappa-core that tracks overall download progress, by iterating over all hypercores and comparing feed.length and feed.downloaded(0, feed.length). This is decent, but it doesn't track uploads, and the totals are across all entries, not per the sync session.

Re: API, maybe something like

var stream = feed.replicate({live:false})
stream.pipe(socket).pipe(stream)

stream.on('progress', function (info) {
  console.log(info)
})

outputs data like

{
  downloadSofar: 7,
  downloadTotal: 80,
  uploadSoFar: 0,
  uploadTotal: 700
}

cc @rangermauve @mafintosh

[RangerMauve]

Would an API on the Peer object work?

For example:

feed.on('peer-add', (peer) => {
  peer.on('progress', (info) => console.log(info))
})

Some questions I have is how we can tell what blocks we can download from a peer and what event we should be listening on to get progress for that.

I think if I can get pointers to how to find that data it shouldn't be too hard to add in. 😁

[RangerMauve]

Related progress from noffle on this: https://github.com/noffle/hypercore-progress/blob/master/index.js

[hackergrrl]

Update: hypercore-progress works with hypercore@8. However, only in non-sparse mode. I haven't tested it in live mode, but it might work ok. Overall I think it was a good experiment, but it relies on undocumented APIs and doesn't feel too reliable.

Re sparse mode, my understanding of it is that peers don't always say "I'm planning to request these specific blocks" ahead of time, they often say something more like "I'm interested in this wide range of blocks. tell me what you have, and I'll request specific blocks as I want them". I'm not sure if/how progress can really be measured in this case. It's possible that things like progress in sparse mode might be best left to userland extensions, which are more aware of each app's intentions, and could send more specific information to peers about what it wants & has.

[mafintosh]

@noffle yea, i think progress only makes sense for non sparse hypercores. fx in hyperdrive being 100% synced means a lot things depending on the mode you are using. i.e. 100% of the latest snapshot or 100% of all history.

[mafintosh]

Closing cause old and lots have changed (and there are some new upload/download stuff in 10) - Feel free to open a new issue if relevant