Usage with Hyperswarm

[jankaszel]

Hello! I'm attemping to use hypermerge with Hyperswarm (reasons for using Hyperswarm are discussed here: https://pfrazee.hashbase.io/blog/hyperswarm). It is possible, though I had to apply two small patches:

• While discovery-swarm defaults announce to be true, Hyperswarm doesn't. I don't know whether this inconsistency is intended by Hyperswarm, but providing swarm.join(..., { announce: true }) within RepoBackend.replicate() and RepoBackend.join() does solve this.
• I can't make replication patches be properly applied to documents as doc.synced isn't changed to true, and I don't know why. However, when I manually set it to true, patches via DocBackend.subscribeToRemoteChanges() are properly applied to the document.

I'd be glad doing a PR fixing the discovery behavior, but I cannot imagine any fix for the doc.synced issue.

If somebody's interested in reproducing the documented behavior, I created an example for using Hyperswarm with hypermerge in this Gist: https://gist.github.com/falafeljan/8108fb350fd47752343f3a2901fa270e

[pvh]

This is an excellent result -- I've wanted to try Hyperswarm with Hypermerge for a long time and haven't got around to figuring out what that would entail so this work is much appreciated.

I'm not sure what's up with the doc.synced issue, but I can take a look. Would you mind typing up some steps to reproduce so that it's easy for me to be sure I'm hitting the same issues that are affecting you?

[pvh]

Also, gee, I was wondering why you hadn't replied to my comment and it's because I hadn't clicked submit! I'm excited about what you're doing and looking forward to working with you on it.

[mjtognetti]

Cool! Excited to see this.

doc.synced getting stuck as false could be an issue with RemoteMetadata extension messages not being broadcast to peers (in DocumentBroadcast.ts).

RemoteMetadata messages contain a clock which communicates a known good clock state of a doc. DocBackend will compare its current clock with a remoteClock to determine if the local document is caught up with what its remote peers are saying is a good clock state. If you aren't receiving this message the backend will never set the synced value to true.

This is a little awkward and hypermerge can make it hard to tell why it's failing.

@falafeljan can you try to log or otherwise monitor extension messages in DocumentBroadcast.ts and see if they're being sent and received?

[jankaszel]

I'm glad the results are of some worth! I see that #31 pushed some major changes which aren't yet described in the docs, so I'll need to figure out where repo.replicate moved to.

[jankaszel]

I pushed a fix for swarm announcements to my fork: https://github.com/falafeljan/hypermerge/tree/debug/hyperswarm.

This leaves us with the sync flag. @mjtognetti, thanks for the input! I logged for messages at DocumentBroatcast.ts (within broadcast and parseMessage)—there are plenty of broadcasts from both peers, but no incoming messages have been received. Here's one of the broadcasts, if that's of any help:

{
  type: 'RemoteMetadata',
  clocks: { GryE64bxF1wTnSb13vYbD28tbpJFdw5rxkSpmUZ9PXrh: {} },
  blocks: [
    {
      id: 'GryE64bxF1wTnSb13vYbD28tbpJFdw5rxkSpmUZ9PXrh',
      actors: [Array],
      merge: {}
    }
  ]
}

PS: I debugged hypercore-protocol a bit, and extension messages are actually received and passed on via this.peer.emit('extension', ...) here: https://github.com/mafintosh/hypercore-protocol/blob/ffb89cd5e6ff453b9f2335737e6c8f6a4436325a/feed.js#L152

Could it be a difference in the implementation of peers between discovery-swarm and Hyperswarm?

[jeffpeterson]

Hyperswarm support has shipped!

repoBackend.setSwarm(Hyperswarm())

Edit: Shipped on master, we'll be releasing a new package version soon.