PubSub/Feeds

[parkan]

Using flooding for now

@whyrusleeping mentioned that some pubsub support was shipped in libp2p?

[vyzo]

No flooding needed, and we have persistence requirements. Plus it's really straightforward to implement the network parts with libp2p streams, we don't need anything special.

Overall approach:

• Feeds are advertised to the directory
• Feeds are subscribed by directly connecting to the peer, syncing and keeping a stream open for updates
• The synchronization happens with per peer timestamp, ntp is a reasonable dependency. It takes a simple query for messages by publisher id and timestamp > last peer sync time.
• When the publisher disconnects, the subscribers will keep polling with exponential backoff by looking up the peer in the directory.

[vyzo]

I think we want to limit the scope on this, as it is a big topic and not quite necessary for v1.

What we need instead for v1 is the query protocol (#22 ) and merging of remote data sets (#25 ). Couple this with published namespace announcements and we already have a usable system without the complexities of pubsub.

[vyzo]

I think we should implement it totally in RSS way, with polling and merging. There is no urgency for push publishing until we have real-time feeds to distribute.

[parkan]

That's fine by me. I think the most lightweight implementation is best right now, and poll + merge (or just merge on demand, really) is that

[denisnazarov]

Implementation suggested by Matthew Roberts on top of kad: https://github.com/kadtools/kad-quasar (paper)

Apparently storj uses it in production, described in this vid.

@parkan:

we'd need to port it to https://github.com/libp2p/js-libp2p-kad-routing

[denisnazarov]

Another thought... The nice thing about https://github.com/libp2p/go-floodsub is that it establishes the interface, so higher-level development can proceed with assumption the underlying implementation will be come performant at some point w/o worrying about details.

We can do something similar with polling..

[parkan]

@denisnazarov not sure the iface would be the same for non-flooding impl

[denisnazarov]

@parkan wanted to revisit this, since we've made lots of progress on mediachain foundations since opening this issue.

after speaking to @haadcode earleir, he suggested it would be fairly trivial to implement pubsub features from orbit-db to notification features to our namespaces.

goal is to implement this as a prototype to showcase the potential of the tech (not meant to be production grade), as well as to bridge our communities and show how mediachain and orbit-db/ipfs are collaborating, and that the sum of the technologies is something really cool

ideal demo would be something like a "decentralized instagram" where two people publish to the dinstagram.* and see their data update in real time.

also down to consider an api that uses polling under the hood and could be switched out for more robust pubsub in the future?

[parkan]

EDIT: mangled this during a stale edit, see below

[haadcode]

Hey,

I'll want to provide you a clear documentation on how ipfs-log works, both generally and in terms of linearization. I don't think I'll have time before Monday, so hopefully you can wait till then. However, if you're in rush to understand it, take a look at the open PR "immutable log" in haadcode/ipfs-log, entry points are append() and join(), play with the tests and use log.toString() to see what it outputs and how it behaves. Essentially ipfs-log sorts the forks on join deterministically, and upon join EventStore will update its index ("view") as a last write wins crdt. It's a partial order as opposed to absolutely order (for which consensus or coordination would be required in order to produce a sequence nro or similar)

Hope this helps! Looking forward to see what we can do and dinstagram sounds like an amazing demo to put our tech together! :)

Samuli

On Jan 13, 2017 00:01, "Arkadiy Kukarkin" notifications@github.com wrote:

Basically participating nodes can instantiate a corresponding orbit-db-eventstore https://github.com/haadcode/orbit-db-eventstore

Documentation on how they implement the linearization is sparse and last I saw it was based on attaching to last known head with no clear way to resolve forks, so need to read through the code to see how that works.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mediachain/concat/issues/13#issuecomment-272311633, or mute the thread https://github.com/notifications/unsubscribe-auth/AHJvruh0aWb5YCnUtTsSlgjBDg6G9l4Hks5rRrDRgaJpZM4J_F9- .

[parkan]

Ooops apparently I left these unposted somehow yesterday? @haadcode addressed this somewhat already so posting as a separate comment for the record

Basically participating nodes can instantiate a corresponding orbit-db-eventstore and ping it for each statement written to the corresponding namespace.

Unclear things:

• linearization (sounds like there's a deterministic join semilattice)
• connectedness/scaling: it seems like currently floodsub keeps complete (?) peer and topic lists on all nodes so there's (n/k)((n/k)-1)/2 edges, is that correct? at what point is this expected to fall over?
• channel creation/control/permissions: what does orbitdb.eventlog(foo.bar) actually do? I am guessing at this point anyone is allowed to write to anywhere and this just creates an entry in the topics map -- sounds like this is the biggest integration point

[parkan]

@haadcode thanks for chiming in, looking forward to exploring this together! No rush, let us know when you have a chance for a more in-depth technical discussion.