search

libp2p / research-pubsub

Research on PubSub algorithms for libp2p
https://github.com/libp2p/specs/tree/master/pubsub
MIT License
76 stars 6 forks source link

Last Value Cache #12

Open dignifiedquire opened 7 years ago

dignifiedquire commented 7 years ago

In the search for a solution to Orbits problem of how to know about the current state of a chat when joining, we had the idea of adding a last value cache to the floodsub implementation.

This [last value caching] solves the problem of how a new subscriber catches up when it joins the network. The theory is that publishers get notified when a new subscriber joins and subscribes to some specific topics. The publisher can then rebroadcast the last message for those topics.

From http://zguide.zeromq.org/php:chapter5#Last-Value-Caching

I have an experimental implementation for js up here: https://github.com/libp2p/js-libp2p-floodsub/pull/32 which caches the last known message for all known topics currently. This definitely needs some refinements, to avoid things like unlimited memory usage.

My first tests with orbit showed that this does solve the issue of message loading on join, without any changes to the codebase.

mitar commented 7 years ago

Interesting. This was in fact exactly the reason why I was looking into IPNS. I wanted to use IPNS to store the last value. My idea was that each subscriber would be storing there the last message they know about, so that they could resume the next time.

What happens if all subscribers are offline, then last value cache disappears, no? So if then they reconnect, they still cannot get the last value? With IPNS they could, no?

whyrusleeping commented 7 years ago

@mitar correct, with ipns they could

dignifiedquire commented 7 years ago

Yes with IPNS they could, but adding IPNS caching requires a good deal of work on the application layer and js-ipfs does not even have IPNS implemented currently.