search

tristanls / gossipmonger

Gossip protocol endpoint for real-time peer-to-peer replication
MIT License
51 stars 10 forks source link

readme

gossipmonger

Stability: 1 - Experimental

NPM version

Gossipmonger is an implementation of the Scuttlebutt gossip protocol endpoint for real-time peer-to-peer state distribution with pluggable storage and transport mechanisms.

Contributors

@tristanls, @KenanSulayman

Usage

var Gossipmonger = require('gossipmonger');

var gossipmonger = new Gossipmonger(
    { // peerInfo
        id: "localId",
        transport: { // default gossipmonger-tcp-transport data
            host: "localhost",
            port: 9742
        }
    },
    { // options
        seeds: [
            {id: "seed1", transport {/*...*/}},
            {id: "seed2", transport {/*...*/}},
            {id: "seed3", transport {/*...*/}}
        ]
    });

gossipmonger.on('error', function (error) {
    console.dir(error); 
});

gossipmonger.on('new peer', function (newPeer) {
    console.log("found new peer " + newPeer.id + " at " + newPeer.transport);
});

gossipmonger.on('peer dead', function (deadPeer) {
    console.log("peer " + deadPeer.id + " is now assumed unreachable");
});

gossipmonger.on('peer live', function (livePeer) {
    console.log("peer " + livePeer.id + " is live again");
});

gossipmonger.on('update', function (peerId, key, value) {
    console.log("peer " + peerId + " updated key " + key + " with " + value);
});

/* **IMPORTANT**
 * Typically, one would create a `transport`, start it (call listen())
 * and then pass it in as `options.transport` in Gossipmonger constructor. This
 * makes the implementation of Gossipmonger less complex and simpler.
 * For development purposes, Gossipmonger comes with a default transport, so
 * it's easier to get a feel for it, but because of that, if you don't provide
 * a `transport`, the default one will be used but **you need to start it**.
 * The call illustrated below will start the default transport. If this isn't done,
 * you will not receive communications from other gossipmongers. */
gossipmonger.transport.listen(function () {
    console.log('default transport is listening');
});

gossipmonger.gossip(); // start gossiping

gossipmonger.update('foo', 'bar');
// this node's foo/bar key value pair will now be gossiped 
//  to the rest of the cluster

Tests

npm test

Visual Test

To watch a cluster of 5 nodes communicate via gossip run scripts/locatest.js via:

npm run localtest

Overview

Gossipmonger is an implementation of the Scuttlebutt gossip protocol endpoint for real-time peer-to-peer peer-state distribution. Gossip protocols are used in a decentralized peer-to-peer manner in order to make every peer that is connected aware of the state of every other peer. The objective is to give every peer global awareness without a centralized server. This is accomplished by heuristically guided message passing between peers.

Peers

Gossipmonger manages information about peers via maintaining peer information in a structure called a peer. A peer stores the details of a particular peer on the network, including the data necessary to estimate whether the peer is "alive" or "dead".

Peers are implemented internally as JavaScript objects, and most details are not necessary to be exposed, however when creating a new Gossipmonger, the following are required to be provided as peerInfo object:

All peer attributes are as implemented in gossipmonger-peer:

Digests

Digests are passed around between peers to communicate what they know about all other peers and the latest version they have seen of those peers.

A digest is an array of peer objects, for example:

[ 
    {
        id: "peer1", 
        maxVersionSeen: 1732, 
        transport: {
            host: "peer1.host", 
            port: 9742
        }
    },
    {
        id: "peer2", 
        maxVersionSeen: 1432, 
        transport: {
            host: "peer2.host", 
            port: 9742
        }
    }    
]

Deltas

Deltas are passed around between peers in response to receiving a digest to update any information that (from the senders perspective) is out of date.

Deltas are an array of delta objects, for example:

[
    ["peer1", "foo", "bar", 1732],
    ["peer1", "bas", "baz", 4322],
    ["peer2", "far", "blh", 422]
]

Documentation

Gossipmonger

Public API

new Gossipmonger(peerInfo, [options])

Creates a new Gossipmonger instance.

The seeds are necessary in order to bootstrap the gossip cluster. Gossipmonger will use these seeds to find out about other nodes and also as peers of last resort if all the peers appear to be dead.

IMPORTANT: If no transport is provided in options, then the default gossipmonger-tcp-transport will be used. However, it needs to be started by explicitily calling gossipmonger.transport.listen() (see Usage). If this does not happen, no communications can be received from other Gossipmongers and they will think this instance is dead.

NOTE: "Why do I have to excplicitly start the transport?". This way, you can create a Gossipmonger instance without worrying about servers being started on the network. It makes things much easier for testing. It also gives you the power of sequencing actions and when you want to start interacting with the outside world.

gossipmonger.digest(livePeers)

CAUTION: reserved for internal use

Creates a digest of peers that are thought to be "live" to send to another peer.

gossipmonger.gossip()

Initiates gossip and will continue to gossip according to GOSSIP_INTERVAL.

The implemented algorithm does the following in order:

  1. Select a random live peer (if any) and send my digest to the peer.
  2. Maybe send my digest to a random dead peer (or do so for sure if all peers appear to be dead).
  3. If number of live peers is below MINIMUM_LIVE_PEERS send my digest to a random seed.
  4. Update my estimate of the liveness of all live peers.
  5. Update my estimate of the deadness of all dead peers.
  6. Set timeout to gossip again GOSSIP_INTERVAL from now.

gossipmonger.update(key, value)

Updates the local peer's key with specified value.

Event deltas receive

CAUTION: reserved for internal use

Emitted when Gossipmonger receives deltas from a remote peer.

Event deltas send

CAUTION: reserved for internal use

Emitted when Gossipmonger sends deltas to a remote peer.

Event digest receive

CAUTION: reserved for internal use

Emitted when Gossipmonger receives a digest from a remote peer.

Event digest send

CAUTION: reserved for internal use

Emitted when Gossipmonger sends a digest to a remote peer.

Event error

Emitted when Gossipmonger or one of its dependencies emits an error. If no handler is registered, an exception will be thrown.

Event new peer

Emitted when Gosspimonger becomes aware of a new peer.

Event peer dead

Emitted when Gossipmonger assumes that a live peer is now dead.

Event peer live

Emitted when Gossipmonger assumes that a dead peer is now live.

Event unknown peer

CAUTION: reserved for internal use

Emitted when Gossipmonger receives deltas for an unknown peer. (It shouldn't happen).

Event update

Emitted when Gossipmonger is aware of a key update on a remote peer.

Gossipmonger Storage

Modules implementing the storage mechanism for Gossipmonger shall conform to the following interface. A storage is a JavaScript object.

Storage modules shall rely on peer.live property to keep track of peer liveness for the purposes of deadPeers() and livePeers() results.

Storage modules shall treat all peer properties as immutable.

Storage implementations shall allow registering and interacting with event listeners as provided by events.EventEmitter interface.

For reference implementation, see gossipmonger-memory-storage.

Gossipmonger Storage API

storage.deadPeers()

storage.get(id)

storage.livePeers()

storage.put(id, peer)

Event error

Emitted when Storage encounters an error. If no handler is registered, an exception will be thrown.

Gossipmonger Transport

Modules implementing the transport mechanism for Gossipmonger shall conform to the following interface. A transport is a JavaScript object.

Transport implementations shall ensure that deltasToSend and digestToSend will be unaltered.

Transport implementations shall ensure that localPeer.id and localPeer.transport are sent to the remote node unaltered.

Transport implementations shall allow registering and interacting with event listeners as provided by events.EventEmitter interface.

For reference implementation, see gossipmonger-tcp-transport.

Gossipmonger Transport API

transport.deltas(remotePeer, localPeer, deltasToSend)

Sends deltasToSend to the remotePeer.

transport.digest(remotePeer, localPeer, digestToSend)

Sends digestToSend to the remotePeer.

Event deltas

Emitted when Transport receives deltas from a peer.

Event digest

Emitted when Transport receives digest from a peer.

Event error

Emitted when Transport encounters an error. If no handler is registered, an exception will be thrown.

Available Modules

Storage

Transports

Sources