Distributed Planning

[mikeal]

I'd like to use this ticket for planning the distributed session.

55 minutes isn't a lot of time :)

My gut feeling is that it's best to work so far outside of a normal person comfort zone that they don't try to think about problems like storing and manipulating data how they have in the past. With a more advanced topic like this I think people have a tendency to want to ease people in, but I find it's better if you just throw people in pool head first and see if they can swim :)

@dominictarr @substack @maxogden @spikeburroughs

I added more people to the above line than are going to be in the session cause I thought you might have something to add.

[dominictarr]

The crazy idea I've had since this morning is getting everyone to write a gossip protocol. This would encompass all the fundamental aspects of distributed systems.

• Intermittent connections
• Disorder (messages may arrive in any order)
• Merging changes on a new connection.

I'd use a few modules so that I could ignore things like streaming, and focus on creating a data structure that is replicatable.

The idea would be to have everyone implement a client that could connect to a server and emit heartbeats, and receive that same data - replicating the set of connected nodes.

The second phase would be introducing gossip - instead of connecting to a centralized server, nodes could connect to each other randomly. This would also require nodes to implement validation - so that they can reject invalid messages from other nodes, instead of crashing.

Then, we could add a chat feature or something like that, or something to turn it into a game.

[ghost]

We just need a few meta modules implementing each of the pieces that a good gossip protocol needs so that rolling your own custom gossip protocol is something that you can do in 15 minutes.

[dominictarr]

I think we already have most of those...

var reconnect = require('reconnect')
var duplex      = require('duplex')
var serializer  = require('stream-serializer')

reconnect(function (stream) {
  var d = duplex()
  d.on('_data', function (obj) {
    //handle incoming data.
  })
  //send object.
  d._data(obj)
  stream.pipe(serializer(d)).pipe(stream)
})

Well, that works for connecting to one server.

[mikeal]

right, just bear in mind a few things. any abstraction you take off the shelf you'll need to take the time to explain, so whatever collection of modules you use need to maintain a relatively low conceptual barrier so that people can follow and keep up.

also, wherever you leave off and give them a task, there needs to be room for them to tweak it and improve it for those that get it right away and finish quickly.

[mikeal]

adding @Raynos

[Raynos]

I like the idea of implementing gossip. When building distributed things, the thing I found the hardest to figure out is how do you know who to connect to and how do you replicate the list of people online. Once you can replicate a list of people the rest is way easier.

One idea would be focus on presence, everyone connects to something we run and figures out how to read the presence data and propagate their own presence data in. Now that they have the list of everyone's presence data they can connect to each other.

Half way through the session we would take our server down and it's everyone else's job to keep trucking in a distributed fashion among themself.

Once people get presence we can suggest they try to build a chat system, we should probably provide some GUIs they can plug into their server to make things more interesting

[dominictarr]

yeah, that was exactly what I was thinking. or maybe a terminal client, so that there is less messing around with browsers and pressing F5 and that crap.

[mikeal]

any new ideas of refinements on this?

[mikeal]

Could someone take the initiative and add the dependencies you're going to need for this session to the nodeconf2013 package.json?

[max-mapper]

just download all modules where the username in #stackvm also is a registered module author on NPM

On Fri, Jun 14, 2013 at 4:39 PM, Mikeal Rogers notifications@github.comwrote:

Could someone take the initiative and add the dependencies you're going to need for this session to the nodeconf2013 package.json?

— Reply to this email directly or view it on GitHubhttps://github.com/mikeal/nodeconf2013/issues/5#issuecomment-19487236 .

[Raynos]

I'm going to sink up with dominictarr and produce a plan this weekend.

[mikeal]

I want to send the email out on Wednesday, so all the deps need to be in prior to that.

[Raynos]

Rough plan, code sections very much subject to change.

Code sections:

• https://gist.github.com/dominictarr/204cbf77033d0de1ef00
• https://gist.github.com/Raynos/a8baa9b0d4c4d6ab8c0f

Initial outline

00:00 - 00:05

• Introduce distributed
• Talk about motivations
• Demonstrate chat

00:05 - 00:15 A simple chat client

Start with a chat server

var through = require("through")
var net = require("net")
var serializer = require("stream-serializer")()

var myIp = require("my-local-ip")()
var port = argv.port || 8000

var broadcast = serializer(through(function (chunk) {
    this.queue(chunk)
}))
var server = net.createServer(function (socket) {
    socket.pipe(broadcast, {
        end: false
    }).pipe(socket)
})

server.listen(port, myIp)

Create a chat client

var through = require("through")
var net = require("net")
var serializer = require("stream-serializer")()

var myIp = require("my-local-ip")()
var port = argv.port || 8000
var name = argv.name || "Anonymous"

var socket = net.connect(port, myIp)
var client = through(function (chunk) {
    console.log(">", chunk.ip + "@" + chunk.name, "::", chunk.message)
})

socket.pipe(serializer(client)).pipe(socket)

process.stdin.on("data", function (chunk) {
    client.queue({
        message: String(chunk),
        ts: Date.now(),
        name: name,
        ip: myIp
    })
})

Discuss serializer and through. Mention how broadcast serves as an echo server

We spawn up a server and attendees connect to us to chat. Those that are faster can spin up their own servers and talk directly to their friends / acquintances

00:15 - 0:30 Network partitions

Take down our central server. Demonstrate the value of reconnection and message history

var through = require("through")
var net = require("net")
var argv = require("optimist").argv
var serializer = require("stream-serializer")()

var myIp = require("my-local-ip")()
var port = argv.port || 8000
var name = argv.name || "Anonymous"

if (argv.server) {
    var broadcast = serializer(through(function (chunk) {
        this.queue(chunk)
    }))
    var server = net.createServer(function (socket) {
        socket.pipe(broadcast, {
            end: false
        }).pipe(socket)
    })

    server.listen(port, myIp)
} else {
    var client = serializer(through(function (chunk) {
        console.log(">", chunk.ip + "@" + chunk.name, "::", chunk.message)
    }))
    client.pause()

    process.stdin.on("data", function write(chunk) {
        client.queue({
            message: String(chunk),
            ts: Date.now(),
            name: name,
            ip: myIp
        })
    })

    connect()
}

function connect() {
    var socket = net.connect(port, myIp)

    socket.on("close", reconnect)
    socket.on("error", reconnect)

    socket.pipe(client, {
        end: false
    }).pipe(socket)

    socket.once("connect", function () {
        client.resume()
    })

    function reconnect(error) {
        socket.removeListener("close", reconnect)
        client.pause()
        console.log("reconnecting")

        setTimeout(connect, 1000)
    }
}

Talk about reconnection logic.

Talk about message history.

// Message history TODO

00:30 - 00:50 going distributed

• Show a mechanism for remembering who to connect to.
• Show a mechanism for gossip
• We take down our central servers. Attendees figure out for themself whom to connect to.

var through = require("through")
var net = require("net")
var argv = require("optimist").argv
var serializer = require("stream-serializer")()
var dirty = require("dirty")

var myIp = require("my-local-ip")()
var port = argv.port || 8000
var name = argv.name || "Anonymous"
var db = dirty("distributed.ips")

var messages = {}

function messageId(chunk) {
    return chunk.ip + "~" + chunk.entropy + "~" + chunk.ts
}

var broadcast = serializer(through(function (chunk) {
    this.queue(chunk)
}))
broadcast.setMaxListeners(Infinity)
var server = net.createServer(function (socket) {
    socket.pipe(broadcast, {
        end: false
    }).pipe(socket)
})

server.listen(port, myIp)

var client = serializer(through(function (chunk) {
    if (!db.get(chunk.ip + ":" + chunk.port)) {
        db.set(chunk.ip + ":" + chunk.port, true)
    }

    var mid = messageId(chunk)
    if (messages[mid]) {
        return
    }

    messages[mid] = true

    console.log(">", chunk.ip + "@" + chunk.name, "::", chunk.message)

    this.queue(chunk)
}))
client.setMaxListeners(Infinity)

process.stdin.on("data", function write(chunk) {
    client.queue({
        message: String(chunk),
        ts: Date.now(),
        name: name,
        ip: myIp,
        port: port,
        entropy: Math.random()
    })
})

db.on("load", function () {
    db.forEach(function (key, port) {
        var parts = key.split(":")
        connect(parts[0], parts[1])
    })
    connect(myIp, 8000)
})

function connect(ip, port) {
    var socket = net.connect(port, ip)

    socket.on("close", reconnect)
    socket.on("error", reconnect)

    socket.pipe(client, {
        end: false
    }).pipe(socket)

    function reconnect() {
        socket.removeListener("close", reconnect)
        // console.log("reconnecting", ip + ":" + port)

        setTimeout(function () {
            connect(ip, port)
        }, 1000)
    }
}

[mikeal]

gentle reminder that all dependencies and materials you want on people's computers needs to be checked in to the package in this repo by tomorrow.

[max123]

Will everyone have access to the node.js documentation (i.e., can we have http://www.nodejs.org/api/all.html as one of the things on people's computers)?

On Jun 18, 2013, at 1:53 PM, Mikeal Rogers notifications@github.com wrote:

gentle reminder that all dependencies and materials you want on people's computers needs to be checked in to the package in this repo by tomorrow.

— Reply to this email directly or view it on GitHub.

[mikeal]

sure, I could just get the current tarball and extract it in to the directory.

[max123]

that should work. I just don't want to find that I need to look at documentation and not be able to get to it.

thanks, max

On Jun 18, 2013, at 2:05 PM, Mikeal Rogers notifications@github.com wrote:

sure, I could just get the current tarball and extract it in to the directory.

— Reply to this email directly or view it on GitHub.