JRFC 30 - Network Sim Vis Tool

[jbenet]

This is cross posted from: https://github.com/jbenet/ipfs/issues/39 Let's use this as discussion

---

(( I posted this on some email lists, but figure the people watching this repo may be able to help ))

We're making a (javascript - d3) visualizer of protocol traces to observe, debug, learn, ... admire... networks. Our main use case is for IPFS (ipfs.io, github.com/jbenet/ipfs) and Filecoin (filecoin.io). Has anybody here seen something like this? Rough scheme:

• protocols output traces following a specific format detailing nodes + events
• protocols can optionally specify viz layouts (e.g. DHT address ring vis, or force graph (based on latency))
• d3 program consumes traces (could be live streams) in json and runs the simulation
• simulation playback controls (play/pause/ffwd/rwind/speed)

Also, we are not experts on d3, so if you're interested in creating a general enough trace visualizer tool applicable to lots of other protocols, I'm open to paying people to do it. (Will be open source MIT licensed).

Thanks, Juan

@juanbenet github.com/jbenet

[jbenet]

moving over this comment for links

---

@zignig yeah, we're looking for a graphical network representation. It can be force directed, though doesn't have to be. You san see a simple example of a network here: http://bl.ocks.org/jbenet/8db0cb6160f505acc12a (that's from an ipfs dht test network). But what we also need is something like raft's vis: http://raftconsensus.github.io/ where you can see messages flow, and the protocol's state changes. (something in between, where a flexible graph layout shows connections, and messages flowing through those edges).

[dborzov]

@jbenet, @whyrusleeping, @cryptix how do you envision the most basic minimum prototype for such a tool that would still be useful for you guys? What would be a good workflow for it from your side?

Also, could you point me towards the API endpoints within go-ipfs to collect all this data on the state of network to visualize? What about the public nodes, like mars.i.ipfs.io?

[jbenet]

@dborzov the example above is generated using @whyrusleeping's https://github.com/whyrusleeping/dhtHell -- we need to setup a proper message / packet format that makes sense to visualize lots of different kinds or protocols. I want to be as general as I can with this tool, so that it can be used by a ton of other projects.

[cryptix]

i really wonder what that raftscope vis is build in. Seems to be very custom (only libs i see are bootstrap and jquery) because it also understands raft itself it seems (you can drop individual votes/packets on the fly).

Does anybody know of an d3 example that couples a network graph of any kind with a time slider for playback control to enable introspection of message flow? I couldn't find any (yet - maybe google filter isn't tuned right today).

I found vis.js which seems to be very similar to d3 but being more frameworky and this example that plays GPS traces. If the time slider can be applied graph data and events traveling on their edges, this might be an option as well?

[jbenet]

@dborzov you're right in tackling to format first, it's the hardest part to get right! I'm thinking something pretty general that we can turn protocol traces into, that captures all the needed information. We can have the tool process the traces to generate other intermediate formats as needed.

Basically, i'm thinking of a data transform pipeline like this:

Program running the Protocol
-> output trace (possible in program/protocol specific format) of messages and state changes
   ^----- within program -----^

-> transform protocol trace into our tool's import format

   v----- within this tool ------v
-> transform into a "visualization" trace, including initialization state, vis layout, and sequence of events
-> add all this to an html page to run it

---

For starters (using Go, for types, but imagine this in json, or whatever):

type Message struct {
  // protocol id in string (e.g. "bittorrent", "ipfs", "http")
  Protocol string

  // distinct message type within protocol
  Type string

  // the state this msg carries (basically arbitrary json object)
  State map[string]interface{}
}

[jbenet]

@cryptix vis.js looks good. And, i dont think raftscope uses much, i think @ongardie built most of it from scratch. @ongardie thoughts?

(( And @dborzov the d3-based charting tool i mentioned is http://plottablejs.org/ -- maybe can use it to visualize some of the aggregate data ))

[ongardie]

@cryptix @jbenet: I only had a couple of days to build the initial version of RaftScope. I talked to @benbjohnson about his experience building http://thesecretlivesofdata.com/raft/ , who told me the learning curve on d3 was steep. I chose to take the predictable struggle of manipulating SVG directly with no abstractions, rather than risk spending time on d3 only to possibly learn later on that it didn't do what I wanted. After a few days, I produced a mess of code that works well for my purpose (talks on Raft), but there's no way that code would generalize to anything else. By the way, I use a pre-recorded form of RaftScope for my talks now, which you can find at https://ramcloud.stanford.edu/~ongaro/raftscope/ at the moment.

What's the right library? To this day, I still haven't invested any time in learning about them. I can say that working with SVG directly isn't that bad, but you'll need to think carefully about building the right abstractions (more so than I did, especially if you're worried about CPU usage).

@benbjohnson, thoughts?

[whyrusleeping]

This will also require hook in ipfs to provide the data, We should start an issue for an 'events service' type thing that can publish events "Node 1 sent a packet AF2B to Node 3"

[benbjohnson]

I would definitely suggest investing the time into learning D3.js. It really gives you a lot of flexibility and power.

Force directed layout is once way to go but I personally dislike its jitteriness. There's a pure JavaScript graphviz implementation called dagre that probably fits your use case better.

As far as a slider goes, D3 has a built-in construct called a "brush" that you can use. Here's an example that lets you draw/resize/move the brush:

http://bl.ocks.org/mbostock/1667367

It sounds like you wouldn't need to size the brush -- just slide it back and forth over a timeline.

[jbenet]

I would definitely suggest investing the time into learning D3.js. It really gives you a lot of flexibility and power.

I would agree with this.

There's a pure JavaScript graphviz implementation called dagre that probably fits your use case better.

Oh man, thank you! I really wanted this 2 years ago!! the internet always delivers... eventually. :) (i should've just built it)

@benbjohnson looks like you're interested in the same things we are (http://thesecretlivesofdata.com/raft/ is beautiful). Would you be interested in working with us on this? I can pay you for your time and all code would be MIT / Apache 2 / similarly permissive license.

[benbjohnson]

@jbenet It sounds like a fun project but I honestly don't have any time available right now. Kudos for paying for open source dev work though. Not enough of that happening in the industry, imo.

Feel free to ping me with any questions. I'll try to help out where I can.

[dborzov]

@jbenet, I see what you mean, that is probably a good start. Thanks everyone for suggestions.

Here is one probable starting point for the design for this tool as I see it. Please let me know what you think:

• It can follow the MVC(Model-View-Controller) pattern. The model part would store the state of the network with instances of Network Nodes, Messages/Packets they exchange, Connections and the Time Arrow to resolve and depict Events. So that we get:

     > Prismo.Nodes.get("mars.i.ipfs.io")
     {
             name: "mars.i.ipfs.io",
             ip: "localhost",
            port:"8080",
            protocol:"HTTP 1.1"
     }
     > Prismo.Packets.get("df45-34fg")
     {
        "header":"GET /hi",
        "body": "is there anybody in there?"
     }

or something like that.

• The Controller Part would include parsers for the network traces that would update the model state accordingly. So you can feed it JSONs from external requests and see the model change accordingly. Or the user can interact with the visualization using page's UI and these actions would be converted into the same message JSON format and fed to the controller (it will follow Message format @jbenet suggested above). Something like that:

   var event = {
       "type": "packet sent",
       "protocol":"HTTP 1.1",
       "recepient_ip":"localhost:8080",
       ...
   };

• The View part would render model into the d3.js-based visualization. It would include all the styling, selections and UI part handlers.

I think the advantages of this approach are:

• The 3 separate domains (model, controller, view) are logically separate. So you can easily interchange them to support anyhting else (like you can, say, rewrite View for shell and run if off Node.js)
• easy generalization for various protocols (you just add parsers to the controllers to include your format of network's output traces)

[jbenet]

@dborzov https://gist.github.com/jbenet/41092a3a55c44b9ed2c5

[max-mapper]

@maurizzzio has a cool project w/ d3 and threejs renderers http://maurizzzio.github.io/PojoViz/public/vulcanize.html#render/jQuery

@maurizzzio can you use pojoviz with requirebin e.g. http://requirebin.com/?gist=4775854a3b6cbe0ac654

[mauriciopoppe]

@maxogden unfortunately pojoviz relies on browserify transform and browserify-cdn doesn't support it yet https://github.com/jfhbrook/browserify-cdn/issues/84, however pojoviz has a playground page which I used to explain how pojoviz works, that should work in the meantime