Implementing a clean API for interactive plotters

[JobLeonard]

Design document for moving towards interactive plots.

I've been looking at other plotting libraries (see #15 ). They all fall short in a one or more of the following ways:

• They make too many assumptions about what type of charts we want, not allowing for enough customisation and making it hard to integrate the interactivity with the rest of the app.
• We have two main bottlenecks in our code: memory access and rendering speed. Most plotting libraries fall short in especially the former
  • To expand on the previous point, almost all use a data model with arrays of objects, where each object contains various pieces of (meta)data. The performance benefit of switching from boxed values to monomorphic typed arrays is so great that I am not going to give that up.
  • many are focused on charts with fancy animations, aiming for 60fps repaints every second even when nothing is happening. I think it's better to minimise repaints to save energy usage and improve performance ("a powerful laptop/tablet during a train-trip with flaky internet access" is still my baseline)
• They don't play nicely with React
• They do too much, which leads to code bloat.

So, after thinking about it I suspect the best way to move from here is extending our current setup of Canvas + plotter functions to one that is interactive.

One issue to cover is that the canvas (or more precisely the drawing context) is inherently stateful: every pixel is a state, after all. Meanwhile, Redux wants to keep as much state to itself, and so do react components. So we need to bridge this sharing and communication of state between Redux, React and the canvas. Furthermore, for interactivity the canvas needs to be able to respond to mouse- and keyboard events (and eventually touch events would be nice too).

The best solution, I think, would be to treat the drawing surface as its own little world, handling its own mouse- and keyboard- events and related state, which only sparsely interacts with the world outside through the occasional message: props being passed on, and .

On a personal level, I am really fond of the Processing API, partially because I'm really familiar with it, partially because it feels like an appropriate mix of low-level drawing/event primitives and convenience functions while still abstracting a lot of boilerplate away. However, p5js (the JavaScript version of Processing) doesn't play nicely with React and does things we don't need (video capture, sound recording, etc).

Here's an example p5js sketch:

// Node is either a canvas element or any generic element.
// If it is a canvas, P5 will attach to it.
// If it is another type of element, a canvas with 
// P5 attached will be inserted inside of it.
// Note that "sketch" is arbitrary and a user 
// may replace it w/ any variable name.

var sketch = function (p) {
  var gray = 0;

  p.setup = function () {
    p.createCanvas(600, 400);
  };

  p.draw = function () {
    p.background(gray);
    p.rect(p.width/2, p.height/2, 200, 200);
  };

  p.mousePressed = function () {
    gray = (mouseX + mouseY) % 256;
  };
};

new p5(sketch, node);

So the way this works is actually really clever: our sketch is a function. It is passed a p object. The p object can hide away the drawing context with a number of convenience functions like background or rect, and keep track of "global" variables like mouseX and mouseY, etc. It's pretty similar to how I already use context in the plotters.

The sketch then assigns a number of closures to pre-determined fields (setup, draw, mousePressed). Then this sketch function is passed to the p5 constructor, alongside a node. The p5 constructor creates the p object from the node, calls sketch, and then lookst at the resulting p object. For each defined event handler function (like mousePressed) it attaches the right event, making sure it doesn't bubble, gets removed upon unmounting, etc. So we only have to define the handler functions for the events we care about, and p5 takes care of the rest.

Furthermore, because of how we use closures, we also don't have to mess around with the this keyword (bet you love reading that, @slinnarsson); defining top-level var (or in our case, const or let) in the sketch will do, because they can be captured by the closures. In the above example, gray is shared by draw and mousePressed.

So the result of all this is that the sketch can be relatively clean, boilerplate-free code.

Another benefit of this sketch approach is that it decouples the code responsible for drawing on the canvas from React, like the painter approach I already implemented. So if we ever switch to another framework, all we have to to is re-implement this minimal API and we can re-use all of our plotters. Similarly, if we ever want to use WebGL instead of the Canvas API for rendering, we could implement this sketch API as a WebGL wrapper and again keep our existing plotters.

So my plan is to implement a similar API, but optimised and slimmed down for our use-case. It's less work that it would be to handle attaching mouse- or keyboard- events manually for ever plotter, and the resulting simplifications in the plotters will save us development- and maintenance time in the long run.

Initially, the following "global" variables and convenience functions will be implemented:

• Context

  • width int
  • height int
  • props object (the React props object, a new addition compared to the p5 API)

• Styling

  • font()
  • fontSize()
  • textStyle() (shorthand for setting everything at once, like we already have)
  • stroke(), noStroke()
  • fill(), noFill()

• Drawing

  • text()
  • circle()
  • rect()
  • image()

As well as these event handler function:

• setup() (run once during initialisation, but after the context to draw on has been defined - although calling the sketch function itself is also an initialisation step, any variables created at the top-level could be captured by other closures. So the setup function exists to ensure we don't create any accidental "global" variables in our sketch)
• draw() (the main drawing function, obviously. Does NOT animate by itself! Has to be triggered in event functions with redraw)
• redraw() (wrapper for window.requestAnimationFrame(draw), for triggering async, debounced redrawing in other events)
• newProps() (a handler to be called from the outside component when passing in new props - note that it does not automatically redraw)

This is enough to migrate our existing plotters, which should require minimal work - it's just a matter of copying the existingpainter code into the draw function, with minor refactoring that should even simplify the code.

Note that communication between the canvas and the rest of React and Redux is pretty simple too: props are passed down, and if a dispatch() function is included we can dispatch to Redux.

From here we could enhance the API with interactivity.

• Mouse

  • mouseX, mouseY int (current mouse coordinates)
  • pmouseX, pmouseY int (previous mouse coordinates)
  • mouseButton LEFT, MIDDLE, RIGHT
  • mouseIsPressed boolean
  • mouseMoved()
  • mouseDragged()
  • mousePressed()
  • mouseReleased()
  • mouseClicked()
  • mouseWheel()

• Keyboard

  • keyIsPressed boolean
  • key, keyCode char, int (most recently pressed key/keyCode)
  • keys (hashmap of which keys are currently pressed (for handling key-combinations))
  • keyPressed()
  • keyReleased()
  • keyTyped()
  • keyIsDown()

So I think this is the best way forward: we get to keep the code we already created for the plotters, even have a chance to clean things up due to a more convenient drawing API, and can then enhance them with interactivity.

I'll first implement the new data structure outlined in #75, then implement the paste genes enhancement and material UI migration of #73 (which also should implement the slider-controls for dot-size, for example). Then I'll get back to this.

[JobLeonard]

When I looked last year, Pixi.js was still too difficult to combine with React, the framework itself seemed more oriented towards interactive games, and the documentation needed a lot of work.

All of that has been improved, and I'm more comfortable with JS myself. So I think wrapping Pixi.js in the above API seems like a good strategy to get WebGL acceleration + interactivity with relatively little developer work on my side.

http://www.pixijs.com/

Combining it with React: (not that different from my current Canvas solution) http://www.rinconstrategies.io/using-react-and-pixijs.html

Note: this demo is for constantly updating game. We're not aiming for 60FPS animations, so the update code would be different to minimise redraws.

Their bunnymark demo has code for blitting a small set of sprites thousands of times. This can be used to speed up the scatterplot.

https://github.com/pixijs/bunny-mark

Z-order demo: (necessary to maintain current "sort by X/Y" strategy for scatterplots)

http://pixijs.github.io/examples/#/display/zorder.js

Pixi.Graphics: (contains lines (for edges) and rectangles (for sparkline plots). Similar to Canvas API) http://pixijs.download/dev/docs/PIXI.Graphics.html

Caching: (for combining scatterplots with edges)

http://pixijs.github.io/examples/#/demos/cacheAsBitmap.js

[JobLeonard]

https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API/Tutorial/Getting_started_with_WebGL

https://webglfundamentals.org/

http://webglsamples.org/

http://webglsamples.org/sprites/readme.html

http://webglsamples.org/lots-o-images/index.html

https://github.com/WebGLSamples/WebGLSamples.github.io

[JobLeonard]

This looks promising:

https://stardustjs.github.io/

[JobLeonard]

Can be used already to speed up rendering: http://jsfiddle.net/loktar/63QZz/

[cornhundred]

Nice project! You might be interested in our interactive WebGL widget Clustergrammer2

[slinnarsson]

Clustergrammer2 looks very cool - I will try it on some of our data!