Support interactive framerates on large heatmaps

[ewhitmire]

Spectrograms are a useful tool for visualizing realtime signals, particularly audio data. It would be great if Vega were fast enough to support interactive frame rates on streaming spectrogram data. Here is an example of the kind of functionality I'm talking about. It seems that fundamentally, using rect marks to represent each point is going to be really slow.

Below is a quick test case using Vega-Lite. It generates a 512x100 element heatmap with random values that attempts to update at 50 Hz. On my browser, it slows to a couple of frames per second once the plot fills up. A javascript profile suggests it spends ~200 ms transforming the data and ~300 ms on rendering.

HTML:

```

FPS: ```

[jheer]

Thanks for the example. Faster performance would be great, but this example is something of a worst case scenario for Vega and its use of retained mode graphics:

• It involves heavy object churn, and every object has a corresponding scenegraph entry.
• Objects change on every frame, requiring re-rendering of the entire scene on every frame.
• In addition, most Vega functionality is superfluous here; a straight data-to-drawing pipeline should work, without need for an intervening scenegraph abstraction.

So, while I could imagine some modular changes to Vega that might improve performance for this case (e.g., perhaps extending the image type to support procedural pixel-coloring for raster generation), I'd recommend going a different route for this application, including looking at direct WebGL approaches or, if sticking with a software (Canvas) renderer, shifting the image and drawing in just the new bits, similar to Cubism.js.

[ewhitmire]

I completely agree. After discussing this challenge with @domoritz, I just wanted to log this feature request in case supporting this example becomes feasible one data.

[domoritz]

Thank you for writing up the test case @ewhitmire. By profiling this example, we might find some small optimizations or even opportunities for parallel execution (#1447). However, this won't do magic so for your specific case you will be better off with a custom implementation for now.

[domoritz]

@ewhitmire found that scrolling performance of a double buffered canvas is exceptionally good. I wonder what it would take for Vega's renderer to support decoupling the render area from the axes/legends so that they can move independently. Pure translation of the data can then be handled by only moving a canvas rather than redrawing.

[yuqianma]

@domoritz It seems like a private repo. Can I get more info about the paper and the source?

[ewhitmire]

@yuqianma I've made the paper publicly accessible and will look into moving the source into a public repo.

[yuqianma]

@ewhitmire Thanks!

[ssfrr]

As of Vega version 5.8.0 (released in Nov 2019) there's the heatmap transform which seems to display the data as a canvas. Is that fast enough for this kind of application?

[yuqianma]

@ssfrr Not really, I think. ewhitmire's case's best solution is in his paper. It's like place two charts there, but show parts of them. Then you can update them in a low rate. As the professor said: "this example is something of a worst case scenario for Vega". I was curious about the solution at first. But then I was aware that it is a special case in Vega's scenario.

[jheer]

The heatmap transform should allow for better performance than using a grid of rect marks, so there is some progress here. If the size of the underlying grid is reasonably small real-time updates should work. One can always scale the resulting image mark with smoothing disabled to get a larger heatmap view.

However, this approach still involves recomputing the entire heatmap upon update -- there is no optimization that performs only incremental updates for new time steps. Moreover, with a single image mark one currently loses more fine grained (cell level) interaction support. So @ewhitmire's approach remains better for that use case.

[cornhundred]

We're using regl.js to achieve high framerates with large matrices (millions of data points) in our library clustergrammer-gl https://github.com/ismms-himc/clustergrammer-gl