Blur-based heatmaps

[nyurik]

A very common visualization, especially in maps, is to draw blur-based heatmaps. They are fundamentally different from the discrete heatmap example, because they operate on a very large set of non-aggregated, scattered points, whose coordinates could be close but not the same. The heatmap is then generated with the large overlapping filled circles (symbol mark), but each circle must be transparent and use gradient blur effect (e.g. spectral scheme, from red value in the middle and least transparent, to high transparent, very blured, blue edges). The more circles appear in the same location, the stronger (more solid) color becomes. Note that these types of visualizations usually do not require mouseover detection for each circle.

Examples:

• http://mourner.github.io/simpleheat/demo/
• https://www.mapbox.com/mapbox-gl-js/example/heatmap/

[domoritz]

Would a contour map be a good (maybe even better) alternative?

[nyurik]

@domoritz contour map for what? I'm not sure I understand. This common visualization is very similar to scatter plots of a very large number of points, and it is based on the visual effect of multiple semi-transparent points partially overlapping to solidify the color. Take a look at the first example, and move the mouse in small circles around some area to see what it would look like.

[domoritz]

That's not quite true. It is not multiple semi-transparent points partially overlapping but instead blurring points and then computing a density that is mapped to color. This is the same as a kernel density estimate as in https://vega.github.io/vega/examples/probability-density/ but in two dimensions. A contour plot shows isodensity lines in this 2D kernel density estimate. https://vega.github.io/vega/examples/contour-plot/ is a good example. In order to support the heatmap that you ask for, Vega could expose the KDE directly instead of only the isodensity lines.

[nyurik]

@domoritz thanks for the explanation. I guess I would have to dive deeper into it - I was reading the source code - its definitely not as straightforward as I imagined.

[domoritz]

Yeah, there are a lot of subtleties. A couple of years ago, I built the leaflet plugin for hetmap.js and this summer I created a faceted contour plot. The latter turned out to be quite tricky because I wanted to use the same contour thresholds for each group.

I recommend that you have a look at https://github.com/d3/d3-contour. The API to get the contours is pretty straightforward. To get the KDE, I had to pass a callback to thresholds as Mike recommended in https://github.com/d3/d3-contour/issues/15.

Happy to talk more about details on Slack.

[jheer]

Flagging this as a feature request. As Dominik noted, I think the best way to proceed would be to use d3-contour but retrieve the underlying KDE raster rather than the contour paths. Also, rather than instantiate individual marks per-pixel, I think we'd probably want to render the KDE output into a canvas instance and add that whole thing as a single image mark. That approach might limit opportunities for interacting with heatmap values, but should be immensely more efficient for rendering.

[amitkaps]

The broad approach is bin-summarise-smooth http://vita.had.co.nz/papers/bigvis.pdf

The two implementation approach to look around for this is the bigvis package in R https://github.com/hadley/bigvis and the Datashader package in Python https://github.com/bokeh/datashader

This would be really helpful in handling medium to large data with vega (for data analytics)

[domoritz]

I think all we need is a kernel smoothing since Vega can already bin and aggregate in multiple dimensions.

[clemens-tolboom]

(subscribing as novice)

I see bins in https://vega.github.io/vega/examples/binned-scatter-plot and kernel KDE in https://vega.github.io/vega/examples/probability-density/

In vega-core.js ~3617 it says

// TODO: support for additional kernels?
var randomKDE = function(support, bandwidth) {

Do we need to add a new dist.X there or something new?

[domoritz]

https://vega.github.io/vega/examples/probability-density/

This example is only 1D.

Vega and Vega-Lite supports 2D heatmaps but not smoothing in 2D. Here is an example https://vega.github.io/vega-lite/examples/rect_binned_heatmap.html.

[ryanbaumann]

This discussion from adding GL Heatmaps to Mapbox GL JS last year could be a valuable reference: https://github.com/mapbox/mapbox-gl-js/issues/4756

[guru-florida]

Could this be done using regular rect-mark type heatmaps but adding the feGaussianBlur SVG filter? This would require outputing some filter specs in the defs part of the SVG output. You could not only do gaussian blurring but also have it look gooey by addng feColorMatrix/feBlend filters. Look up "SVG filter feGaussianBlur feBlend feColorMatrix" in an Internet search and there are many examples.

[haji08-ksd]

I am interested in this discussion. I've tried example - https://vega.github.io/vega/examples/density-heatmaps/