Heatmap support

[davidmanzanares]

• [x] Custom PoCs: NY foot trafic dataset, gecat dataset and WWI_ships dataset
• [x] Support for grid-alignment (NY dataset)
• [x] Bi-cubic interpolation
• [x] Isolines PoC
• [ ] Support non-square windows
• [ ] API
• [ ] Support for limitting heatmap grid size based on zoom
• [ ] Workarounds lack of OES_texture_float and OES_texture_float_linear. Important for mobile.
• [ ] Automatic discovery of min/max values

[davidmanzanares]

API pseudo Proposal

Create a heatmap by accumulating the count with the SUM function and use 0 as the low value for the ramp, and 100 for the high value. Each heatmap cell will cover 20x20 pixels

heatmap: ramp(linear(cellSum($count), 0, 100), tealrose) 
heatmapResolution: 20

Implementation proposal

The rendering part has already been more or less addressed.

Regarding the API, we probably won't support every feature from the start, but, I think this API is achievable and flexible. For example, it allows setting different aggregation functions. Internally, those aggregations could be sent to Maps API, greatly reducing MVT size.

It also allows to specify the MIN/MAX values for the ramp, which at first could be delegated to the user, but, it could be set dynamically in the future, like:

heatmap: ramp(linear(
                                    cellSum($count),
                                    viewportMin(cellSum($count)),
                                    viewportMax(cellSum($count))
                          ),
                         tealrose) 

Where viewportMin(...) returns the minimum value of the cells on the viewport. However, having a constant expression, like a constant number or a zoom-dependant expression is probably fine for most cases.

We could have isopleth maps with:

heatmap: ramp(buckets(cellSum($count),
                                    0, 20, 50, 100
                          ),
                         tealrose) 

And isolines with:

heatmap: ramp(isolines(cellSum($count),
                                    0, 20,50,100, 0.1
                          ),
                         tealrose) 

??? I don't like this one

...

[davidmanzanares]

cc @rochoa

[davidmanzanares]

Some ideas:

{
    // Automatic switch to pixel shading?
    @categories: buckets($cat, 'a', 'b')
    symbol: sprite(@categories,
        ['https...',
        'https...',
        'https...']
    )
    color: ramp(@categories, prism)
}

{
    // Automatic switch to heatmap rendering
    color: ramp(linear(heatmapSum($temp, 10), 0, 30), temps)
}

{
    // Iso Pleth
    color: ramp(buckets(heatmapSum($temp), 0, 10, 20, 30), temps)
}

{
    // Iso lines
    color: ramp(heatmapSum($cat), prism)
    filter: buckets(heatmapSum($cat), [0, 1], [10, 11], [20, 21], [30, 31])
}

{
    // Grid map
    @grid: ramp(linear(gridSum($temp), 0, 30), temps)
    color: opacity(@grid, 0.9)
    strokeColor: opacity(@grid, 1.)
    strokeWidth: 4
}

viz.color.blendTo(opacity(viz.color, 1))

createIsolines($cat, 10, 1, temps) // Returns a carto.viz

[makella]

@davidmanzanares

Above, you mention: "Create a heatmap by accumulating the count with the SUM function and use 0 as the low value for the ramp, and 100 for the high value."

Just to make sure, we are after more than just a visual representation of density, right? In order to get deeper into the map types that we should support, I think I need to understand the basics of what is being implemented a little better.

In addition, depending on what the data are (crime vs. GPS vs. weather) the user will likely want more control on the methods used and the visual output of the interpolated surface.

I'm not an expert in "heat maps" but to go beyond visual density, we need:

• cell size (20 x 20)
• the search radius for the density calculation (do we have this?)
• the density calculations/interpolation methods (count,min,max,avg... what else? IDW,... )

In addition, I'm curious:

• how we handle this through zoom
• we need better CARTOColor schemes to map to the range (0-100) likely going from "cool" to "hot" with a good, multi-hue change in between.

I think once I understand what our goal is and how we will be implementing it, that I can contribute to the conversation better!

[makella]

In addition, aside from enhanced CARTOColor schemes, if a user defines their own colors, we will likely need to think about another color interpolation space as cielab mainly provides a uniform transition without introducing additional hues... and in the case of these interpolated surfaces, that will be key.

[makella]

this is interesting read on advantages and disadvantages of interpolation methods for lots of points.

"There are many methods for interpolating values from a dataset. Which interpolation is best depends on the source dataset and on how you will use the values."

https://www.movebank.org/node/6400

[AdriSolid]

Something new about it? Thanks!

@davidmanzanares @makella @rochoa

[rochoa]

No news. Sadly, this won't be part of the initial release of CARTO VL.

We will work to incorporate it in the next versions.

[rochoa]

@AdriSolid, what's your use-case?

[AdriSolid]

@rochoa I needed it in order to render some sensors data, I developed the app using pure Mapbox methods.. But waiting for CARTO VL heatmap methods for next apps!!

[nerik]

Hey there. Just a quick note, we are just starting to evaluate different solutions for a new version of our animated heatmap at GlobalFishingWatch. So far our potential options are deck.gl, CARTO VL or go with writing our own MGL custom layer. (in case you're interested this is our current approach)

Very cool to see that gif posted by @davidmanzanares . This feature could really be a dealmaker for us. Is it still on the roadmap?

[jorgesancha]

Hi @nerik Thanks for the feedback! And really interesting post.

We have not forgotten about this and it is definitely coming, but unfortunately maybe not fast enough. It is unlikely we get to this before end of Q2. Hope that gives you a frame of reference

[nerik]

Hi @jorgesancha. Thanks for the reply, and noted for the timeline. One question though: there's a gif attached to the issue, does that mean you have some working proof-of-concept somewhere? If possible we'd be interested in checking it out.

[jorgesancha]

I believe this is the branch. Bear in mind that this way behind the master branch, based in code that no longer exists and probably with a lot of hacks just to proof the concept

[davidmanzanares]

Yes, there were multiple branches. The last one I think it is square-grid, but it also contains some unwanted changes to the sources.

I think that a merge would be too difficult at this point too. Rewriting the functionality while reading the diff of the branch is probably the easiest way.

You can read the diff here: https://github.com/CartoDB/carto-vl/compare/square-grid

The interesting parts that could be copied are on renderer.js and on the shaders.

Keep in mind that the branch never made a public API to configure the heatmap.