[Feat] Incorporate data widgets to the new widgets module

[alasarr]

Target Use Case

In v9, we are working on the new widgets module, I would like to propose adding a set of advanced data widgets, which allows not only a visual representation but a rich interaction with data & map layers, such as filtering or an automatic data refresh on viewport change.

Consider the following dataset:

[
    {
        "store_id": 445,
        "storetype": "Drugstore",
        "zip": "2111",
        "revenue": 1646773,
        "size_m2": 809,
        "geom": {
            "type": "Point",
            "coordinates": [
                -71.06158,
                42.35157
            ]
        }
    }, 
    // ...
]

A widget for the previous dataset looks like this:

A widget allows:

• Visualize the data of the layer as a chart.
• Filter the layer from the widget.
• Filter other widgets' data.

The widget has two modes: global or viewport. When the viewport is enabled the data of the widget is filtered based on the viewport (so it's updated when the ViewState changes).

Multiple widgets are available, check here the different types of widgets.

It's a very common feature widely used in App Development at CARTO, but there is a strong limitation, they are only available via CARTO for React, so:

• Only React users can use it.
• Only React users who want to use our template architecture.

I feel this would be a great feature to add because:

1. Having widgets natively will be something really valuable for all users.
2. A native integration of widgets will be simpler to use for the final user.
3. CARTO can work on this feature as it will make widgets available on all platforms beyond React.

This map includes a layer with widgets where you can see this functionality in action.

Proposal

This proposal is not simple as it will require deciding on multiple things of the core, so I'm writing down the first ideas, but in this case, I feel we will need a design meeting and move from there.

A new set of widgets could be incorporated into the new widgets module (@deck.gl/widgets):

import { FormulaWidget, CategoryWidget } from '@deck.gl/widgets';

const deckgl = new Deck(
   ...
   layers: [
      new GeoJsonLayer({
        id: 'airports',
        // See comments below
        extensions: [new DataFilterExtension()]
      })
    ]
   widgets: [
      new FormulaWidget ({
          id: '',
          title: '',
          getAggregationValue: (d) => d.properties.revenue,
          aggregationFunction: 'max|min|avg|sum|count',
          layerId: 'airports', 
          filterByViewport: true|false
        }),
        new CategoryWidget ({
          id: '',
          title: '',
          getAggregationValue: (d) => d.properties.revenue 
          getCategoryValue:  (d) => d.properties.storetype
          aggregationFunction: 'max|min|avg|sum|count',
          layerId: 'airports', 
          filterByViewport: true | false
          allowToFilterLayer: true | false
        })
   ]
});

A widget is linked to a layer via the layerIdprop. A widget can filter a layer when allowFilterLayer is set to true. Widgets can filter other widgets if they are associated with the same layer. Check this example to see the expected functionality in action.

Filtering system As the widgets filter each other, a common state between widgets is required to manage the filtering.

Each widget (except the formula) should be able to filter the layer in the map, in other words, each widget defines a way to filter the layer, and the current implementation of DataFilterExtension doesn't support all the filters required by the widgets. I think it's a great opportunity to expand it as it was commented at https://github.com/visgl/deck.gl/issues/7827#issuecomment-1505013840. For example, @felixpalmer added support to filter by category at https://github.com/visgl/deck.gl/issues/7827.

TODO: I will add during the following days a table with the filters each widget requires in the layer.

Gathering data The initial idea is that the layer will provide the data for the widgets. It should return the data filtered by the viewport (when filterByViewport is true). It's not the rendered data because the widgets are showing the filtered values too.

Tile Layers

Tile layers (like MVT generated with Tippecanoe) sometimes drop features at lower zoom levels to reduce the tile size. For example, a layer of zipcodes at Z=3 doesn't include all the features. Tilers often keep the most prominent features to guarantee a decent size of tiles across multiple zoom levels. If the widgets are calculated with the data of the tiles the result is not always accurate because some features could have been dropped. For those use cases, a backend is required to perform an accurate calculation; or limit the zoom level where the widget is available to guarantee the data is accurate.

For instance, CARTO has two internal modes to calculate widgets: with the data of the tile (for tilesets) and with an external API (for tables and queries).

An option might be to include a prop with an async function to be used in that case.

 new CategoryWidget ({
    ....
    dataFetcher:  async (viewState, props) => {
          const url = `https://${apiBaseUrl}/widgets/formula`;
          const response = await fetch(url);
          return response.json();
        }
  })

Scope I would keep it at the beginning for the GeoJSON layer and the MVTLayer. We can also reduce the number of widgets to include in v1. We can start with Formula, Category, and Histogram

In general, it's a very large feature. If we decide to go ahead, we will need to split it into smaller pieces.

Should we require a new module for these widgets? Maybe it's too soon yet, but the features required for those data widgets could be more specific. Something to decide as the RFC is taking shape...

[ibgreen]

No doubt such widgets would be exceedingly useful and popular, and if you are willing to lead and contribute to this, we will find a way!

A couple of concerns from my side:

• Scope: This is a form of "charting", and once we get started on charting, there is an almost endless tail.
• Reusability - building such charts specialized to deck.gl interface is kind of limiting
• Data/UI separation: deck.gl already suffers from not sufficiently separating data processing and rendering, see below.

Depending on scope, it could be a separate module but perhaps eventually even a separate, composable framework. As an example math.gl and loaders.gl both grew out of luma.gl, as the code base became too big.

Examples of data separation issues:

• how to get the tiles covering a viewport programatically, even though the tile layer knows how to do it,
• how to get the aggregated data from the heatmap layer for processing instead of rendering etc.

[alasarr]

Thanks @ibgreen for your comments. They have really helped me to evolve the idea of this feature. During this time I have seen the following issues:

• Doing widgets with data coming from tiles is not a good approach, tiling is in general a visualization strategy and tiles often have droppings and the data calculation is not accurate. Thus, it's an incomplete feature that could not work in certain scenarios. In many cases, users need to fall back on a backend implementation to provide accurate data calculation.
• Creating a set of widgets is a never-ending task. We will start with 4 widgets and really soon users will ask for more widgets... I feel we need a more flexible/extensible approach for a library-like deck.gl. I agree we need a better data/ui separation.

Having the previous two issues in mind I'm more likely to go for the following solution.

• The data of the widget should come from a JS async. In the CARTO implementation we will call CARTO SQL API, other backend implementations could follow a similar way.
• Provide a mechanism in deck.gl to bind widgets with layers.
• Use an external Open Source charting library like echarts to build the UI. By using an external library we will increase the catalog of charts available and the flexibility to customize widgets will be higher.
• Improve the DataFilterExtension to be able to apply more filters in the GPU (like categories). That's for me something to do independently of this feature.

Filtering system Widgets can filter a layer (that's why we need to improve the DataFilterExtension). Widgets can filter between themselves. Widgets can be filtered by the viewport of the map. To implement this we need a filtering language that could be serialized/deserialized. The JS function used to populate the data will require to receive the filters and pass it to the backend.

const dataSource = new WidgetDataSource({
    filters: {...},
    data: async ({filters}) => {
          const url = `https://${apiBaseUrl}/widgets/formula?filters=${filters}`;
          const response = await fetch(url);
          return response.json();
        }
})

I'd like to design a data source concept to have a better separation between data/ui.

I would like to hear your thoughts about this approach. I'm aware it's not completed, but I'd like to gather some feedback before moving forward.