Add vectorgrid support

[emilhe]

Add wrapper(s) of leaflet vectorgrid,

https://github.com/mhasbie/react-leaflet-vectorgrid

[prl900]

Hi @emilhe, I wonder how much effort or difficult would be to add this feature to the library. Being able to display vector tiles would be quite useful for working with dense polygon data in Dash applications. Thanks!

[emilhe]

I don't think it will take a crazy amount of time. At least not to implement the basic functionality (:

[henrygsys]

Hi, is this something that is the pipeline to do soon? I would like to be add Mapbox Vector Tile sources to Dash Leaflet but it seems they are not supported. Are there any workarounds for this?

[emilhe]

@henrygsys It's one of the top 3 (larger) features to be requested. I haven't got an exat timeline, but it would help if you could post details here with respect to your usecase (e.g. the URLs that should be loaded), so that I can use it for testing.

[henrygsys]

Sure.

Essentially, I am using Mapbox as a tileserver to store some layers of vector data, mostly global sets of polygons to show protected areas and that sort of thing. The urls follow the below convention and they are added to mapbox using the vector tiles url.

f"https://api.mapbox.com/v4/{tile_set_name}/{{z}}/{{x}}/{{y}}.mvt" https://docs.mapbox.com/api/maps/vector-tiles/

It is possible to add these using the general Dash map figures by passing it as a mapbox layer source using the below. But as i understand it, these do not render when using dl.TileLayer.

fig.update_layout( mapbox={ 'accesstoken': MAPBOX_TOKEN, 'style': style, 'zoom': 2, 'layers': map_layer }

[emilhe]

I did a bit of research on possible options yesterday. It seems that VectorGrid doesn't support mapbox tiles (and it also seems abandoned). I then stumled upon maplibre-gl-leaflet, which supports mapbox files, but doesn't have the same amount of configuration options as VectorGrid. If anyone has experience with mapbox/vector tiles using "raw" leaflet, I would be gratefull for some input on which plugin to choose here in 20203 :)

[ivvnnnn]

@emilhe It seems Vector Grid can support Map Box Vector tiles, if call VectorGrid.Protobuf (link: https://leaflet.github.io/Leaflet.VectorGrid/vectorgrid-api-docs.html). In Leaflet it works. React-leaflet support protobuf too (link: https://github.com/mhasbie/react-leaflet-vectorgrid). May be, it help

[emacollins]

Hi, just wanted to follow up on this thread and the proposed vectorgrid support. Wondering if there are any updates (or potential workarounds?). I am trying to display a very large GeoJSON and it is really slowing down/crashing the app.

Seems the alternative in this situation would be to use a vector tile server (like Tegola).

[emilhe]

While the discussion was active (~ a year ago), i played a bit around with vectorgrid and a few of the alternatives (they differ in their interfaces, which formats are supported, where/how/if slicing is supported, etc.). However, it was never settled which component(s) to add to dash-leaflet.

To my knowledge, there hasn't been any work since then. I haven't encountered any use cases in a work related context, and I thus haven't been able to prioritize development time on the topic.

[Eric-UW-CRL]

First off, really love this project and what it enables as far as building mapping applications with Dash!

It seems like adding the VectorTileLayer would be a huge unlock to the functionality and bring together the ability to display raster and vector data.

I see there is a branch started for this. Where did you leave off or run into issues with this? I would love to help contribute but am still familiarizing myself with the codebase and don't have experience with JS.

[emacollins]

Hey @emilhe , just wanted to follow up. I am curious what issues (if any) you ran into on the vector grid branch? Would love to pick up where you left off and try to get this feature added. I am a beginner in TS and JS.

[emilhe]

The different components address slightly different use cases (various formats, slicing of geojson, etc.) and hold different features. I didn't face any technical issues, but rather the issue of choosing which component to adopt. Do you have a favorite?

[emacollins]

I think that the ideal component to adopt would be the Leaflet VectorGrid, mentioned above by @ivvnnnn using VectorGrid.Protobuf.

In practice it should function quite similarly to TileLayer, except that it will be expecting .pbf responses based on the [Mapbox Vector Tile Spec](MapBox Vector Tile Specification).

It seems like this vector tile spec is widely used, and plays well with PostGIS databases, which has built-in functionality to generate these tiles from geometry data. There are open source tile servers like Tegola that anyone could use to serve their own data directly to dash-leaflet. This way users can now serve vector and raster data to dash-leaflet, which would really round out and enhance the project. In my own project, the biggest bottleneck has been trying to load huge GeoJSONs containing my vector data.

I have found dash-leaflet a joy to use and is a really great project, and well documented! Thanks for all the work you have put into it. @emilhe

[emilhe]

@emacollins it was also the first library (or rather, it's react wrapper) that I looked at. However, with the latest commit dating back years, it seems unmaintained, which makes me hesitant to port it to dash-leaflet. Another project, which more recent development activities is VectorTileLayer , though it lacks some features, and I haven't found any React bindings. There's also react-leaflet-vector-tile-layer , which again has a slighly different feature set, and probably more. Do you have an overview of, which of these options would support your usecase(s)?

Thanks! I am happy that you like it 👍

[emacollins]

After looking through both of them, it looks like VectorTileLayer is more comprehensive and has the functionality that I think fits this use case. The other one claims to support vector tiles, but the documentation is a bit sparse and it doesn't seem like it implements what I am thinking of. Maybe it would work but there is not much there to know.

I realize now that VectorGrid was not part of React-Leaflet. I assume you are referring to this react wrapper that seems unmaintained .

[emilhe]

Reviewing the components in their current state, this is also the option that seems most promising to me at a glance. When I can find the time, I'll start there. If you manage to get something working in the mean time, please create a PR and link it here.

[emilhe]

I have implemented the bare minimal bindings to get data flowing into the VectorTileLayer library from Dash. There is still a lot of work before the component is ready for publication, but it's a start. Here is a sample app,

import dash_leaflet as dl
from dash import Dash

app = Dash()
app.layout = dl.Map(
    [
        dl.TileLayer(),
        dl.VectorTileLayer(url="https://openinframap.org/tiles/{z}/{x}/{y}.pbf", maxDetailZoom=6, style={}),
    ],
    center=[56, 10],
    zoom=8,
    style={"height": "50vh"},
)

if __name__ == "__main__":
    app.run_server(debug=True)

that yields,

where the blue paths are rendered from vector tile data (default styling, default rendering). If you run

pip install dash-leaflet==1.0.17rc1

you should be able to run the example. I have crated a draft PR with the WIP code, https://github.com/emilhe/dash-leaflet/pull/255. @emacollins could you try it out? Testing the (initial) implementation toward a real use case would help map out were to go next in the implementation.

I don't have any (work-related) use cases at the moment. But I was thinking about wrapping up a complete example in docker compose for visualizing huge GeoJSON files, possibly using Tegola as backend. But that'll have to wait for anoter day :)

[emacollins]

This is amazing, thank you! @emilhe. I will put some time in at some point this week to test it out as well. I can put together a simple app using Tegola and some data from my use case using a PostGIS database. Can help out a building out a complete example.

[emilhe]

I have just pushed a new update which includes bindings of event handlers,

pip install dash-leaflet==1.0.17rc2

Building on the previous example, the syntax would be,

import dash_leaflet as dl
from dash import Dash
from dash_extensions.javascript import assign

eventHandlers = dict(click=assign("function(e, ctx){console.log(e); console.log(ctx);}"))

app = Dash()
app.layout = dl.Map(
    [
        dl.TileLayer(),
        dl.VectorTileLayer(url="https://openinframap.org/tiles/{z}/{x}/{y}.pbf", maxDetailZoom=6, style={}, eventHandlers=eventHandlers),
    ],
    center=[56, 10],
    zoom=8,
    style={"height": "50vh"},
)

if __name__ == "__main__":
    app.run_server(debug=True)

I guess next step is to decide, if default event handler bindings should be included (similar to the GeoJSON component).

[zifanw9]

I am experimenting with vector tile. To me, an event handler is useful for retrieving info of the clicked feature (in my case, I want to display popup), and the current event handler is sufficient with the following steps:

Step 1

buildingEventHandlers = dict(click=assign("""function(e, ctx) {
                                                ctx.setProps({
                                                    n_clicks: ctx.n_clicks == undefined ? 1 : ctx.n_clicks + 1,  // increment counter
                                                    clickData: {
                                                        latlng: e.latlng,
                                                        properties:  e.layer.properties
                                                    }  // collect data (must be JSON serializeable)
                                                });  // send data back to Dash
                                            }"""))

Step 2: Initialize the tile layer and a popup inside a map container (to hide it when the page is initially loaded and the user has not clicked on any feature yet, I also set its z-index to very low)

dl.Pane(style=dict(zIndex=1),name="bottompane",id="paneforpopup"),

dl.Popup(position=[34.432546, -119.699944],children=Purify(id="buildingpopupcontent"),id="buildingattributepopup",pane="bottompane"),

dl.VectorTileLayer(url="<TiPG Tile Server Endpoint URL>/collections/public.building/tiles/WebMercatorQuad/{z}/{x}/{y}",style=building_style,filter=building_filter,eventHandlers=buildingEventHandlers,id="building"),

Step 3: Define the callback function to update the popup when user clicks on the vector tile layer

clientside_callback(
    """
    function(nclicks,click_data) {
        if (document.querySelector(".leaflet-popup").parentNode.style.zIndex != 1001) {
            document.querySelector(".leaflet-popup").parentNode.style.zIndex = 1001;
        }
        const popupcontent = `Res: ${click_data.properties.res}<br>` +
                             `ComInd: ${click_data.properties.comind}` ;
        return [[click_data.latlng.lat,click_data.latlng.lng],popupcontent];
    }
    """,
    [Output("buildingattributepopup","position"), Output("buildingpopupcontent","html")],
    Input("building","n_clicks"),
    State("building","clickData"),
    prevent_initial_call=True
)

Not sure if my example here will help on deciding whether to include a default event handler.

Edited Also can I suggest to introduce a hideout property to the vector tile? Not sure if it is convenient to include hideout, but I think this property is useful so that it will allow users to filter features interactively.

Another concern about vector tile/grid is memory usage on clientside browser. I intend to use it as a way to render large dataset (which GeoJson method consumes a lot of memory). When I use Martin tile server docker deployment on Azure App Service, the dash app initially opens with 736MB memory on Chrome, and at some point reached 1.3 GB and later lower to 232 MB memory. Later I try TiPG tile server deployed on render.com $7 plan, I see the memory usage is about 471-520 MB for my dash app. A benchmark comparison with a different map service, using ArcGIS ExperienceBuilder (probably Esri's own tile server) to render the same data consumes 575-711 MB memory. Not sure if anyone has insights on tile server choices or perhaps good practice on reducing memory usage of dash app.

Edit 2 Another note: when I try to map buildings:

1. If I do not specify minDetailZoom=13, tiles corresponding to areas with many small buildings fail to show properly if zoom level is below 13
2. If I do specify minDetailZoom=13, when I zoom to zoom level 11, the memory usage can reach around1 GB even with TiPG tile server. I know this is more of a tile server issue rather than dash issue (Esri Experience Builder in contrast seems to apply a good generalization when zoom to low zoom level), but I feel it is still necessary to mention the issue here. My current mitigation strategy is to use the following filter so that the layer will not display at low zoom level

   building_filter = assign(
   """function(properties,layername,zoom) {
       if (zoom >= 11) { return true; }
       else { return false; }
   }
   """
   )

[zifanw9]

Hi @emilhe, in PR#255 line 100 of src/ts/react-leaflet/VectorTileLayer.ts, vector tile layer does not seem to have method setUrl(url), and it causes error if I try to update the url of the layer. Not sure if there can be any fix to this issue.

I also just realize it might be possible to do filtering on vector tile layer without introducing hideout property, but this requires updating the url of the layer. This will involve using CQL if the tile server (e.g., TiPG) supports it, a few examples:

url="<Tile Server Endpoint URL>/collections/public.building/tiles/WebMercatorQuad/{z}/{x}/{y}?filter-lang=cql2-text&filter=res IS NULL AND comind IS NULL"
url="<Tile Server Endpoint URL>/collections/public.building/tiles/WebMercatorQuad/{z}/{x}/{y}?filter-lang=cql2-text&filter=res IN ('0','1')"

I feel this vector tile layer is ready for release if its url update method can be fixed. Hideout property might still be useful if we want to change the style, but this can just be future enhancement maybe. Thank you!