[CWM] Investigation into vector tileset solution

[rogup]

Background

Downloading and rendering a large GeoJSON source in MapLibre takes a lot of time (~5s on a very fast internet connection), which is problematic because we need the map to be usable sooner. There are probably many places in the world where this takes much longer due to internet speeds.

In our in-person planning day, we didn't have time to properly think about a solution for what to do in the meantime whilst the full JSON is downloading. We made a rough decision to display a small subset of the data initially, with no zoom past a certain point and no filter/search functionality, until the full dataset has downloaded.

It's worth doing more investigation into using a vector tileset, since this may provide better functionality and a smoother UX. Vector tilesets are basically designed for this purpose, where pieces of data are quickly fetched depending on the zoom and current lng-lat bounds, without having to download the full set of data.

[EDIT]: Conclusions from this investigation

1. How easy is it to create and host a tileset that contains the marker data? It's very easy to create and host the tileset. It looks like it can be easily automated. For $25/month we can host more tilesets, containing more detailed data e.g. info we want to include in a popup.
2. How long does the initial render take and how smooth is it? The initial render of the vector tiles is very quick for me, under 2 seconds, and is smooth to navigate
3. Is client-side filtering possible? If so, maybe we only need to use a tileset and can forget about the GeoJSON? Client side filtering of vector tiles is possible, but this won't work with our clustering UI, since we need the clustering to change to match the new filtered result. We can't change the geometry of the clusters in a vector tileset at runtime, only the appearance/visibility. So we're definitely going to need to add a GeoJSON source for filter functionality.
4. If not, can the appearance of the vector tile source be consistent with the GeoJSON source, so the transition is seamless? Yes, the appearance and clustering in the vector tiles is consistent with GeoJSON, by using SuperTiler.. There is a split-second of lag and flicker when the sources are switched, so not totally seamless, but good enough to not be too noticeable
5. Can a large JSON be downloaded in background without blocking requests to tileserver, which are needed to refresh the UI? Yes it can be downloaded in the background using the Web Workers API. Whilst it is downloading the large file, the smoothness of navigation is impacted. It's probably something worth trying to improve later but good enough for a MVP. See below for more details

[rogup]

In my initial investigation, I transformed our GeoJSON data into a tileset (.mbtiles file) using the OSS https://github.com/felt/tippecanoe. The resulting file is ~50MB.

tippecanoe -z8 -kg -o test-500000.mbtiles -l initiatives -r1 test-500000.geojson

I then uploaded the tileset to MapTiler, which allows me to host 1 tileset for free. It can then be used directly in MapLibre as the initial source. I've forked the MM branch we were using for previous demos and done a rough implementation on this branch https://github.com/DigitalCommons/mykomap/commits/cwm-test-maps-maplibre/. The resulting map build is deployed here: https://dev.maps.solidarityeconomy.coop/experiment/cwm-test-maps/

In response to some of the questions we need to answer:

1. It's very easy to create and host the tileset. It looks like it can be easily automated. For $25/month we can host more tilesets, containing more detailed data e.g. info we want to include in a popup.
2. The initial render is very quick for me, under 2 seconds, and is smooth to navigate

The appearance of the clusters generated by Tippecanoe is different to SuperCluster. Some of the clusters are quite bunched up and don't look as good. A next step is to try using this library instead, which will hopefully be consistent https://github.com/ChrisLoer/supertiler

Edit: the above published map is now using a tileset I generated with SuperTiler (with a few tweaks to get it working since it wasn't updated for 5 years. Maybe we can eventually publish this working fork if we end up using it) The command I used was

node supertiler.js -i test-500000.geojson -o test-500000_supercluster.mbtiles --minZoom 0 --maxZoom 8 --radius 60 --storeClusterExpansionZoom true --logPerformance

[wu-lee]

Just trying this map above, and I don't see any pins? In the console there are several errors like this:

 Error: Source layer "initiatives" does not exist on source "initiatives-vector" as specified by style layer "clusters".
    _validateLayer maplibre-gl.js:46
 [snip stack trace, which is entirely within MapBox]
 MapView map.ts:183
    MapPresenter map.ts:15
    createPresenter map-ui.ts:112
    createMap map-ui.ts:86
    initUI ui.ts:60
    webRun index.ts:192

(Except that the name of the style layer changes in each case)

I've checked WebGL is enabled in my browser, which is LibreWolf (i.e. Firefox) 128.0-1 on Ubuntu 22.04, repeated on my laptop (LW 127.9.2-1 on Ubuntu 22.04, also Firefox nightly 111.0a1)

[rogup]

It looks like Mapbox can do client-side filtering of vector sources https://maplibre.org/maplibre-gl-js/docs/API/classes/Map/#setfilter

However, this isn't going to work with our clustering, since we want the clustering to change to match the new filtered result. We can't change the geometry of the clusters in a vector tileset at runtime, only the appearance/visibility. So we're definitely going to need to add a GeoJSON source for filter functionality.

[rogup]

Just trying this map above, and I don't see any pins? In the console there are several errors like this:

@wu-lee Sorry, I was just messing around with the tilesets so the API URL was wrong. I've fixed it now.

[rogup]

To finish off this investigation, I tried to get the GeoJSON to download in the background on a worker thread so that API requests to the tileserver are not blocked and the map stays responsive.

The code is here https://github.com/DigitalCommons/mykomap/commits/cwm-test-maps-maplibre/ and the demo map is published here: https://dev.maps.solidarityeconomy.coop/experiment/cwm-test-maps/

I'm seeing that the download successfully works in the background, and the map is responsive but the clustering is slower to respond to zooming in/out whilst the background download is happening. I think this may be because a lot of the network bandwith is being used, despite not totally blocking everything. It's okay and good enough that I think we should go ahead with this vector tile solution. Maybe there are ways to improve the UX further, e.g. by downloading the GeoJSON more gradually in chunks over a longer period of time and/or waiting until the first filter/search is performed by the user.

Once the download has finished, there's a split second where the UI is unresponsive, when the map re-renders the canvas using the new GeoJSON source and the markers flicker. I think this is okay, and not too noticeable.