[CWM] Mykomaps with Mapbox deep dive

[lin-d-hop]

Track under the Cooperative World Map project

Description

This issue is a place to document the findings of a deep dive to unpack how much work is required to get a functional MykoMap for the CWM project with Mapbox.

By 'functional MykoMap' the criteria we are using is:

• 500k markers
• Loads map and markers in 2 seconds
• Using your own desktop (assuming it is not under 50 tabs duress)
• Ask yourself 'If I can a mostly disinterested user on this site, is this slow enough that I'd just close it out of frustration?'

The task is a 2 day timeboxed exploration of the work it will take to create a functional MM using Mapbox. The goal is try and understand as deeply as possible the work making MM work with Mapbox.

The idea is that this get's a bit hacky. Make forks, cull functionality, mock backend calls., play in the sandbox. Whatever it takes.

Acceptance Criteria

• As much understanding as possible about the scale of the changes required to get Mapbox working
• As detailed a breakdown of tasks and plan of attack as possible within the timebox
• Rough estimates on complexity. Tshirt sizes will be okay, but go as granular as time allows.

A word of caution - it might be tempting to think only breadth first (what are all the options) or only depth first (how do I get this one option perfect). The real skill of this task is to balance between both perspectives. You might think of them as two different voices in your brain. Call them both to the table in equal measure :)

[rogup]

I've spent ~1 day so far.

I've successfully swapped in Mapbox: https://dev.maps.solidarityeconomy.coop/experiment/cwm-test-maps

The map takes ~12 seconds to load all markers now, and is much smoother

Profile: https://share.firefox.dev/4eXUKYz

[rogup]

I've experimented further, and the best I can do so far is get loading to ~4s, which feels on the edge of 'good enough'. Demo: https://dev.maps.solidarityeconomy.coop/experiment/cwm-test-maps

This is by creating a GeoJSON before runtime, in the map's config repo. I used https://github.com/node-geojson/geojson-pick and https://github.com/perrygeo/geojson-precision to reduce the size of the GeoJSON as much as possible. And I skipped the parsing of the CSV altogether, since this was taking up CPU time which Mapbox needed to render the GeoJSON data.

I think this parsing can be re-added to happen after the initial Mapbox render, to allow for more functionality such as prop filtering.

I experimented with using Mapbox vector tilesets https://docs.mapbox.com/mapbox-tiling-service/guides These load much faster but can get quite expensive (£100s per month for processing/storage) if we want decent data quality, especially since we're doing a world map over such a huge area. The free version doesn't let you see the markers when fully zoomed out and, when zooming in, the markers are off by ~10m. It also doesn't allow clustering.

So in conclusion:

• Mapbox is a workable option. It can be slotted in without a huge amount of labour, as we were fearing.
• We will need to make the creation of GeoJSON datasets part of our Myokomaps data pipeline.
• Load time is down to 4s. This is basically just downloading a very minimal dataset and loading straight into Mapbox, without any other overhead. The only way to get this down further would be to load and render markers in batches, which appear whilst the user is using the map e.g. we could break up the GeoJSON into world regions and display the user's region first. I don't have time to have a go at this within this spike.

Edit: Oh... actually, it was 4s on my localhost build.

When uploading it to https://dev.maps.solidarityeconomy.coop/experiment/cwm-test-maps , it's now ~10s for me, due to the GeoJSON taking around 5s to download (71 MB)! See here, basically the app is doing nothing whilst it downloads, so it's definitely the main bottleneck: https://share.firefox.dev/4cR4xO8 Which indicates that we'll probably have to do the batching approach

[lin-d-hop]

Awesome @rogup. These investigations are spot on for helping our conversations next week :D