Performance - improve handling of large boundary files (eg. RBD)

[ericboucher]

We should augment the caching time allowed for boundary files to speed up app initialization

[wadhwamatic]

@ericboucher - I'm not sure if what you have in mind will help with an issue we have with large boundary files with complex geometries. For example, there are three admin boundary files used in the upcoming RBD deployment ranging from 14 - 40 mb in size. Loading time is slow as you'd expect (see https://demo-prism-rbd-v4.surge.sh).

What are your thoughts?

[ericboucher]

My first thought would be to simplify the boundaries with the script that @DanielJDufour created. That would probably help quite a bit already. We might also want to look at cloud optimized geojson

[ericboucher]

In some cases, we also seem to be loading admin boundaries that do not get used, for example, http://prism-510.surge.sh/, we load admin_boundaries.json but I am not sure it actually gets used

[PysarenkoDS-BWT]

I'm try to save save files locally, but not have some results Try save in LocalStorage, SessionStorage, Cookies but for all this files to large Also I'm try to use Cache this is experimental feature but with this large response it not work too

[PysarenkoDS-BWT]

Investigate how we can use topojson package converting every GeoJson file in TopoJson file doesn't make sense because single TopoJson file takes more space then GeoJson (for example try convert ecu_admbnda_adm2 to TopoJson and file size increase from 37.2 to 41.6 MB)

But we can covert some GeoJson file in one TopoJson, I'm try this and convert ecu_admbnda_adm1 and ecu_admbnda_adm2 their size is 17.9 + 37.2 = 55.1 MB and when I convert it their size in TopoJson is 41.7

So that we can convert and store 1 TopoJson file for each country and on the front side convert them back to GeoJson files to reduce the size of the downloaded data from server.

[wadhwamatic]

@ericboucher - I think we should look into using vector tiles for boundaries and potentially generate them using tippecanoe https://github.com/mapbox/vector-tile-spec/ https://github.com/felt/tippecanoe

[ericboucher]

Also relevant and "new" - https://www.openstreetmap.org/user/daniel-j-h/diary/402706

[laurentS]

I had a quick look at this for rbd and tried removing admin boundary layers from defaultDisplayBoundaries in prism.json to see impact on load time and memory (according to the firefox profiler) in fully local conditions (so no impact of network slowdowns):

• load time is almost the same, the difference (<0.5s) is most likely due to external factors. It only becomes visible when disabling the level 3 layer (which of course we don't want to do). With caching, we'd not have these savings, but savings on network transfer time, which are likely more important on slow networks.
• in dev mode, total memory use drops from 1.7GB (with all 3 layers) to 1.3GB (removing admin1_boundaries) then 1.16GB (keeping only admin_boundaries).
• with the production build, total ram is 1.53GB with all 3 layers, and 944MB with just admin_boundaries, and 723MB with only admin0_boundaries.

It's just a hunch, but it's likely that vector tiles, as suggested above, would indeed help with faster inits, and most probably with memory usage as well. This would probably require changing the code that handles clicks and such to determine where clicks happened, zoom to a region, etc...

[ericboucher]

https://maplibre.org/maplibre-gl-js/docs/guides/large-data/ lists a few ideas as well. I tried running https://reducegeojson.radicaldata.org/ to reduce file size in the PR I just opened, we'll see how it goes. It also gives a fw other ideas and tiling options

https://github.com/WFP-VAM/prism-app/pull/1054

[ericboucher]

A few ideas (ordered):

• reduce file size #1054
• investigate getAllLayerDays / getLayerDays / getLayersAvailableDates functions (super slow / cpu consuming) #1060
• date processing for json files (admin level data) could be handled at build time. see generateIntermediateDateItemFromDataFile
• start download of boundaries earlier (process seems blocked by date processing)
• investigate caching behavior. The app seems to reload files that should be cached
• consider getting boundary data out of the redux state?
• zip/protobuff/msgpack
• consider running https://reducegeojson.radicaldata.org/ at build time?

[ericboucher]

It seems that rasterized borders are available for level 0,1,2 in HDC, https://data.earthobservation.vam.wfp.org/stac/#/collections/wldrfh_admin_lvl2?.language=en

[laurentS]

For what it's worth, I gave msgpack a quick try for something else, and it's a tradeoff:

• JSON.load is a builtin in modern browser, and highly optimised, so it runs faster than msgpack.load which is pure js, by a factor of 5-6.
• the file size is about half with naive msgpack, but can be improved with a custom parser (it's possible to write custom object parsers, which should be very effective for arrays of a specific object). With that custom parser, the resulting msgpack file is likely to be a lot smaller in the end, but I haven't tested it yet.
• switching to msgpack would only make sense if the savings on network transfer time outweigh the slower loading in the browser.

As an alternative, there might be a way to use columnar JSON, which something like polars uses, ie, instead of an array of objects:

[{
  "key1": "value11",
  "key2": "value12",
},
{
  "key1": "value21",
  "key2": "value22",
}
]

use an object with an array of values for each key:

{
"key1": ["value11", "value21"],
"key1": ["value12", "value22"]
}

If all objects have the same shape, the resulting JSON file is about 10-12x smaller. This transformation can be done at build time for the server, and then in a few milliseconds on the browser after loading the JSON. The benefit is faster network transfer and no extra dependencies (the code to convert is ~10 lines of js). It would be worth it if network takes more than ~50ms. Only works for regular structures like the ones in public/data/rbd/ica.json for instance. Probably useless for admin boundary files.

[ericboucher]

After digging into it it seems that geojson is not the main issue here actually, but the fact that the files are way too precise. I am closing this issue in favor of https://github.com/WFP-VAM/prism-app/issues/1057 and suggest to better source the boundary files and/or use a tool like mapshaper.org to simplify the geometries.