Memory Issues

[nabeelamjad]

maptalks's version and what browser you use?

0.42.4 / Chrome 71.0.3578.98

Issue description

I've been drawing a very large amount of objects, around 150K+ (mix of polygons / markers / rectangles) and have been noticing that Chrome just crashes as soon as it passes 1.7-1.8 GB memory on the JS instance (Firefox doesn't appear to cap at). I ran a memory profile on a smaller data set and I wonder if it's worthwhile to take a look to see if some structure can be refactored/optimized to result in a smaller footprint.

I did some comparison with Leaflet and I definitely consume a lot less (same place, zoomlevel and dataset is only 700 MB in Leaflet and 1800 MB in MapTalks)

I can also see that memory increases constantly when panning to different parts but rarely decreases at the same rate (over time it appears to increase slightly eventually leading to a tab crash on bigger data sets).

In addition to that is there a way to "unload" drawn / loaded geometries at a certain zoom level defined by user (other than disabling the layer on certain zoom levels)? Right now at zoom level 1 I can see a black dot that is in actuality 100k+ geometries being rendered. I wonder if it's possible to cluster close geometries (depending on the zoom level), having 100 geometries next to each at level 15 might not be obvious, better served as as many as needed to give the same impression, though that would probably require calculation of each closeby collection to figure out the center.

Thanks for the awesome library in any case!

Please provide a reproduction URL (on any jsfiddle like site)

Memory Heapfile (Chrome) on smaller data set: https://drive.google.com/file/d/1r7Lw1WGwW36vko5NDSOslKNq4pU6Dk7X/view?usp=sharing

[nabeelamjad]

Just to add, I just discovered maptalks.ClusterLayer and that pretty much solves the issues for all the markers I have, I wonder if that can be applied to non Marker shapes as easily? The code seems to be only for markers.

[fuzhenn]

Thanks very much for your comments!

VectorLayer uses a little bit aggressive stategy in caching and memory allocation:

• it caches all the 2D coordinates (for drawing, converted from geographic coordinates) to save the coordinate conversion in the next paiting round.
• it also caches all the external images needed by painting.

As a SVG rendering library, leaflet just converts data to SVG dom elements and let browser take care of memory allocation, I think it's the main reason of difference between leaflet and maptalks's memory consumption.

I will find some time to see if anything can be done to improve the memory allocation.

You can also have a look at the maptalks.biglayer plugin which is for large datasets's rendering.

[nabeelamjad]

Thanks, I need the drawing functionality and adding/remove geometries on the existing data set layer so I don't think biglayer will work for that unfortunately, I'm using canvas rendering (rather than SVG) on Leaflet as well (otherwise it's painfully slow).

I did a quick check and it seems adding a few event handlers on each geometry might be causing an extra footprint (Leaflet allows to set event handlers on the layer (feature group), though I'm unsure if that's the same underlying thing whereby each geometry will get the handler added to it.

[deyihu]

@nabeelamjad

More than 80K+ Markers，maptalks render It's really a bit slow. If 150k+ geometies (mix of polygons / markers / rectangles) ,That's too difficult.(I've never tested successfully. :bowtie: )

For leaflet, I tested data rendering for 300,000 points，That is ok,But you need your own manual write plug-in For maptalks ,If it's 150,000 dots, it's OK. https://deyihu.github.io/src/maptalks-gmvi/examples/icon-big.html

If render 150k+ geometies (mix of polygons / markers / rectangles) ,You may need a webgl renderer such as deck.gl , three.js.

Related plug-ins maptalks.three maptalks.deck-gl

• For three.js, I tested 1 million points and 300,000 buildings.

That is ok; The disadvantage is that you can write a lot of code and you need to be familiar with three. js.

• Deck. GL is very simple, you can use it to support the point layer, polygon layer you need, And the event support is very good

IconLayer: https://deyihu.github.io/src/maptalks-deckgl/examples/iconLayer.html

PolygonLayer: https://deyihu.github.io/src/maptalks-deckgl/examples/polygonLayer.html

[nabeelamjad]

Yeah, I'll look into WebGL renderers and play with that. Thanks!

[nabeelamjad]

@deyihu seems that did the trick, the rendering is very smooth with Deck.GL even at 200k+ objects.

I just have one issue, I can embed this in my browser using something like this and that works fine (maptalks.deckgl.js is not NPM published but I made a local copy)

<script src="https://cdn.jsdelivr.net/npm/maptalks@0.41.1/dist/maptalks.min.js"></script>
<script src="https://unpkg.com/deck.gl@latest/deckgl.min.js"></script>
<script src="../dist/maptalks.deckgl.js"></script>

but when I try a normal ES6 import in my Webpack app it won't work:

import * as deck from 'deck.gl'
import * as maptalks from 'maptalks'
import { DeckGLLayer } from './maptalks.deckgl.js'

....

new DeckGLLayer('', {}).addTo(map) // returns error " this.deckgl.setProps is not a function"

I've also tried all other kind of import orders but it seems nothing really works and maptalks.DeckGLLayer is also not recognized.

What's the correct way using this in an ES6 import app (without having to resort to script tags in my index.html file)?

[deyihu]

@nabeelamjad
deck.gl is externals

I didn't pack deck.gl into maptalks-deckgl

you need config webpack

index.html

<script src="https://unpkg.com/deck.gl@latest/deckgl.min.js"></script>

rep: https://github.com/deyihu/webpacktest

I tested it. It's OK.

[nabeelamjad]

thanks - that does work, I was under the impression it was the same installation as https://github.com/maptalks/maptalks.three