About ocean layer & performance issue

[huming2207]

Hi @systemed

Firstly, thanks for your great map tile generation tool!

I have some questions in regards to the ocean layer in the generated mbtiles bundle:

1. In the data inspection view of tileserver-gl, I can see the ocean layer is somehow always included in the map, even if I am on the land.

For example, here's a screenshot at Albert Park, Melbourne, Australia. As you can see, there are two ocean classes. I believe it should be grassland only.

2. It seems like the mbtiles that tilemaker generated has a performance regression issue on our apps on some cheap Android phones. The rendering library we used is the latest maplibre-gl-native. The symptom basically is:

   • When zooming in and out, sometimes it may lock up for 5 seconds;
   • Or sometimes if we drag the map rapidly (from one city to another), it may also lock itself up;
   • I used the Android Studio profiler tool and tested it out. It seems like the cause of this regression is a method called nativeQueryRenderedFeaturesForPoint, which is from here: https://github.com/maplibre/maplibre-gl-native/blob/ee94520bffeae2f6246284e1931d0cc1d615f708/platform/android/MapboxGLAndroidSDK/src/cpp/native_map_view.cpp#L857
   • We tried removing the ocean layer, and the app will never lock up again! But of course, the map will then become very ugly and unreadable.

May I ask:

1. Why there's an ocean layer included everywhere on the map?
2. How can we get rid of the ocean layer when we don't need it (i.e. on the land)?
3. For the performance regression issue, do you think it's the ocean layer's fault? Or could it be something else went wrong as well?

We also suspect we may have somehow misconfigured the tilemaker. Here's the Lua script and JSON config we are using for your reference: scripts.zip

The building data in the script is generated from the Australian government data and we bundled them into a Shapefile. We have tried excluding the data but it didn't help resolve the performance regression.

Thanks in advance!

Regards, Jackson

[systemed]

I wouldn't necessarily trust the Inspect plugin (which I think is what tileserver-gl is using) - it often seems to suggest that a multipolygon is present at a certain point when it isn't.

If you use vt2geojson to capture a vector tile as GeoJSON, and then open that in QGIS, it usually appears correct to me even when the Inspect plugin claims there's something there. Similarly it's drawing correctly, which is the ultimate test. It might be interesting to compare with a non-tilemaker vector tile source to see if the issue is present there as well, but it's certainly something I've seen before.

My guess is that the performance regression is something to do with over-complex multipolygons being created from the coastline data. Do you have a .pbf extract of just Melbourne alone that I could use to try and reproduce it?

[huming2207]

Hi @systemed

Thanks for your reply. Here's a dump for Williamstown, VIC:

dump.zip

Also thanks for your advice, we will have a look with vt2geojson and QGIS.

Regards, Jackson

[huming2207]

Here's the GeoJSON converted from PBF: 14-14786-10056.zip

The 14-14786-10056.json is dumped from our map, and 14-14786-10056-2.json is from Maptiler. Looks like our map data is not a perfect-ish square. Here's the screenshot from QGIS. The red layer is our data and the green one is Maptiler's data:

[systemed]

You need to look at your attribute_function in the Lua script, which parses shapefile attributes and converts them into vector tile attributes. At present you have:

function attribute_function(attr,layer)
    if attr["featurecla"]=="Glaciated areas" then
        return { subclass="glacier" }
    elseif attr["featurecla"]=="Antarctic Ice Shelf" then
        return { subclass="ice_shelf" }
    elseif attr["featurecla"]=="Urban area" then
        return { class="residential" }
    else
        return { class="ocean" }
    end
end

which is fine for the standard set of shapefiles used in the OMT-compatible schema, but obviously you're adding an extra layer, Buildings.shp. The attribute_function above will set class="ocean" for your buildings which isn't what you want! So you'll need to put an extra line in here that looks at the attributes for the buildings. Note that attribute_function can now take a second layer parameter, which will pass in the layer name.

I think there's probably more to look at here but I'd start with that and see how you get on.

[systemed]

Do you have the buildings shapefile somewhere I could take a look at?

[huming2207]

Hi @systemed

Sorry for the late reply. We've tried something like this:

function attribute_function(attr,layer)
    if attr["featurecla"]=="Glaciated areas" then
        return { subclass="glacier" }
    elseif attr["featurecla"]=="Antarctic Ice Shelf" then
        return { subclass="ice_shelf" }
    elseif attr["featurecla"]=="Urban area" then
        return { class="residential" }
    elseif layer == "ocean" then
        return { class="ocean" }
    else
        return {}
    end
end

The outcomes are: 1. excess "ocean" class is gone; 2. performance seems to be improved a little bit, but sometimes it still stuck for a few seconds on Android phones.

In regards to the building shapefile, we cannot provide it to you as it's huge. But it's available online for everyone. You can download a copy and convert it to Shapefile via GDAL. Here are the links:

• for Victoria (the state where Melbourne is), you can download it from here: https://datashare.maps.vic.gov.au/search?q=uuid%3D9e7590d4-ec98-5aa4-a251-2d957d3da0c7
• for New South Wales (Sydney, Newcastle etc.), you can download it from here: https://portal.spatial.nsw.gov.au/portal/home/item.html?id=0c7f504c2db247b095f41b55010024cd
• for Tasmania (Launceston, Hobart etc.), you can download it from here: https://listdata.thelist.tas.gov.au/opendata/

Regards, Jackson

[huming2207]

We grabbed some of those state government building outline GDB data, and use some sorts of commands like this to bundle and convert into Shapefile:

find . -type d -name "*.gdb" -exec docker run --rm -i -v $PWD:/home/test osgeo/gdal:alpine-normal-latest ogr2ogr -f "ESRI Shapefile" -append "/home/test/bundled.shp" /home/test/{} \;

[systemed]

I've had a play around with using the building data. I don't have an old Android phone but I've been testing on a 2014 Mac. Two things that seem to make a difference to performance:

• You can use the combine_polygons_below layer option to merge adjoining building outlines together. This results in smaller (and therefore faster) geometries. Add it to the building layer definition like this: "combine_polygons_below": 15
• Do you need both OSM and shapefile building data? At present you're writing both and there's a lot of overlap. I'd think that if they represent the same buildings, you could get away without including the OSM ones.

Note that combine_polygons_below generation performance is not great right now as per #270 - it's on my list to look at (unless @kleunen beats me to it ;) ).

[kleunen]

I am pretty busy at work and other open source project. So not sure if i can help you :)

[huming2207]

Thanks for your reply.

• You can use the combine_polygons_below layer option to merge adjoining building outlines together. This results in smaller (and therefore faster) geometries. Add it to the building layer definition like this: "combine_polygons_below": 15

Thanks for your suggestion, I will put it into my config.

• Do you need both OSM and shapefile building data? At present you're writing both and there's a lot of overlap. I'd think that if they represent the same buildings, you could get away without including the OSM ones.

To achieve this, should I just comment out these parts in way_function?

I mean these:

    -- Set 'building' and associated
    if building~="" then
        way:Layer("building", true)
        SetBuildingHeightAttributes(way)
        SetMinZoomByArea(way)
    end

...and:

    if (building~="" or write_name) and way:Holds("name") then
        way:LayerAsCentroid("poi_detail")
        SetNameAttributes(way)
        if write_name then rank=6 else rank=25 end
        way:AttributeNumeric("rank", rank)
    end

I will have a try and see what will happen.

Note that combine_polygons_below generation performance is not great right now as per https://github.com/systemed/tilemaker/issues/270 - it's on my list to look at (unless @kleunen beats me to it ;) ).

That's not a problem, I have a beefy-ish machine here. :)

Regards, Jackson

[huming2207]

Oops my bad, the second one:

    if (building~="" or write_name) and way:Holds("name") then
        way:LayerAsCentroid("poi_detail")
        SetNameAttributes(way)
        if write_name then rank=6 else rank=25 end
        way:AttributeNumeric("rank", rank)
    end

...should be related to POI not building? I will leave it there and only comment out the first one.

Regards, Jackson

[systemed]

I'd just comment out the first one, I think - you probably want to retain POIs.

That's not a problem, I have a beefy-ish machine here. :)

Even so, #383 has a big performance improvement for combine_polygons_below so I'd suggest trying that!

[huming2207]

@systemed yea you are right! My machine just got stuck at around Zoom level 14, writing tile 16605113 of 16605115 for the whole night, versus the #383 takes only an hour or so.

My machines are an M1 MacBook Pro with 16GB RAM, macOS 12.2.1; and a ThinkCentre M75q Gen 2 with Ryzen 4750G, 16GB RAM, Ubuntu 21.10.

And I haven't heard the fan spinning for my M1 Mac for more than 5 minutes in the whole year last year. Tilemaker (particularly the one on the master branch) made it spins! :)