Support type=boundary in addition to type=multipolygon

Nakaner commented 3 years ago

EDIT: See my comment below. It's not about broken geometries but Tilemaker not using boundary relations to build polygon geometries.

I observed that large administrative polygons are missing from the output. For a tileset of the whole world, most countries miss from zoom level 0 to 2. Only small exclaves (islands on sea) are in the output. This is independent from using the raw .osm.pbf as input for boundaries or a shape file of simplified polygons.

Here is a test case:

.osm.pbf input file: germany.osm.zip – valid multipolygon relation according to JOSM validator
shape file with simplified boundary polygon of Germany (maximum error 1000 m, all start and end nodes of boundary ways preserved): germany-simplified.zip

Tilemaker is called this way: tilemaker --bbox -180,-85,180,85 --input germany.osm.pbf --output polygon-test.mbtiles --config config-polygon-test-shapefile.json --process process-polygon-test.lua --verbose (using the bbox branch)

The following tiles are produced:

./0/0/0.pbf
./1/1/0.pbf
./2/2/1.pbf
./3/4/2.pbf
./4/8/5.pbf
./5/16/10.pbf
./6/33/20.pbf
./6/33/21.pbf
./7/66/41.pbf
./7/66/43.pbf
./8/132/86.pbf
./8/133/82.pbf
./metadata.json

On zoom level 6, only two polygons are in the file: the exclave "Tiefwasserreede" (OSM way 177629938) and another exclave along the German-Belgian border (OSM way 24718735). Exclaves which are no closed rings do not appear in the output file (e.g. Büsingen).

Configuration for test based on .osm.pbf file:

{
    "layers": {
        "boundaries": {
            "minzoom": 0,
            "maxzoom": 10,
            "simplify_below": 0,
            "simplify_level": 0.0001,
            "simplify_ratio": 2
        }
    },
    "settings": {
        "minzoom": 0,
        "maxzoom": 8,
        "basezoom": 14,
        "include_ids": false,
        "name": "Tilemaker admin polygons test",
        "version": "3.0",
        "description": "test",
        "compress": "gzip",
        "filemetadata": {
            "tilejson": "2.0.0",
            "scheme": "xyz",
            "type": "baselayer",
            "format": "pbf",
            "tiles": [
                "https://example.com/liechtenstein/{z}/{x}/{y}.pbf"
            ]
        },
        "metadata": {
          "author": "OpenStreetMap contributors",
          "license": "Open Database License 1.0"
        }
    }
}

Configuration for test based on shape file:

{
    "layers": {
        "boundaries": {
            "minzoom": 0,
            "maxzoom": 10,
            "simplify_below": 0,
            "simplify_level": 0.0001,
            "simplify_ratio": 2,
            "source": "germany-simplified.shp",
        "source_columns": true
        }
    },
    "settings": {
        "minzoom": 0,
        "maxzoom": 8,
        "basezoom": 14,
        "include_ids": false,
        "name": "Tilemaker admin polygons test from shape files",
        "version": "3.0",
        "description": "test",
        "compress": "gzip",
        "filemetadata": {
            "tilejson": "2.0.0",
            "scheme": "xyz",
            "type": "baselayer",
            "format": "pbf",
            "tiles": [
                "https://example.com/liechtenstein/{z}/{x}/{y}.pbf"
            ]
        },
        "metadata": {
          "author": "OpenStreetMap contributors, Geofabrik GmbH",
          "license": "Open Database License 1.0"
        }
    }
}

Lua processing file:

-- Enter/exit Tilemaker
function init_function()
end
function exit_function()
end

-- Process node tags

node_keys = { }

inf_zoom = 99

function node_function(node)
end

function process_boundary(way)
    local area = way:Area()
    if area == 0 or not way:Find("type") == "boundary" then
        return
    end
    local mz = inf_zoom
    local admin_level = tonumber(way:Find("admin_level"))
    if admin_level == nil then
        return
    end
    if admin_level == 2 then
        mz = 0
    elseif admin_level == 4 then
        mz = 7
    end
    if mz < inf_zoom then
        way:Layer("boundaries", true)
        way:MinZoom(mz)
        way:Attribute("admin_level", admin_level)
        way:AttributeNumeric("osm_id", way:Id())
        if way:Holds("name") then
            way:Attribute("name", way:Find("name"))
        end
    end
end

function way_function(way)
    area = way:Area() > 0
    -- Layer boundaries
    if area and (way:Find("boundary") == "administrative") then
        process_boundary(way)
    end
end

kleunen commented 3 years ago

Does tilemaker give any output the geometries are invalid ? And does it try to correct them ?

Nakaner commented 3 years ago

Note: I overlooked a warning during the test with the shape file. The shape file geometries are processed as expected. Tilemaker failed to find the shape file and warned me. :-(

The .mbtiles files can be found here:

.mbtiles file from .osm.pbf file only: from-osm.mbtiles.zip
.mbiltes file from shape file: from-shapefile.mbtiles.zip

Does tilemaker give any output the geometries are invalid ? And does it try to correct them ?

The output is:

michael@globe:~/jobs/tilemaker/polygon-test$ ~/git/github.com/systemed/tilemaker/build/tilemaker --bbox -180,-85,180,85 --input ~/jobs/tilemaker/germany.osm.pbf --output ~/jobs/tilemaker/polygon-test.mbtiles --config ~/jobs/tilemaker/polygon-test/config-
polygon-test.json --process ~/jobs/tilemaker/polygon-test/process-polygon-test.lua --verbose
mbtiles file exists, will overwrite (Ctrl-C to abort, rerun with --merge to keep)
Bounding box -180, 180, -85, 85
Layer boundaries (z0-10)
Reading .pbf /home/michael/jobs/tilemaker/germany.osm.pbf
Block 19/22 ways 0 relations 0        
Sorting nodes
Way 24718735 has the wrong orientation
Way 177629938 has the wrong orientation
Block 20/22 ways 1433 relations 0        
Sorting ways
Stored 153987 nodes, 1433 ways, 0 relations
Shape points: 0, lines: 0, polygons: 0
Generated points: 0, lines: 0, polygons: 2
Zoom level 8, writing tile 12 of 12               
Memory used: 1151564

Filled the tileset with good things at /home/michael/jobs/tilemaker/polygon-test.mbtiles

It claims that the two exclaves which were written to the output file have the wrong orientation. However, there is no good or bad orientation for closed ways in OSM.

Nakaner commented 2 years ago

I digged a bit deeper w.r.t. to this issue. Invalid geometries or geometry issues in general are not the problem. However, Tilemaker does not build polygon geometries for relations with type=boundary although they follow the same rules as multipolygons.

Therefore, building their polygon geometries during loading of the OSM input file would solve this issue.

If there are doubts about performance (some boundaries are huge and complex), a new Lua callback function can make it possible to select which relations to build polygons from.

systemed commented 2 years ago

Yep. The current approach (v2.1 onwards) taken by the OMT-compatible schema is that tilemaker notes which ways are members of boundary relations (type=boundary, boundary=administrative) using relation_scan_function. It then writes out any such ways to a boundary VT layer, with admin_level set to the most important boundary (=smallest number) of which the way is part.

With your own schema, you could follow this approach. Alternatively, rather than writing each member way into the boundary layer, you could use relation_function to assemble complete geometries.

https://github.com/systemed/tilemaker/blob/master/docs/RELATIONS.md explains how this works. In the example given, relation:Layer("bike_routes", false) writes them as multilinestrings. If you wanted to write them as multipolygons instead, you'd change the second parameter to true.

systemed / tilemaker

Support type=boundary in addition to type=multipolygon #284