Low-zoom ocean shapefile not rendered correctly on some setups

[codebrane]

Using Mapnik 3.0.9 with openstreetmap-carto 2.39.0 produces tiles with a blue tint. The coastlines are not visible. Using the version from master, the land and inland water is fine but the sea isn't rendered. This tile shows the issue with 2.39.0. I'm using polytiles.py with osm.xml generated by carto from project.mml. If I use the default Mapnik osm.xml on the same data it renders fine but of course with the older OSM style.

[codebrane]

This is what I get using openstreetmap-carto master and this is what it should look like

[matthijsmelissen]

Have you tried running ./get-shapefiles.sh? We switched from land-polygons to ocean-polygons, this might have something to do with this.

I also saw some blue tiles in Europe on the openstreetmap.org servers, so there might also be a data problem.

[codebrane]

yes. The flow was:

./get-shapefiles.sh osm2pgsql -c -d gis --slim -C 1000 --flat-nodes flatnodes GB.pbf --style openstreetmap-carto.style carto -l project.mml > osm.xml polytiles.py -b -5.93 55.68 -2.29 58.79 -z 10 11 -s osm.xml -t tiles

if I use: polytiles.py -b -5.93 55.68 -2.29 58.79 -z 10 11 -s osm.xml -t tiles using the original Mapnik osm.xml the tiles are fine

[pnorman]

The shapefiles are not rendering on either 2.39.0 or master for you.

On master go to the same directory as project.mml and osm.xml and run

ogrinfo -so -al data/simplified-water-polygons-complete-3857/simplified_water_polygons.shp
ogrinfo -so -al data/water-polygons-split-3857/water_polygons.shp

The results should be

INFO: Open of `data/simplified-water-polygons-complete-3857/simplified_water_polygons.shp'
      using driver `ESRI Shapefile' successful.

Layer name: simplified_water_polygons
Geometry: Polygon
Feature Count: 6101
Extent: (-20037558.342789, -20037408.342789) - (20037558.342789, 20037558.342789)
Layer SRS WKT:
PROJCS["WGS 84 / Pseudo-Mercator",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Mercator_1SP"],
    PARAMETER["central_meridian",0],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",0],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["X",EAST],
    AXIS["Y",NORTH],
    EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext  +no_defs"],
    AUTHORITY["EPSG","3857"]]
FID: Real (11.0)

INFO: Open of `data/water-polygons-split-3857/water_polygons.shp'
      using driver `ESRI Shapefile' successful.

Layer name: water_polygons
Geometry: Polygon
Feature Count: 35214
Extent: (-20037508.342789, -20037508.342789) - (20037508.342789, 20037508.342789)
Layer SRS WKT:
PROJCS["WGS 84 / Pseudo-Mercator",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Mercator_1SP"],
    PARAMETER["central_meridian",0],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",0],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["X",EAST],
    AXIS["Y",NORTH],
    EXTENSION["PROJ4","+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0 +k=1.0 +units=m +nadgrids=@null +wktext  +no_defs"],
    AUTHORITY["EPSG","3857"]]
FID: Real (11.0)

Check that the feature count is approximately the same.

Also, post the result of ls -hl data/water-polygons-split-3857 data/simplified-water-polygons-complete-3857

[codebrane]

simplified_water_polygons.shp output is identical to above. water_polygons.shp Feature Count = 35211 instead of 35214

ls -hl data/water-polygons-split-3857 data/simplified-water-polygons-complete-3857

data/simplified-water-polygons-complete-3857:
total 29M
-rw-r--r-- 1 vagrant vagrant    6 Mar  1 08:37 simplified_water_polygons.cpg
-rw-r--r-- 1 vagrant vagrant  72K Mar  1 08:37 simplified_water_polygons.dbf
-rw-rw-r-- 1 vagrant vagrant 169K Mar 30 12:15 simplified_water_polygons.index
-rw-r--r-- 1 vagrant vagrant  847 Mar  1 08:37 simplified_water_polygons.prj
-rw-r--r-- 1 vagrant vagrant  29M Mar  1 08:37 simplified_water_polygons.shp
-rw-r--r-- 1 vagrant vagrant  48K Mar  1 08:37 simplified_water_polygons.shx

data/water-polygons-split-3857:
total 565M
-rw-r--r-- 1 vagrant vagrant    6 Mar  1 08:38 water_polygons.cpg
-rw-r--r-- 1 vagrant vagrant 413K Mar  1 08:38 water_polygons.dbf
-rw-rw-r-- 1 vagrant vagrant 497K Mar 30 12:15 water_polygons.index
-rw-r--r-- 1 vagrant vagrant  847 Mar  1 08:38 water_polygons.prj
-rw-r--r-- 1 vagrant vagrant 564M Mar  1 08:38 water_polygons.shp
-rw-r--r-- 1 vagrant vagrant 276K Mar  1 08:38 water_polygons.shx

[codebrane]

I'm using this to import GB.pbf to the db:

osm2pgsql version 0.91.0-dev (64 bit id space)

[tds4u]

Problem is still there, even with ogr2ogr for all shape files to fix possible inconsistencies.

[matthijsmelissen]

I cannot reproduce this.

@codebrane Did you already find the cause of this problem by any chance?

[tds4u]

I can reproduce that:

mapnik-config -v
2.2.0

node -v
v0.10.29

Distributor ID: Debian
Description:    Debian GNU/Linux 8.4 (jessie)
Release:        8.4
Codename:       jessi

[matthijsmelissen]

@springmeyer Do you perhaps have any idea what might cause this rendering glitch? The screenshot above looks quite particular.

[imagico]

The image shown looks like an 180-degree-meridian reprojection error.

Keep in mind what i wrote in

http://blog.imagico.de/on-slicing-the-world-news-from-openstreetmapdata-com/

on the simplified water polygons - that you should not attempt to reproject them, not even a simple round trip conversion Mercator->Geographic->Mercator. This will fail since the polygons slightly extend over the 180-degree-meridian.

I assume if the coordinate system you have set in Mapnik and the coordinate system in the shapefile are formally different you get a coordinate system conversion and therefore a failure. I am not familiar enough with Mapnik to know if and under what conditions this happens.

[tds4u]

Projection is EPSG:3857 / WGS 84 Pseudo Mercator and in project file it is EPSG:900913 - so the google one. Should be no problem at all.

@springmeyer: Did you test it with planet import?

[tds4u]

QuantumGIS can render this correctly:

I think the shape file is broken, the following is working:

#  - id: "ocean-lz"
#    name: "ocean-lz"
#    class: "ocean"
#    geometry: "polygon"
#    <<: *extents
#    extent: *world
#    srs-name: "mercator"
#    srs: "+proj=merc +datum=WGS84 +over"
#    Datasource:
#      file: "data/simplified-water-polygons-complete-3857/simplified_water_polygons.shp"
#      type: "shape"
#    advanced: {}
#    properties:
#      maxzoom: 9
  - id: "ocean"
    name: "ocean"
    class: "ocean"
    geometry: "polygon"
    <<: *extents
    Datasource:
      file: "data/water-polygons-split-3857/water_polygons.shp"
      type: "shape"
    properties:
#      minzoom: 10
    advanced: {}

[xan-der]

Can confirm this problem with Mapnik 3.0.11 and Mapnik 2.2 and the current openstreetmap-carto pulled earlier this week. All of the global level maps (through level 5 or so) are malformed. I see exactly the same image as reported by tds4u earlier. Note that OSMBright from MapBox works correctly using Mapnik 3.0.11, but it doesn't use water-polygons, it only uses simplified-land-polygons-complete-3857 and land-polygons-split-3857.

[matthijsmelissen]

Thanks for the confirmation. Certainly something we need to look at.

@pnorman Any idea why some users suffer from this and other don't?

[pnorman]

@pnorman Any idea why some users suffer from this and other don't?

No. I'll see if I can reproduce.

[xan-der]

Followup. I was able to get clean world level images by changing project.mml to use the ocean color for the background and land-polygons-split-3857/simplified-land-polygons-complete-3857. Have only tried the first few zoom levels so far. This was the only change made to the mml file - replacing background color and the two layers from water to land.

From this experiment, I have to conclude that there is something squirrely about the water polygons as they are processed by Mapnik.

[pnorman]

Having tried recent water polygons on several versions of Mapnik, I'm unable to reproduce, so I'm concluding that the issue is something local or a problem specific to some version of Mapnik, neither of which are fixable in this repo.

[matthijsmelissen]

Note that we have three people reporting the same problem, so I'd still like to keep an eye on this. I wouldn't like to see this problem suddenly showing up on production without us having an idea of what might cause it.

[springmeyer]

Those that are able to replicate: does the problem go away if you delete the .index file that comes along with the shapefiles?

[xan-der]

The problem did not go away after deleting the .index files for the water polygons. Attached are two images that show (1) what you get when using water color as the background and land polygons as the foreground and (2) what you get when using land color as the background with water polygons as the foreground . Besides the water/land inversion, the rest of the yaml/mml/xml files are identical.

The configuration is: Debian 8.3, Mapnik 3.0.11 (using GDAL 2.0.2, boost 1.60), simplified_water_polygons and water-polygons (split-3857) both dated 18 April 2016, Python 2.7.9.

While Mapproxy is being used as the front end, caching for the example layers is turned off, so the images are regenerated every time.

[springmeyer]

I don't know what would cause this.

[tds4u]

Let me think...

• Carto will convert YAML into CSS via NodeJS. So no problems so far, right?
• Mapnik parses XML and render stuff, equal if 2.2 or 3.x, right?
• Using MapProxy or other render tools is not important because all use Mapnik, right?

So the problem maybe is not inside Mapnik, maybe inside conversion or rendering tool chain. Maybe OGR or similar libraries? But how to check this?

[pnorman]

Mapnik doesn't read CartoCSS, it reads Mapnik XML. Carto converts CartoCSS to Mapnik XML. You could rule it out by building the XML on a known good machine, but I can't see any way for it to be a cartocss problem with that last picture of inverted water bands.

[tds4u]

So, I tested further. With my workaround (https://github.com/gravitystorm/openstreetmap-carto/issues/2101#issuecomment-211353096) I had some blocks on the 180° meridian. That happen if rendering was made till level 10 via MapProxy - rendering till level 8 won't generate such blocks!!! After that all is fine for rendering from 8 to maximum level. So maybe it's a "feature" / bug on higher zoom levels with shape files?

[pnorman]

I don't know what would cause this.

As @imagico commented, could it be related to the reprojection? Is mapnik making any reprojections here? If so, is it using any libraries which might change the behavior depending on version?

---

@codebrane what zoom range did you observe problems on?

[codebrane]

zoom 10 shows the water problems. Below that there isn't any boundary between the missing water and the land (screenshot of level 5 attached)

[pnorman]

zoom 10 shows the water problems

If you're seeing them on zoom 10+, that means you're seeing problems with both the ocean-lz and ocean layers.

What OS are you using, as well as what geos and proj versions is your Mapnik linked against?

[codebrane]

Ubuntu 15.04 mapnik-config -v = 3.0.9 geos-config --version = 3.4.2

[pnorman]

@codebrane proj version? proj | head -n1 will get you it

[codebrane]

sorry about that proj | head -n1 Rel. 4.8.0, 6 March 2012

[tds4u]

Two different servers but same problem, here are the versions:

Planet (businessmaps.info):

root@server1 / # lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 8.4 (jessie)
Release:        8.4
Codename:       jessie
root@server1 / # lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 8.4 (jessie)
Release:        8.4
Codename:       jessie
root@server1 / # proj
Rel. 4.8.0, 6 March 2012
usage: proj [ -beEfiIlormsStTvVwW [args] ] [ +opts[=arg] ] [ files ]
root@server1 / # node -v
v6.2.0
root@server1 / # mapnik-config -v
2.2.0
root@server1 / # geos-config --version
3.4.2
root@server1 / # gdalinfo --version
GDAL 1.10.1, released 2013/08/26
root@server1 / # osm2pgsql
osm2pgsql version 0.91.0-dev (64 bit id space)

D-A-CH (maps42.de):

root@server0:/# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 8.4 (jessie)
Release:        8.4
Codename:       jessie
root@server0:/# proj
Rel. 4.9.1, 04 March 2015
usage: proj [ -beEfiIlormsStTvVwW [args] ] [ +opts[=arg] ] [ files ]
root@server0:/# node -v
v0.12.14
root@server0:/# mapnik-config -v
2.2.0
root@server0:/# geos-config --version
3.4.2
root@server0:/# gdalinfo --version
GDAL 1.10.1, released 2013/08/26
root@server0:/# osm2pgsql
osm2pgsql SVN version 0.88.0 (64bit id space)

Howto:

lsb_release -a proj node -v mapnik-config -v geos-config --version gdalinfo --version osm2pgsql

[pnorman]

@tds4u What software are you using to render the maps? (e.g. kosmtik, mapproxy, renderd)

[tds4u]

I use MapProxy 1.8.x for pre-rendering tiles (WMS, WMTS, TMS).

[pnorman]

Is anyone getting this with something other than MapProxy?

[StyXman]

Maybe this is related? Parts of Marseille are overwater! https://www.openstreetmap.org/#map=18/43.29154/5.36511 https://www.openstreetmap.org/#map=17/43.29179/5.36164 https://www.openstreetmap.org/#map=16/43.2901/5.3592

Notice that here is ok: https://www.openstreetmap.org/#map=15/43.2901/5.3592

[pnorman]

Maybe this is related? Parts of Marseille are overwater!

No, that's unrelated to this issue.

[pnorman]

Is anyone getting this with something other than MapProxy?

The original reported was using polytiles.py, but like MapProxy this uses Python.

It would be interesting if there were any reports from nodejs bindings (e.g. kosmtik, tilemill) or C++ bindings (e.g. renderd)

[matthijsmelissen]

And, for completeness' sake, any reports from MapProxy where this problem does not occur :)

[pnorman]

Can someone reproducing the bug try going into the produced JSON and changing "srs-name": "900913", to "srs-name": "3857", for the ocean-lz layer, regenerating the Mapnik XML from the JSON MML file, then clearing your cache and re-rendering?

Don't make any changes to the YAML.

[tds4u]

On my own server there is no difference in rendering when switching 900913 to 3857 in ocean-lz layer.

Maybe we can use the approach with TIFF files, using DEM data and colored by a shade ramp file?

[xan-der]

While it doesn't fix the current situation, there is a simple workaround. As I mentioned in an earlier post, simply use water color for the background and then use the land polygons shapefile instead of the water polygons shapefile. Conceptually, I'm not clear why the continents are being drawn as negative space in water anyway :)

FWIW, it isn't just MapProxy. A straight Mapnik render results in the same effect.

[pnorman]

While it doesn't fix the current situation, there is a simple workaround

We're interested in fixing the problem, not reverting #2066

FWIW, it isn't just MapProxy. A straight Mapnik render results in the same effect.

With what bindings?

[xan-der]

A non-MapProxy example, using python bindings, OGCServer (pulled today, with edits to change Envelope to Box2d)) via mod_wsgi as the front end. The request looked like this: http://192.168.218.219/osm?LAYERS=__all__&STYLES=&FORMAT=image/png&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:3857&BBOX=-20037508.34,-20037508.34,20037508.3384,20037508.3384&WIDTH=256&HEIGHT=256

Standard map_factory.py and mapnix.xml as configuration files. Following the example above, the tool information is:

debian8# lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 8.4 (jessie) Release: 8.4 Codename: jessie debian8# proj Rel. 4.8.0, 6 March 2012 usage: proj [ -beEfiIlormsStTvVwW [args] ] [ +opts[=arg] ] [ files ] debian8# node -v bash: node: command not found debian8# mapnik-config -v 3.0.11 debian8# geos-config --version 3.4.2 debian8# gdalinfo --version GDAL 2.0.2, released 2016/01/26

The resulting image looked like this:

[pnorman]

I reduced it to a minimal testcase: https://github.com/pnorman/openstreetmap-carto/tree/onlyocean

Could someone who has the problem try that and verify that they still have problems? A low zoom is best, you should get just ocean/land rendering

[tds4u]

Here is the result: 4326: http://maps42.de:8444/service?LAYERS=OSMLIVE&FORMAT=image%2Fpng&SRS=EPSG%3A4326&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&TRANSPARENT=TRUE&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&STYLES=&BBOX=-212.6953125,-105.46875,212.6953125,105.46875&WIDTH=1210&HEIGHT=600

3857: http://maps42.de:8444/service?LAYERS=OSMLIVE&FORMAT=image%2Fpng&SRS=EPSG%3A3857&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&TRANSPARENT=TRUE&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&STYLES=&BBOX=-47354267.76322,-23481455.0892,47354267.76322,23481455.0892&WIDTH=1210&HEIGHT=600

[xan-der]

Perhaps narrowing down a bit.

The two attached files render the world very quickly and with no visible artifacts. The same backend, but rendering the same bounds via WMs through either OGCServer or MapProxy give the bad result that tds4u and I have been seeing.

When rendering via OGCServer or MapProxy, it takes a really long time (many minutes) whereas running the C++ or Python code renders in seconds.

mapniktests.zip

On Debian, I have to compile the C++ with this: $ gcc -std=c++11 -I /usr/local/include/mapnik -L /usr/local/lib -lmapnik -lstdc++ -licuuc basic.cpp

[mboeringa]

Hi all,

I have downloaded the "simplified_water_polygons.shp" shapefile today (31th May) from http://openstreetmapdata.com/data/water-polygons and ran it through a Repair Geometry operation in ArcGIS. ArcGIS detected over 1200 "empty geometries" and maybe a dozen "self-intersections".

I have attached the repaired shapefile in a ZIP. It also includes the logging text file output of the repair tool showing the records in error ("ArcGIS_Repair_Geometry_simplified_water_polygons.txt").

Can someone try this version? ArcGIS_Repair_Geometry_simplified-water-polygons-complete-3857.zip

I have also ran the shapefile of the simplified water polygons through QGIS 2.14.3 (Essen)'s Check geometry validaty tool. That reported a total of 2484 errors, but each error geometry seems to lead to a double error record in QGIS for this type of error, so the total number is actually quite consistent with ArcGIS's results. Also, at first glance, the error record numbers of the originally downloaded shapefile seem consistent between ArcGIS and QGIS, so that is another good sign of proper checks.

I have attached QGIS's log as ZIPed textfile QGIS_Check_geometry_validaty.zip

[xan-der]

And here's another variant to narrow down the problem. In these examples, Node.js is used for invoking Mapnik, with a locally generated image coming out good and one generated via WMS server failing.

Using node.js 4.4.5 and node-mapnik (from https://github.com/mapnik/node-mapnik), executing locally with a command like, $ nodejs genimg.js, the result is good:

The code to generate the image is:

var mapnik = require('mapnik');
var fs = require('fs');

// register fonts and datasource plugins
mapnik.register_default_fonts();
mapnik.register_default_input_plugins();

var map = new mapnik.Map(256, 256);
map.load('/tools/openstreetmap-carto/mapnik.xml', function(err,map) {
    if (err) throw err;
    merc_bbox = [-20037508.34, -20037508.34, 20037508.34, 20037508.34 ]
    map.zoomToBox(merc_bbox);
    var im = new mapnik.Image(256, 256);
    map.render(im, function(err,im) {
      if (err) throw err;
      im.encode('png', function(err,buffer) {
          if (err) throw err;
          fs.writeFile('map.png',buffer, function(err) {
              if (err) throw err;
              console.log('saved map image to map.png');
          });
      });
    });
});

Using node-mapnik-wms (from https://github.com/plepe/node-mapnik-wms, the result is:

The call to node-mapnik-wms was: http://192.168.218.219:8001/?LAYERS=__all__&STYLES=&FORMAT=image%2Fpng&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&SRS=EPSG:3857&BBOX=-20037508.34,-20037508.34,20037508.34,20037508.34&WIDTH=256&HEIGHT=256

The WMS server was the demo code provide with node-mapnik-wms running with openstreetmap-carto XML: ./server.js /tools/openstreetmap-carto/mapnik.xml 8001

The module node-mapnik-wms loads node-mapnik to do the rendering. In this example, only the C++ interface is used (eliminating the Python interfaces as the only problem). However, as with the Python interfaces to Mapnik, the WMS server code probably is doing some transforms under the hood.

[mboeringa]

By the way, I also did a quick check on the non-simplified water polygons. That turned up just over 400 'self-intersections' but no 'empty geometries'. I can't attach the repaired shapefile here, as it is to big as you all know.