search

mapnik / node-mapnik

Bindings to mapnik for node.js
http://mapnik.org/documentation/node-mapnik
BSD 3-Clause "New" or "Revised" License
530 stars 165 forks source link

Rendering from PBF to PNG, linestrings are very narrow #346

Closed StevenLooman closed 9 years ago

StevenLooman commented 9 years ago

I am rendering a PNG from a PBF vector-tile, using the OSM-Bright style. The PBF which is used to render the PNG, is also rendered using the OSM-Bright style.

The polygons render fine, as well al the labels. However, lines are very narrow, barely visible even. Attached is tile at zoom: 19, x: 269484, y: 172448: 19_269484_172448.

Rendering the PNG directly (from database to raster tile), the lines in the tile are as expected. (Not attached)

Is this normal behavior? I can imagine that the PBF does include line style info, therefore the source data already contains a narrower line. Apply a/the same stylesheet again, the line gets even thinner. Also, on the left of the tile, there should be a bridge, but none is rendered. Rendering the PNG directly, the bridge is shown.

Or am I missing some render options? None are given during the render-the-PBF call, and none are given during the render-the-PNG call.

Using node-mapnik 3.1.0.

springmeyer commented 9 years ago

Is this normal behavior?

No, I can't think of any good reason why rendering would be different from a vector tile vs from the database as long as the png encoding options are the same. Maybe try encoding to jpeg or webp as a quick check: do those encoding also show differences?

StevenLooman commented 9 years ago

The behavior is the same when encoding the tile as jpeg: 19_269484_172448

Script used to render the tile:

var mapnik = require('mapnik');
var mercator = new(require('sphericalmercator'));

mapnik.register_default_fonts();
mapnik.register_default_input_plugins();
var map = new mapnik.Map(256, 256);
map.loadSync('worker_input/OSMBright.xml', {'base': 'worker_input'});

var zoom = 19;
var x = 269484;
var y = 172448;
var vTile = new mapnik.VectorTile(zoom, x, y);
var rTile = new mapnik.Image(256, 256);

map.extent = mercator.bbox(x, y, zoom, false, '900913');
map.render(vTile, function(err, vTile) {
    vTile.render(map, rTile, function(err, rTile) {
        if (err) throw err;

        rTile.save('19_269484_172448.jpg', 'jpeg');
    });
});
StevenLooman commented 9 years ago

Extended test script to render either the raster tile from vector tile, or directly from database. Toggle the variable renderFromPbf to choose either option:

var mapnik = require('mapnik');
var mercator = new(require('sphericalmercator'));

mapnik.register_default_fonts();
mapnik.register_default_input_plugins();
var map = new mapnik.Map(256, 256);
map.loadSync('worker_input/OSMBright.xml', {'base': 'worker_input'});

var renderFromPbf = true;
var zoom = 19;
var x = 269484;
var y = 172448;
var vTile = new mapnik.VectorTile(zoom, x, y);
var rTile = new mapnik.Image(256, 256);

map.zoomToBox(mercator.bbox(x, y, zoom, false, '900913')); // changed from map.extent = ...

if (renderFromPbf) {
    map.render(vTile, function(err, vTile) {
        if (err) throw err;

        vTile.render(map, rTile, function(err, rTile) {
            if (err) throw err;

            rTile.save('19_269484_172448.pbf.png', 'png');
        });
    });
}

if (!renderFromPbf) {
    map.render(rTile, function(err, rTile) {
        if (err) throw err;

        rTile.save('19_269484_172448.png', 'png');
    });
}

The tile directly rendered from database looks like this: 19_269484_172448

springmeyer commented 9 years ago

Okay, thanks - so its not image encoding then. Next guess is that the filters may be matching differently leading to different styling to be applied. Can you try to whittle down the style further to something that still replicates the problem but it as simple as possible? Ideally whittle down such that no database is installed and use something like the CSV plugin for reading data: https://github.com/mapnik/mapnik/wiki/A-perfect-testcase

StevenLooman commented 9 years ago

Okay, will do. Might take some time.

StevenLooman commented 9 years ago

First of all, I've modified the OSM-Bright stylesheet to serve from SQLite instead of Postgresql/PostGIS. Transformation was done use ogr and the imposm mapping supplied with the project. Some queries were further materialized due to limitations of SQLite and the SQLite datasource driver of mapnik, others were not.

I have created a minimal stylesheet with inline CSV, see below. The interesting thing is that using the inline CSV, the tiles are rendered properly, i.e., lines with proper width. Using the same stylesheet, but with the SQLite datasource, the lines are rendered as before: hardly visible.

The SQLite database is around 4MB, 200KB compressed. As far as I know, I can not attach files to issues on Github. Can I mail them to you? Or is there a better way so that the data is preserved for future use/people having the same issue?

Next step, I'll try to find the differences between the two rendered vector tiles. Perhaps there is some kind of mismatch between (the types of) stored attributes of features?

Stylesheet used for rendering from CSV:

<Map srs="+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0.0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs +over" background-color="#c4dff6" maximum-extent="-20037508.34,-20037508.34,20037508.34,20037508.34">

<Style name="roads_high" filter-mode="first">
  <Rule>
    <MaxScaleDenominator>2500</MaxScaleDenominator>
    <MinScaleDenominator>500</MinScaleDenominator>
    <Filter>([stylegroup] = 'noauto')</Filter>
    <LineSymbolizer stroke-width="3.5" stroke-dasharray="3, 3" stroke-linejoin="round" stroke="#ffffff"/>
  </Rule>
  <Rule>
    <MaxScaleDenominator>2500</MaxScaleDenominator>
    <MinScaleDenominator>500</MinScaleDenominator>
    <Filter>([stylegroup] = 'minorroad')</Filter>
    <LineSymbolizer stroke-width="14" stroke-linecap="round" stroke-linejoin="round" stroke="#ffffff"/>
  </Rule>
  <Rule>
    <MaxScaleDenominator>400000</MaxScaleDenominator>
    <MinScaleDenominator>500</MinScaleDenominator>
    <LineSymbolizer stroke="#ffffff"/>
  </Rule>
</Style>
<Style name="roads_high-outline" filter-mode="first">
  <Rule>
    <MaxScaleDenominator>2500</MaxScaleDenominator>
    <MinScaleDenominator>500</MinScaleDenominator>
    <Filter>([stylegroup] = 'noauto') and ([bridge] = 1)</Filter>
    <LineSymbolizer stroke-width="9.5" stroke="#b5b5a6" stroke-linecap="butt" stroke-linejoin="round"/>
  </Rule>
  <Rule>
    <MaxScaleDenominator>2500</MaxScaleDenominator>
    <MinScaleDenominator>500</MinScaleDenominator>
    <Filter>([stylegroup] = 'noauto')</Filter>
    <LineSymbolizer stroke-width="9.5" stroke-linecap="round" stroke-linejoin="round" stroke="#e3e2d0"/>
  </Rule>
  <Rule>
    <MaxScaleDenominator>2500</MaxScaleDenominator>
    <MinScaleDenominator>500</MinScaleDenominator>
    <Filter>([stylegroup] = 'minorroad') and ([bridge] = 1)</Filter>
    <LineSymbolizer stroke-width="17.5" stroke="#b5b5a6" stroke-linecap="butt" stroke-linejoin="round"/>
  </Rule>
  <Rule>
    <MaxScaleDenominator>2500</MaxScaleDenominator>
    <MinScaleDenominator>500</MinScaleDenominator>
    <Filter>([stylegroup] = 'minorroad')</Filter>
    <LineSymbolizer stroke-width="17.5" stroke-linecap="round" stroke-linejoin="round" stroke="#e3e2d0"/>
  </Rule>
  <Rule>
    <MaxScaleDenominator>400000</MaxScaleDenominator>
    <MinScaleDenominator>500</MinScaleDenominator>
    <Filter>([bridge] = 1)</Filter>
    <LineSymbolizer stroke="#b5b5a6" stroke-linecap="butt" stroke-linejoin="round"/>
  </Rule>
  <Rule>
    <MaxScaleDenominator>400000</MaxScaleDenominator>
    <MinScaleDenominator>500</MinScaleDenominator>
    <LineSymbolizer stroke-linecap="round" stroke-linejoin="round" stroke="#e3e2d0"/>
  </Rule>
</Style>
<Layer name="roads_high" srs="+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0.0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs +over">
    <StyleName>roads_high-outline</StyleName>
    <StyleName>roads_high</StyleName>
    <Datasource>
       <Parameter name="type">csv</Parameter>
       <Parameter name="inline">
osm_id|osm_version|GEOMETRY|type|tunnel|bridge|access|render|stylegroup
7038038|2|LINESTRING(561197.676264 6856060.964059, 561086.835447 6856065.042414)|pedestrian|0|0||fill|noauto
7037930|4|LINESTRING(561086.835447 6856065.042414, 561085.299238 6856052.406804, 561087.859586 6856001.336568)|unclassified|0|0||fill|minorroad
7037945|4|LINESTRING(561196.117791 6856106.845666, 561091.176907 6856086.963604, 561086.835447 6856065.042414)|unclassified|0|0||fill|minorroad
38212533|2|LINESTRING(561081.458715 6856064.82393, 561086.835447 6856065.042414)|unclassified|0|0||fill|minorroad
223407868|2|LINESTRING(561087.102613 6855865.496776, 561067.343404 6855890.840255, 561064.616076 6855898.304944, 561056.912768 6855919.333557, 561043.554429 6855973.953587, 561040.214844 6856004.905102, 561040.148052 6856063.167099)|tertiary|0|0||fill|minorroad
       </Parameter>
    </Datasource>
  </Layer>

<Style name="bridge" filter-mode="first">
  <Rule>
    <MaxScaleDenominator>2500</MaxScaleDenominator>
    <MinScaleDenominator>500</MinScaleDenominator>
    <Filter>([render] = '2_line') and ([stylegroup] = 'noauto')</Filter>
    <LineSymbolizer stroke-width="6.5" stroke="#fcfbe7" stroke-linejoin="round"/>
  </Rule>
  <Rule>
    <MaxScaleDenominator>2500</MaxScaleDenominator>
    <MinScaleDenominator>500</MinScaleDenominator>
    <Filter>([render] = '3_inline') and ([stylegroup] = 'noauto')</Filter>
    <LineSymbolizer stroke-width="3.5" stroke-dasharray="3, 3" stroke-linejoin="round" stroke="#ffffff"/>
  </Rule>
  <Rule>
    <MaxScaleDenominator>2500</MaxScaleDenominator>
    <MinScaleDenominator>500</MinScaleDenominator>
    <Filter>([render] = '3_inline') and ([stylegroup] = 'minorroad')</Filter>
    <LineSymbolizer stroke-width="14" stroke-linecap="round" stroke-linejoin="round" stroke="#ffffff"/>
  </Rule>
  <Rule>
    <MaxScaleDenominator>2500</MaxScaleDenominator>
    <MinScaleDenominator>500</MinScaleDenominator>
    <Filter>([render] = '1_outline') and ([stylegroup] = 'noauto') and ([bridge] = 1)</Filter>
    <LineSymbolizer stroke-width="9.5" stroke="#b5b5a6" stroke-linecap="butt" stroke-linejoin="round"/>
  </Rule>
  <Rule>
    <MaxScaleDenominator>2500</MaxScaleDenominator>
    <MinScaleDenominator>500</MinScaleDenominator>
    <Filter>([render] = '1_outline') and ([stylegroup] = 'noauto')</Filter>
    <LineSymbolizer stroke-width="9.5" stroke-linecap="round" stroke-linejoin="round" stroke="#e3e2d0"/>
  </Rule>
  <Rule>
    <MaxScaleDenominator>2500</MaxScaleDenominator>
    <MinScaleDenominator>500</MinScaleDenominator>
    <Filter>([render] = '1_outline') and ([stylegroup] = 'minorroad') and ([bridge] = 1)</Filter>
    <LineSymbolizer stroke-width="17.5" stroke="#b5b5a6" stroke-linecap="butt" stroke-linejoin="round"/>
  </Rule>
  <Rule>
    <MaxScaleDenominator>2500</MaxScaleDenominator>
    <MinScaleDenominator>500</MinScaleDenominator>
    <Filter>([render] = '1_outline') and ([stylegroup] = 'minorroad')</Filter>
    <LineSymbolizer stroke-width="17.5" stroke-linecap="round" stroke-linejoin="round" stroke="#e3e2d0"/>
  </Rule>
  <Rule>
    <MaxScaleDenominator>400000</MaxScaleDenominator>
    <MinScaleDenominator>500</MinScaleDenominator>
    <Filter>([render] = '3_inline')</Filter>
    <LineSymbolizer stroke="#ffffff"/>
  </Rule>
  <Rule>
    <MaxScaleDenominator>400000</MaxScaleDenominator>
    <MinScaleDenominator>500</MinScaleDenominator>
    <Filter>([render] = '1_outline') and ([bridge] = 1)</Filter>
    <LineSymbolizer stroke="#b5b5a6" stroke-linecap="butt" stroke-linejoin="round"/>
  </Rule>
  <Rule>
    <MaxScaleDenominator>400000</MaxScaleDenominator>
    <MinScaleDenominator>500</MinScaleDenominator>
    <Filter>([render] = '1_outline')</Filter>
    <LineSymbolizer stroke-linecap="round" stroke-linejoin="round" stroke="#e3e2d0"/>
  </Rule>
</Style>
<Layer name="bridge" status="on" srs="+proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0.0 +k=1.0 +units=m +nadgrids=@null +wktext +no_defs +over">
    <StyleName>bridge</StyleName>
    <Datasource>
       <Parameter name="type">csv</Parameter>
       <Parameter name="inline">
osm_id|osm_version|GEOMETRY|type|bridge|access|render|layer|tunnel|stylegroup
38212532|2|LINESTRING(561044.055366 6856063.330961, 561081.458715 6856064.82393)|unclassified|1||1_outline|1|0|minorroad
38212532|2|LINESTRING(561044.055366 6856063.330961, 561081.458715 6856064.82393)|unclassified|1||2_line|1|0|minorroad
38212532|2|LINESTRING(561044.055366 6856063.330961, 561081.458715 6856064.82393)|unclassified|1||3_inline|1|0|minorroad
       </Parameter>
    </Datasource>
  </Layer>

</Map>
springmeyer commented 9 years ago

Great work narrowing this down - yes I also suspect that the sqlite plugin is not handling attributes the same or correctly. Can you create a github repo and check all the data into it so that I can run the testcase using the sqlite db?

StevenLooman commented 9 years ago

As expected, the data in the VTiles differ. The features in the VTile from the CSV datasoure contain more attributes than the features from the VTile from the SQLite datasource. See the dumps below.

I'll create a repo.

GeoJSON representation of the CSV VTile, first layer:

{
    "type": "FeatureCollection",
    "name": "roads_high",
    "features": [
        {
            "type": "Feature",
            "properties": {
                "access": "",
                "bridge": 0,
                "osm_id": 7038038,
                "osm_version": 2,
                "render": "fill",
                "stylegroup": "noauto",
                "tunnel": 0,
                "type": "pedestrian"
            },
            "geometry": {
                "type": "LineString",
                "coordinates": [
                    [
                        5.04066467285157,
                        52.3083027408992
                    ],
                    [
                        5.04032872617245,
                        52.3083103256016
                    ]
                ]
            }
        },
        {
            "type": "Feature",
            "properties": {
                "access": "",
                "bridge": 0,
                "osm_id": 7037930,
                "osm_version": 4,
                "render": "fill",
                "stylegroup": "minorroad",
                "tunnel": 0,
                "type": "unclassified"
            },
            "geometry": {
                "type": "LineString",
                "coordinates": [
                    [
                        5.04032872617245,
                        52.3083103256016
                    ],
                    [
                        5.04031497985125,
                        52.308240935776
                    ],
                    [
                        5.0403298996389,
                        52.3080587997783
                    ]
                ]
            }
        },
        {
            "type": "Feature",
            "properties": {
                "access": "",
                "bridge": 0,
                "osm_id": 7037945,
                "osm_version": 4,
                "render": "fill",
                "stylegroup": "minorroad",
                "tunnel": 0,
                "type": "unclassified"
            },
            "geometry": {
                "type": "LineString",
                "coordinates": [
                    [
                        5.04066467285157,
                        52.3084650942448
                    ],
                    [
                        5.04036778584123,
                        52.3084306557061
                    ],
                    [
                        5.04032872617245,
                        52.3083103256016
                    ]
                ]
            }
        },
        {
            "type": "Feature",
            "properties": {
                "access": "",
                "bridge": 0,
                "osm_id": 38212533,
                "osm_version": 2,
                "render": "fill",
                "stylegroup": "minorroad",
                "tunnel": 0,
                "type": "unclassified"
            },
            "geometry": {
                "type": "LineString",
                "coordinates": [
                    [
                        5.04028044641019,
                        52.3083090956499
                    ],
                    [
                        5.04032872617245,
                        52.3083103256016
                    ]
                ]
            }
        }
    ]
}

GeoJSON representation of the SQLite VTile, first layer:

{
    "type": "FeatureCollection",
    "name": "roads_high",
    "features": [
        {
            "type": "Feature",
            "properties": {
                "osm_id": "7038038",
                "osm_version": 2,
                "rowid": 1,
                "type": "pedestrian"
            },
            "geometry": {
                "type": "LineString",
                "coordinates": [
                    [
                        5.04066467285157,
                        52.3083027408992
                    ],
                    [
                        5.04032872617245,
                        52.3083103256016
                    ]
                ]
            }
        },
        {
            "type": "Feature",
            "properties": {
                "osm_id": "7037930",
                "osm_version": 4,
                "rowid": 2,
                "type": "unclassified"
            },
            "geometry": {
                "type": "LineString",
                "coordinates": [
                    [
                        5.04032872617245,
                        52.3083103256016
                    ],
                    [
                        5.04031497985125,
                        52.308240935776
                    ],
                    [
                        5.0403298996389,
                        52.3080587997783
                    ]
                ]
            }
        },
        {
            "type": "Feature",
            "properties": {
                "osm_id": "7037945",
                "osm_version": 4,
                "rowid": 3,
                "type": "unclassified"
            },
            "geometry": {
                "type": "LineString",
                "coordinates": [
                    [
                        5.04066467285157,
                        52.3084650942448
                    ],
                    [
                        5.04036778584123,
                        52.3084306557061
                    ],
                    [
                        5.04032872617245,
                        52.3083103256016
                    ]
                ]
            }
        },
        {
            "type": "Feature",
            "properties": {
                "osm_id": "38212533",
                "osm_version": 2,
                "rowid": 4,
                "type": "unclassified"
            },
            "geometry": {
                "type": "LineString",
                "coordinates": [
                    [
                        5.04028044641019,
                        52.3083090956499
                    ],
                    [
                        5.04032872617245,
                        52.3083103256016
                    ]
                ]
            }
        }
    ]
}
StevenLooman commented 9 years ago

See https://github.com/StevenLooman/node-mapnik_issue-346.

Just do a npm install and npm start and it will give you two files: actual.png and actual.pbf.png. These are rendered directly to PNG and rendered from PBF to PNG, respectively.

StevenLooman commented 9 years ago

@springmeyer How do I do install a debug-build of mapnik with node-mapnik? Using npm install mapnik --debug (after removing node-modules) seems to install the normal/non-debug version. Using Ubuntu 14.04. Also setting the log level to DEBUG does not seem to do anything.

springmeyer commented 9 years ago

How do I do install a debug-build of mapnik with node-mapnik?

Currently I only provide binaries for Release builds. Debug builds are in the plans but not soon.

StevenLooman commented 9 years ago

Ok, can I use a custom mapnik? Or let node-mapnik build mapnik with debug enabled?

springmeyer commented 9 years ago

Sure, if you:

springmeyer commented 9 years ago

have mapnik installed already

I should add: a very recent version of Mapnik 3.x (github master)

StevenLooman commented 9 years ago

Ok, thank you for the instructions. I had some problems due to another mapnik being install system wide, next to a custom build/install locally.

The mapnik debug messages seems to be ok. I'll investigate further later on.

For completeness, the mapnik output:

// rendering the vector tile
Mapnik LOG> 2014-11-14 21:03:21: sqlite_datasource: SELECT geometry,rowid,[osm_id],[osm_version],[rowid],[type] FROM layer_roads_high WHERE rowid IN (SELECT pkid FROM "idx_layer_roads_high_geometry" WHERE xmax>=561047.7876131928 AND xmin<=561124.2246414798 AND ymax>=6856019.252038882 AND ymin<=6856095.689067169)
Mapnik LOG> 2014-11-14 21:03:21: sqlite_datasource: SELECT GEOMETRY,rowid,[osm_id],[osm_version],[rowid],[type] FROM layer_bridge WHERE rowid IN (SELECT pkid FROM "idx_layer_bridge_GEOMETRY" WHERE xmax>=561047.7876131928 AND xmin<=561124.2246414798 AND ymax>=6856019.252038882 AND ymin<=6856095.689067169)
// rendering the raster tile from vector tile
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: Scale=0.298582
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: Start map processing bbox=box2d(561047.7876131928060204,6856019.2520388821139932,561124.2246414797846228,6856095.6890671690925956)
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: Start processing layer=roads_high
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: -- datasource=0x7fa988006918
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: -- query_extent=box2d(561047.7876131937373430,6856019.2520388886332512,561124.2246414788533002,6856095.6890671690925956)
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: Start processing style
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: End processing style
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: Start processing style
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: End processing style
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: End layer processing
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: Start processing layer=bridge
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: -- datasource=0x7fa988006dc8
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: -- query_extent=box2d(561047.7876131937373430,6856019.2520388886332512,561124.2246414788533002,6856095.6890671690925956)
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: Start processing style
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: End processing style
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: End layer processing
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: End map processing
Mapnik LOG> 2014-11-14 21:03:21: stroker: Destroy stroker=0x7fa9880067c0

// rendering the raster tile
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: Scale=0.298582
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: Start map processing bbox=box2d(561047.7876131928060204,6856019.2520388821139932,561124.2246414797846228,6856095.6890671690925956)
Mapnik LOG> 2014-11-14 21:03:21: sqlite_datasource: SELECT geometry,rowid,[bridge],[stylegroup] FROM layer_roads_high WHERE rowid IN (SELECT pkid FROM "idx_layer_roads_high_geometry" WHERE xmax>=561047.7876131928 AND xmin<=561124.2246414798 AND ymax>=6856019.252038882 AND ymin<=6856095.689067169)
Mapnik LOG> 2014-11-14 21:03:21: sqlite_datasource: SELECT geometry,rowid,[bridge],[stylegroup] FROM layer_roads_high WHERE rowid IN (SELECT pkid FROM "idx_layer_roads_high_geometry" WHERE xmax>=561047.7876131928 AND xmin<=561124.2246414798 AND ymax>=6856019.252038882 AND ymin<=6856095.689067169)
Mapnik LOG> 2014-11-14 21:03:21: sqlite_datasource: SELECT GEOMETRY,rowid,[bridge],[render],[stylegroup] FROM layer_bridge WHERE rowid IN (SELECT pkid FROM "idx_layer_bridge_GEOMETRY" WHERE xmax>=561047.7876131928 AND xmin<=561124.2246414798 AND ymax>=6856019.252038882 AND ymin<=6856095.689067169)
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: Start processing layer=roads_high
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: -- datasource=0x2302a20
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: -- query_extent=box2d(561047.7876131928060204,6856019.2520388821139932,561124.2246414797846228,6856095.6890671690925956)
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: Start processing style
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: End processing style
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: Start processing style
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: End processing style
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: End layer processing
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: Start processing layer=bridge
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: -- datasource=0x243c150
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: -- query_extent=box2d(561047.7876131928060204,6856019.2520388821139932,561124.2246414797846228,6856095.6890671690925956)
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: Start processing style
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: End processing style
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: End layer processing
Mapnik LOG> 2014-11-14 21:03:21: agg_renderer: End map processing
Mapnik LOG> 2014-11-14 21:03:21: stroker: Destroy stroker=0x7fa98c0067c0
StevenLooman commented 9 years ago

Actually, something strange is going on. In the first two queries (rendering the VTile), the columns geometry,rowid,[osm_id],[osm_version],[rowid],[type] are fetched from both layers/datasources.(Why does it need osm_id and osm_version? Never referred to...)

Later on, rendering the raster tile (PNG) directly, the columns geometry,rowid,[bridge],[stylegroup] (layer_roads_high) and the columns GEOMETRY,rowid,[bridge],[render],[stylegroup] (layer_bridge) are fetched. Also, there are 3 queries (probably due to 3 Style elements in the stylesheet, two for layer_roads_high, one for layer_bridge.

You can argue if you would need the same amount of queries for both the VTile and the PNG. The VTile has little (less) to do with styles when it is created. But you would assume the same columns to be fetched. So it seems that the Style is completely ignored when rendering the VTile.

I haven't really looked at any code yet. There are (at least) two ways to fix this: 1) Make sure the VTile render mechanism also looks at Style (preferred I think) 2) Provide the ability to force additional columns to be stored for Layers (kind of a hack.)

StevenLooman commented 9 years ago

I have poked around a bit and it seems the Style isn't used when determining the needed properties of the vector data. I'll check it out some more tomorrow, maybe have a patch ready. All this seems to happen in mapnik-vector-tile.

StevenLooman commented 9 years ago

Ok, I have commited a fix which works for me (and leaves the current unit tests unbroken, coincidentally.) See https://github.com/StevenLooman/mapnik-vector-tile/commit/f2de3b951e0ba36e77cff380e9ba0f10b3243690. Can you take a look at it?

If all is well, I will:

This shouldn't be tied to node-mapnik.

springmeyer commented 9 years ago

great work narrowing this down. Let's keep the issue here for now until it's totally clear where the problem is. Based on a quick glance at your fix that indicates to me that lay_desc.get_descriptors() is not returning the correct values for the sqlite db. If that is the case then the correct fix would be in the Mapnik sqlite plugin.

StevenLooman commented 9 years ago

Ok, I'll take a look at that when I have the time. Thank you for checking and pointing this out.

StevenLooman commented 9 years ago

Thinking about it, it this really correct? Should the plugin/datasource figure out which columns are needed by the rendered?

springmeyer commented 9 years ago

Thinking about it, it this really correct? Should the plugin/datasource figure out which columns are needed by the rendered?

Late here. I can comment more in the morning, but basically the idea is that all fields should be included in vector tiles. By design the mapnik-vector-tile creator does not and should not have to know about the styles that any set of vector tiles may later be rendered with. This is because the idea is one set of vector tiles might be useful for many different styles. So, the <Style> elements in a Mapnik XML are intentionally ignored during vector tile creation.

Now, obviously if you only want to target a single style and you want top performance then restricting the set of columns burned into a vector tile by detecting columns actually needed/used in a style would be great. But this is not something that is supported yet. I'm currently thinking that post-processing the vector tiles would be a better route than limiting columns at first creation.

So anyway, the problem here I think is that the SQLite plugin is not returning all the columns correctly. But I will try to take a closer look tomorrow.

Also, in the future I plan to move on https://github.com/mapnik/mapnik/issues/1881, which will help you limit columns that obviously should be thrown out (if they are never going to be used in a style ever) if you are not using the SQLite or PostGIS plugins.

StevenLooman commented 9 years ago

Good to hear that this approach is intended (vector tiles containing all data.) This is the way I actually need later on.

I'll take a look at the SQLite plugin now.

StevenLooman commented 9 years ago

Turns out the pragma statement does not always return a type for the column:

sqlite> PRAGMA table_info('layer_bridge');
cid         name        type        notnull     dflt_value  pk        
----------  ----------  ----------  ----------  ----------  ----------
0           osm_id      TEXT        0                       0         
1           osm_versio  INT         0                       0         
2           GEOMETRY    NUM         0                       0         
3           type        TEXT        0                       0         
4           bridge                  0                       0         
5           access                  0                       0         
6           render                  0                       0         
7           layer                   0                       0         
8           tunnel                  0                       0         
9           stylegroup              0                       0

From the SQLite documentation, about Type Affinity:

A column with affinity NONE does not prefer one storage class over another and no attempt is made to coerce data from one storage class into another.

Types individual columns in SQLite:

sqlite> select typeof(osm_id) osm_id, typeof(osm_version) osm_version, typeof(geometry) geometry, typeof(type) type, typeof(bridge) bridge, typeof(access) access, typeof(render) render, typeof(layer) layer, typeof(tunnel) tunnel, typeof(stylegroup) stylegroup from layer_bridge;
osm_id      osm_version  geometry    type        bridge      access      render      layer       tunnel      stylegroup
----------  -----------  ----------  ----------  ----------  ----------  ----------  ----------  ----------  ----------
text        integer      blob        text        integer     null        text        text        integer     text      
text        integer      blob        text        integer     null        text        text        integer     text      
text        integer      blob        text        integer     null        text        text        integer     text

Multiple steps are possible from here. For one, we could state this SQLite database is not usable with mapnik (or at least those columns.) Or an approach like the postgis datasource can be taken: I.e., doing a SELECT * FROM <table> LIMIT 0. The latter has my preference, of course!

springmeyer commented 9 years ago

i.e., doing a SELECT * FROM <table> LIMIT 0. The latter has my preference, of course!

This sounds very familiar. I feel like I tried that but can't recall right now why it did not work (at least in the same way as postgres). Perhaps it was that it failed with LIMIT 0 and instead needed LIMIT 1. Definitely worth a shot to revisit.

StevenLooman commented 9 years ago

Added a peace of code to sqlite_utils.hpp, static function sqlite_utils::table_info():

        std::ostringstream t ;
        t << "SELECT * FROM  " << table << " LIMIT 1";
        std::cout << "rs2 query: " << t.str() << std::endl;
        std::shared_ptr<sqlite_resultset> rs2(ds->execute_query(t.str()));
        std::cout << "rs2, is_valid: " << rs2->is_valid() << ", column_count: " << rs2->column_count() << std::endl;
        int count = rs2->column_count();
        rs2->step_next(); // this is required, column_type gets the value of that column in that row, not for the whole table
        for (int i = 0; i < count; ++i)
        {
            int field_type = rs2->column_type(i);
            std::string field_name = rs2->column_name(i);
            std::cout << "field_name: " << field_name << ", field_type: " << field_type << std::endl;
        }

Output:

field_name: osm_id, field_type: 3
field_name: osm_version, field_type: 1
field_name: GEOMETRY, field_type: 4
field_name: type, field_type: 3
field_name: bridge, field_type: 1
field_name: access, field_type: 5
field_name: render, field_type: 3
field_name: layer, field_type: 3
field_name: tunnel, field_type: 1
field_name: stylegroup, field_type: 3

Keeping this in mind from the SQLite3 API: If the SQL statement does not currently point to a valid row, or if the column index is out of range, the result is undefined.

This is useful, but a side note should be made: in SQLite, column_type() returns the type of that column for that specific row. SQLite is a lot less restrictive in terms of data types. You can store any type in any column of any row (probably not entirely that free, but you catch my drift.) So this might work, but can also be a cause of a future-hear-pulling-scenario.

What is your opinion, @springmeyer ? Implement the SELECT * FROM ... LIMIT 1 method, or try to make SQLite3 return the correct data from PRAGMA table_info and put a big warning in the documentation, while also giving a warning from within the plugin, when this is detected?

springmeyer commented 9 years ago

What is your opinion, @springmeyer ? Implement the SELECT * FROM ... LIMIT 1 method, or try to make SQLite3 return the correct data from PRAGMA table_info and put a big warning in the documentation, while also giving a warning from within the plugin, when this is detected?

Can you try this patch?

diff --git a/plugins/input/sqlite/sqlite_utils.hpp b/plugins/input/sqlite/sqlite_utils.hpp
index 06832f6..64d7ba3 100644
--- a/plugins/input/sqlite/sqlite_utils.hpp
+++ b/plugins/input/sqlite/sqlite_utils.hpp
@@ -616,7 +616,7 @@ public:
                     break;

                 default:
-                    MAPNIK_LOG_DEBUG(sqlite) << "detect_types_from_subquery: unknown type_oid=" << type_oid;
+                    desc.add_descriptor(mapnik::attribute_descriptor(fld_name, mapnik::String));
                     break;
                 }
             }

If table_info does return the field name, just without a type, then storing as a string should be enough. The type is auxiliary metadata and its not a huge deal if it is wrong. The important thing is that the column is known to exist.

StevenLooman commented 9 years ago

The function detect_types_from_subquery is never called, the table_info function is called. Anyway, if the code defaults to type mapnik::String, the tiles are rendered correctly.

Do you think this is good enough? Should I make a pull request at the mapnik repo?

springmeyer commented 9 years ago

Pull request would be great - containing a regression test of a very small sqlitedb we can use to add a test to make sure this never regresses. Thanks!

StevenLooman commented 9 years ago

See https://github.com/mapnik/mapnik/pull/2580 and https://github.com/mapnik/mapnik/issues/2579

springmeyer commented 9 years ago

really excellent work on this @StevenLooman - closed by your pull in https://github.com/mapnik/mapnik/pull/2580

StevenLooman commented 9 years ago

Thanks! Great work on node-mapnik and mapnik.