search

mapbox / mapnik-vector-tile

Mapnik implemention of Mapbox Vector Tile specification
BSD 3-Clause "New" or "Revised" License
553 stars 117 forks source link

How to build mapnik vector tile on windows ? #195

Closed nouryf closed 8 years ago

nouryf commented 8 years ago

Hi,

I'm trying to buid Mapnik vector tile on Windows. I'm able to build 'mapnik' using guidelines posted at https://github.com/mapbox/windows-builds.

Is there is a how to steps to follow in order to build 'Mapnik with vector tile support' ?

Thanks Noury

springmeyer commented 8 years ago

@nouryf - It depends greatly on what you are trying to accomplish. The mapnik-vector-tile code can be used as a header only library and therefore there is nothing to build if you want to include it within another project. If you want to contribute to the mapnik-vector-tile code then you'd built its examples and tests. Is this what you are interested in doing? Can you share more details about your goals?

nouryf commented 8 years ago

Thanks @springmeyer for the quick response. My goal is to generate vector tiles from OSM data for offline display. Vector tiles provide more flexibility for styling and less disk size.

So, vector tiles are mostly headers and hence I can just include them in a C++ project linking the mapnik .lib and I should be able to produce some vector tiles ? Is there an example how to do it ?

Thanks again

springmeyer commented 8 years ago

@nouryf As a first step I recommend checking out https://www.mapbox.com/mapbox-studio-classic if you have not already. It is a GUI app with support for generating vector tiles locally (can dump them into an mbtiles) and provides a Windows version. You can plug in and extend something like https://github.com/kartotherian/osm-bright.tm2source for the complex transformations necessary for working with OSM data.

nouryf commented 8 years ago

Actually I did: I import OSM data of one country (France) into PostGIS database then using MapBox Studio I added that PostGIS database as new data source. Data were displayed ok but when I tried to export to a .mbtiles, it got stuck at 0.10 %. I imagine because the data are large (whole country). So I have no idea how long this would take (hours or days) but 0.1 % running one hour did not encourage me that it will finish in few hours.

Since this export looks like it's done in python and there is an extra overhead of IO PostGIS, I was wondering if I can make it faster by using C++ directly. Also I want to be able to streamline the process that I can just call it from a command line .exe.

I found this example here how to do it in C++ but I'm not sure how it would work since I'm in step of building 'Mapnik with vector support'.

Thanks

springmeyer commented 8 years ago

Mapbox Studio Classic uses this C++ code (mapnik-vector-tile). So it will be no faster as a command line program. Usually slow exports like this are due to Mapbox Studio Classic (and the underlying libraries, including mapnik-vector-tile - used via node-mapnik) waiting on PostGIS queries. So you will end up right back where you started: you will need to optimize PostGIS to return the data faster.

nouryf commented 8 years ago

Hmm.. Interesting. Thanks for saving me the trip. Any hints for PostGIS optimization on data imported using Osm2pgsql ?

If I create a datasource using some local .shp or .json the process of creating an mbtiles for a country should be much faster ? I just started doing this after I received your feddback about bottleneck being PostGIS queries, and I'm now using .geojson with one polygon covering coundaries of one country (France) then I'm exporting to .mbtiles. The rate of generated tiles per secons seems faster that using PostGIS. After 5mn its now 1% progress (Windows machine with 8 core and 16 GB RAM).

Would using local files (.shp or other formats) would be faster that using local PostGIS database ?

Thanks

nouryf commented 8 years ago

I think I still would like to be able to build mapnik with vector tile support. It provide me more flexibility to play with code and things. Is building vector support means that I need to copy the headers to mapnik 'src' folder and do a simple rebuild of mapnik ? Or there is more into it ?

Thanks

wilhelmberg commented 8 years ago

Would using local files (.shp or other formats) would be faster that using local PostGIS database ?

@nouryf Shapefiles, or any other file based formats will definitely be worse than PostGIS. With PostGIS you can optimize your data per zoom level, e.g. simplification or area.

I think it would really be a good idea to look into optimizing your PostGIS queries before you dig into writing a new vector tile creator.

The PostGIS manual might be a good starting point: https://www.mapbox.com/help/postgis-manual/#spatial-indexes https://github.com/mapbox/postgis-vt-util

Like @springmeyer mentioned above it might also be a good idea to look into the SQL tricks used in osm-bright, e.g. scale_denominator: https://github.com/kartotherian/osm-bright.tm2source/blob/master/data.yml#L146-L147

springmeyer commented 8 years ago

I'm now using .geojson with one polygon covering coundaries of one country (France) then I'm exporting to .mbtiles. The rate of generated tiles per secons seems faster that using PostGIS. After 5mn its now 1% progress (Windows machine with 8 core and 16 GB RAM).

What is your maxzoom setting? For country level data you should be able to get away with only rendering down to around z8-10 (and then still view the rendered tiles offline by "overzooming" several zoom levels higher). For street level data you'll need to render down to >= z15. This is a major undertaking at a global level which is why Mapbox provides the Mapbox Streets vector tileset (https://www.mapbox.com/developers/vector-tiles/mapbox-streets-v7/).

pnorman commented 8 years ago

Using shapefiles as a vector source is not generally a great idea because there is missing functionality for common vector tile requirements that is worked around with postgres/postgis magic (e.g. layers that change based on zoom level).

There is a lack of published examples of vector styles, and I have no idea how efficient Wikimedia's tm2source style is. Some estimates I had was 2-8 days for the world to z15.

If you're only worried about France, make extra sure that the software you're using only will generate vector tiles that intersect with France.

nouryf commented 8 years ago

Thank you @BergWerkGIS for your answer. I'll take a closer look at PostGIS queries optimization. To do this the flow would be:

Would the editing of the '.yml' done manually or there is an UI interface to put the sql command ? I don't mind doing it manually, just was curious if this can be done otherwise.

Export one country like France if PostGIS and queries are set properyl in the .yml would take approximately how many hours ? (Windows box 16 GB RAM and 8 cores). Would it be 6-10 hours or less or may be days ? Just a bulk idea for any one that did this before.

Thanks

nouryf commented 8 years ago

@springmeyer: I did not change the default zoom level when I added that country boundary dataset. That was my mistake trying to generate tiles without knowing that zoom ranges affect things and since we are dealing with vector, a higher zoom level for boundaries is not needed.

Using Mapbox Streets vector tileset is a good idea but first I want to be able first to generate own tiles for learning more about this and take route of using my created tiles or MapBox's.

Thanks

nouryf commented 8 years ago

Thanks #pnorman. I'll stick with PostGIS then.

wilhelmberg commented 8 years ago

Would the editing of the '.yml' done manually or there is an UI interface to put the sql command ?

No need to manually edit .yml. Mapbox Studio Classic would be the tool of choice.

See here how to create a new source project: https://www.mapbox.com/help/getting-started-studio/#start-source

nouryf commented 8 years ago

Thanks. I followed the steps and created a new postgis db and converted my dataset .osm.pbf as outlined in https://github.com/kartotherian/kartotherian/blob/master/README.md#in-depth-step-by-step

I followed all the steps but when I tried to open the .tm2source in MapBox Studio Classic I got an error about 'LabelGrid' arguments. The error was: _Postgis Plugin: ERROR: function labelgrid(geometry, integer, integer) does not exist LINE 51: ORDER BY labelgrid(way, 64, 0), sortorder DESC ^ HINT: No function matches the given name and argument types. You might need to add explicit type casts. in executeQuery Full sql was: 'SELECT * FROM (

When looking at it closely, the function as defined in the .sql expects two arguments but in the .yml its called with three arguments

[https://github.com/mapbox/postgis-vt-util/blob/master/postgis-vt-util.sql#L122-124]

2 arguments are expected.

In [https://github.com/kartotherian/osm-bright.tm2source/blob/master/data.yml#L471]

3 arguments are passed.

Are the two files (.sql and .yml) out of sync because I'm not using the right util .sql version ?

Thanks

p.s: I just filed an issue under the https://github.com/kartotherian/osm-bright.tm2source project: https://github.com/kartotherian/osm-bright.tm2source/issues/41

noury

springmeyer commented 8 years ago

Noury - great you are making progress. I'm going to close this issue now as we've moved far from the original question. Best to create issues at specific repositories of code, like you are doing.

nouryf commented 8 years ago

Thanks @springmeyer and every one for your help on this.