efficient rendering of multilingual maps

[artemp]

We (Wikipedia-people) try to render Openstreetmap in a lot of languages (nearly 270):

http://toolserver.org/~osm/locale/

The way with so many parallel mapnik styles is very inefficient and the new complete re-rendering for each language needs a lot of resources. So the idea is to make it more clever:

*Render a base-tile only once without all names that need a translation. Use something like Metawriter to save all names that are in different languages with there positions and save collision-areas (areas where no label should be).

*In the second step render only the names in the right language with the help of the metawriter data (without a new database request). Use the base-tile as background. Prevent collision with labels in the background and with other names.

*Store the final tile or if creation is very fast render it on each request.

The restart of mapnik needs with more than 270 styles in the moment nearly 15 minutes. So it would be better to have only one style with a language parameter. (But this point is not so important.)

After a talk with the creator of Metawriter, I know that this wish is not easy to fulfil, but perhaps somebody think that it is charming. There are a lot of use cases where multilingual maps are important (use it in schools for childrens, use Openstreetmap for orientation e.g. in asia, compare Openstreetmap with historial maps...).

Perhaps this is also a project-idea for GSoC next year, but it would be nice to get a earlier result because we still have the Openstreetmap map still in more than 5 Wikipedia language versions. [http://de.wikipedia.org/wiki/Hilfe:OpenStreetMap/en]

[artemp]

[Kolossos] Here an other idea to make the solution much easier. The problem is that we will have on higher zoomlevels a lot of meta tiles where is no name-tag in a specific language inside. So rendering and storage these tiles in different languages will be a waste of render-time and storagespace.

So the idea is to render the normal metatile in OSM-style and use metawriter to store a file with a list of languages that are inside the area of the tile. If later comes an request for this area in a specific language it is only to render if this language is in the list. If the language not in the list it would be enough to create a link-file (symlink) or use a script to deliver the standard-tile to the customer.

This second concept seem not so complex to include in mapnik but can save us a lot of space and server load.

[springmeyer]

/cc @ajashton, @artemp, @joto, @herm, and @kolossos.

I just read http://blog.jochentopf.com/2012-06-21-wikipedia-multilingual-maps-project.html, so I've learned that this project has some momentum, and I want to know how we can help make this work in Mapnik.

My sense is that metawriters are not an exact solution as they stand because they require changes to the stylesheet to work (see also #1240 which notes that their design is in question). However, there is a good python api to metawriters and text symbolizers in master, so maybe you could manipulate maps, in memory, and on the fly (after requests are made) change out the textsymbolizer/metawriter parameters, but this could be messy and is unlikely to scale well.

Here's a straw dog:

Perhaps a new renderer could be added to Mapnik that could be chained onto the main renderer (see #1254) and used to pre-seed tiles in two formats (one raster and one vector, to be re-rendered). In a single pass the main renderer could do its normal work rendering all things that do not relate to collision avoidance (lines,polygons,polygon patterns), offering some easy api to request no rendering of points/labels/shields (or perhaps the map could be pre-populated with a detector bbox that matches the entire map extent, leading to no placements: see #909).

Then a secondary renderer could be written, chained onto the first, that collects and can output all the specific features and styles relating to collisions (text,shield,point/marker - see #952) in a highly optimized (only pull feature attributes relevant to multilingual data), tiled format for later rendering (likely with a slight buffer). This could work without any changes to a map style and with only minimal additions to core mapnik code - it would just be a new Processor (https://github.com/mapnik/mapnik/blob/master/src/feature_style_processor.cpp#L92) requiring just a listing at https://github.com/mapnik/mapnik/blob/master/src/feature_style_processor.cpp#L648-658.

So then all requests for wikipedia maps could be for two tiles sources: one of the lines/polygons (and which will likely be aggressively pre-rendered for speed), and an second (points/labels/shields) which could be rendered on the fly for each language and cached on demand. The second tile could be rendered on top of the original tile (serverside) so that alpha in labels or halos respect the background pixels, then composited and returned (to allow a single tile uri to be exposed).

[alex73]

2 @springmeyer /cc @ajashton, @artemp, @joto, @herm, and @kolossos.

Since Mapnik 3.0 released, probably it's good time to implement this issue for 3.1 ?