Vector Tiles

[wonder-sk]

QGIS Enhancement: Vector Tiles

Date 2020/01/22

Author Martin Dobias (@wonder-sk)

Contact wonder-sk at gmail dot com

maintainer @wonder-sk

Version QGIS 3.14

Summary

This QEP proposes addition of native client support of Vector Tiles to QGIS: to be able to load and display vector tiles in map canvas within QGIS.

Overview

Feel free to skip this section if you are already familiar with concepts of vector tiles.

About Vector Tiles

Vector tiles are a somehow similar to raster map tiles: map data are stored within regular tiles of multiple zoom levels. The crucial difference to raster tiles is that each tile contains encoded vector data instead of a raster image (e.g. in PNG or JPG format). Individual tiles therefore contain collections of vector features (each with a geometry and a set of attributes). Most of the time, source vector data get clipped, filtered and simplified before converted into vector tile format. Thanks to these optimizations during tile creation, clients can display data faster without having to spend time to simplify geometries or filter out irelevant data.

Vector tiles do not have any styling information assigned so clients need to apply a cartographic style in order to display the data. This is in contrast to raster tiles which already contain the "final" images as desired by authors of raster tiles.

Vector tiles are becoming increasingly common in GIS world. There are several reasons for that:

1. Greater flexibility of displaying tiled maps. For example, it is possible to rotate map and still have labels horizontally oriented to improve readability. It is possible to enable/disable some elements on the map (or define multiple completely different styles) for a single set of input vector tiles without having to generate multiple sets of tiles.
2. Lower data storage requirements. Vector tiles need fewer zoom levels compared to raster tiles without degradation of map quality. Fewer zoom levels mean orders of magnitude less storage used by tiles.
3. Lower cost of producing vector tiles. Compared to raster tiles, producers of vector tiles do not need to do expensive map rendering as this is done on the client side.

Vector Tile Content

A single vector tile contains features from one or more layers. Authors of vector tiles normally publish schema of their tiles: what layers to expect, what are geometry types and fields (attributes) of those layers. Some layers may be only published in particular zoom levels or some filtering may have been applied to layers based on zoom level.

Different producers use different schemas. Few examples:

• OpenMapTiles
• Mapbox Streets, Mapbox Terrain
• Mapzen Tilezen
• Thunderforest Transport, Thunderforest Outdoors

Encoding of Vector Tiles

There are few formats that have been adopted by the industry so far:

• Mapbox Vector Tiles (MVT) - most popular format nowadays, based on Protobuf binary encoding. It is both quite efficient and fast to decode.
• GeoJSON - the main advantage is that it is human readable, but more verbose.
• TopoJSON - an extension of GeoJSON that encodes topology. It is more complex than GeoJSON and generally not that much used

Vector Tiles Data Sources

Given that vector tiles have pretty much the same high-level structure of tilesets like raster map tiles, the same protocols/containers are used:

• HTTP - XYZ requests - e.g. http://www.example.com/2/0/1.pbf to access tile with at zoom level 2, with coordinates X=0, Y=1
• local files in a directory structure: /mypath/Z/X/Y.pbf
• MBTiles file - SQLite database that contains tile data stored in blob columns in a dedicated table

Styling of Vector Tiles

As mentioned above, a single vector tile may contain data from several different vector sub-layers, so any definition of styling for multiple vector layers would do the job. There are various ways to define styling in existing clients that support vector tiles - in fact, every rendering engine uses a completely different format:

• OpenLayers and Leaflet do styling with code in JavaScript (see OpenLayers.VectorTileLayer and Leaflet.VectorGrid)
• Mapbox GL uses Mapbox GL style (JSON)
• Tangram uses Tangram scene file (YAML)
• Mapnik uses XML or CartoCSS
• GeoServer uses SLD (and others through extensions)

Related OGC Standards

To my best knowledge, there are no existing OGC standards regarding vector tiles. But there has been an ongoing effort within OGC to come up with standards for vector layers:

• integration into APIs: OGC API - Features, OGC API - Tiles, OGC API - Styles
• metadata - how to describe a vector tiles store (tiling scheme, sub-layers, styles etc.)
• storage of vector tiles in GeoPackage
• filtering language - for clients to be able to request filtered data
• styling

https://www.opengeospatial.org/projects/initiatives/vtp2

Creation of Vector Tiles

There are plenty of open source tools around that are able to produce vector tiles - best check out the Awesome Vector Tiles list. There are various vector tiles servers, CLI tools and libraries for that. It is even possible to create tiles directly from PostGIS >= 2.5.

Adding support for generation of vector tiles from within QGIS (e.g. as a part of Processing framework) is out of scope for this QEP.

Existing Solutions for QGIS

There is an existing plugin for QGIS that allows display of vector tiles: Vector Tiles QGIS plugin.

It supports tiles in MVT format and they may be coming from a remote tile server (XYZ) or from a local source (either MBTiles database or a directory with files). The plugin creates a single vector layer in the current QGIS project for each sub-layer within tiles and updates data of those vector layers whenever the map extent changes. It also supports reading of Mapbox GL style (in JSON format) and transforms it into QGIS styling thanks to the mapbox-gl-style-to-qgis project (which produces .qml styles for sub-layers).

Few details about the implementation:

• tile data get parsed to GeoJSON either via a pure python module from mapbox or (if available for the platform) through a compiled binary (which is faster)
• GeoJSON data from tiles are cached in temporary files (loaded/saved with python's pickle module)
• on change of map view extent, tiles from data are grouped into temporary GeoJSON files - these GeoJSON files are the sources of individual vector layers using OGR provider (one for each sub-layer)
• tile data are loaded in a worker thread

The existing approach has various issues:

1. updates of map canvas are relatively slow even with basic styling
2. the mechanism does not work outside of the main map canvas (print layouts, extra 2D and 3D map views only get "snapshot" from the extents of the main map canvas)
3. multiple vector layers need to be created, cluttering the project
4. python plugin is not available out of the box and it does not provide API

Proposal

We would like to propose creation of a new type of map layer: Vector Tile Layer. The rest of this section tries to add some background why this may be a good choice :-)

After having thought about how vector tiles fit into QGIS architecture, it does not seem right to use vector layers for tiled vector data. The main reason is that vector tiles actually consist of multiple sub-layers: in a single tile, there may be a mixture of sub-layers each with a different geometry type and a different set of fields. The other option (which seems inferior) would be to split a single vector tile data source into multiple vector layers, so that each sub-layer corresponds to a single QGIS vector layer - just like the way the QGIS Vector Tiles Reader plugin does it.

A big problem with representation of vector tiles as vector layers is that they are not "proper" vector layers. Vector tiles are data that have been modified for efficient visualization, making compromises elsewhere:

• Geometries have been split, clipped, simplified, features may have been filtered out altogether at some zoom levels.
• Having a polygon in one vector tile, we do not know if that is the complete representation of the feature or if the feature continues in some adjacent (or distant!) tiles.
• We cannot find a feature by its ID - we would need to browse through all tiles.
• We do not know how many features there are or cannot show attribute table of the whole sub-layer without fetching all tiles.

All of these limitations of vector tiles come from the idea to make their rendering as fast as possible. We should not try to use them for other than display purposes. It is better to think of the data encoded in vector tiles as raw data for map renderer, not "true" vector data for analysis or other common GIS uses.

For example, manual editing of geometries in vector tiles does not seem like a well-defined task: how would we propagate changes to other zoom levels? With a new map layer type, we do not need to worry about all that -it would not be possible to use ordinary vector layer tools to interact with vector tile layers (unless they are explicitly supported).

Another reason against multiple vector layers for a vector tile layer is that it makes organization of a project more messy from user's point of view... What was meant as a basemap layer becomes a group of inter-dependent layers. It also adds complexity to the internal code because multiple vector layers may need to fetch/access single vector tile's data simultaneously.

Data Sources and Formats

For the initial implementation we propose to have support for vector tiles through:

• remote sources (HTTP/S) - with XYZ template - type=xyz&url=http://example.com/{z}/{x}/{y}.pbf
• local files - with XYZ template - e.g. type=xyz&url=file:///path/to/tiles/{z}/{x}/{y}.pbf
• local MBTiles database - e.g. type=mbtiles&url=file:///path/to/file.mbtiles

To start with, only MVT encoding would be supported as it seems to be the industry standard nowadays (even ESRI have adopted it). Other encodings such as GeoJSON may be added later if needed.

We would initially support the usual Web Mercator based XYZ tiling scheme, but it should not be difficult to support any other CRS later.

For styling, we should use existing QGIS 2D map rendering engine. To help users start with vector tiles, we would like to add support for import of Mapbox GL JSON style files which seem to be quite popular by vector tiles users as they support reasonably complex map styling, have been around for some time and there is also an open source project for their creation: Maputnik

User Interface

It should be possible to add a vector tiles source either through browser dock or by drag'n'drop to QGIS in case of MBTiles file. For addition of local and remote vector tile sources using XYZ template (e.g. file:///home/martin/tiles/{z}-{x}-{y}.pbf or http://example.com/{z}/{x}/{y}.pbf) there should be a dialog to add such connections (together with some more details such as max. zoom level and others). Browser dock widget should have another root item (e.g. "Vector Tiles") and within it there should be a list of defined connections.

Once a vector tile layer is loaded, its layer styling widget should show a list of sub-layers - for each sub-layer it will be possible to assign a renderer (a subclass of QgsFeatureRenderer object).

Implementation

A new QgsVectorTileLayer class derived from QgsMapLayer would be added. It will implement 2D map rendering which would fetch vector tiles from the data source, decode content of each tile and do rendering of each tile separately (clipped to that tile's extent) using standard QGIS vector rendering capabilities.

The new layer class should provide API to:

• get list of sub-layers and their schemas (geometry type, list of fields)
• read decoded content of a single vector tile
• get/set feature renderers for sub-layers
• get access to the tiling scheme

Sub-layers (identified by their unique layer names) of vector tiles may be figured out from metadata (e.g. in case of MBTiles source) or discovered on the fly as vector tiles are read.

Caching: tiles fetched from a remote source would be catched like any other network content (e.g. raster tiles). A limited number of decoded tiles would be kept in an in-memory cache to further speed up rendering.

Merging of Vector Tiles

Most of the time, QGIS vector rendering works fine if the input data are tiled and the results look the same as if they weren't tiled. But there are some advanced effects where the rendered output would be different if the input geometries are clipped to tiles. For such use cases, we should have an option to enable merging vector tiles together before rendering (off by default!). Its use should be discouraged though, because that adds a lot of extra processing.

Side Thought on Raster Tiles

How about raster data - should we also have a new raster tile layer for those? The answer is... maybe :-)

Currently we have native support for tiled raster data sources within WMS provider - it can handle WMTS or request raster tiles in XYZ format either through HTTP or from a local directory or MBTiles file. Most of the time it works fine, because tiled raster data are not that different from non-tiled raster data - the main difference is that we just need to determine zoom level which will be used for particular scale. And even with untiled rasters there is a well-known concept of raster overviews (pyramids) which are scaled down versions of the original layer, serving as an equivalent of raster tiles at lower zoom levels.

So with tiled raster layers there are just minor annoyances such as being unable to access the tiling scheme through API, being unable to force a particular zoom level for raster block request or somewhat more complex implementation of WMS provider that mixes tiled and untiled approaches.

Whatever is the right solution for raster tiles, this QEP is only focused on vector tiles and does not propose any changes to the raster tiles implementation.

Outlook for Future

Vector tiles from OpenStreetMap for the whole world require less than 100 GB. In theory we could have a vector tiles server run by QGIS.org to provide all our users with a decent base map that they could even style by themselves if needed!

If the experience with vector tiles will be good, in the future we may think of more thorough use of them for 2D map rendering even for data sources that are not tiles:

• the rendering engine could internally build a cache of vector tiles for some/all vector layers to speed up subsequent redraws (avoiding the need to clip and simplify geometries every time)
• if we ever decide to move parts of 2D map rendering to GPU, vector tiles would be likely at the heart of the implementation

Backwards Compatibility

Not applicable

Votes

(required)

[elpaso]

Nice!

I've been thinking at vector tiles for QGIS Server too, but that's a complete different story.

I'm curious about the GPU rendering (for the future), can you please elaborate on that idea and how it crosses with the vector tiles concept?

[NathanW2]

Neat!

On Wed., 22 Jan. 2020, 7:26 pm Alessandro Pasotti, notifications@github.com wrote:

Nice!

I've been thinking at vector tiles for QGIS Server too, but that's a complete different story.

I'm curious about the GPU rendering (for the future), can you please elaborate on that idea and how it crosses with the vector tiles concept?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/qgis/QGIS-Enhancement-Proposals/issues/162?email_source=notifications&email_token=AAC5FXH43GJNARQILJQQ6VDQ7AGNRA5CNFSM4KKCLAX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJS25UA#issuecomment-577089232, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC5FXHG2KGJ4S24FDSSKHTQ7AGNRANCNFSM4KKCLAXQ .

[pblottiere]

I've been thinking at vector tiles for QGIS Server too

:heart:

[nirvn]

Great to see this moving ahead.

[haubourg]

Great proposal!

[nyalldawson]

All of these limitations of vector tiles come from the idea to make their rendering as fast as possible. We should not try to use them for other than display purposes. I

So to confirm -- these would be blocked for use from tools like processing?

Once a vector tile layer is loaded, its layer styling widget should show a list of sub-layers - for each sub-layer it will be possible to assign a renderer (a subclass of QgsFeatureRenderer object).

Also would need to consider labeling here

It will implement 2D map rendering which would fetch vector tiles from the data source, decode content of each tile and do rendering of each tile separately (clipped to that tile's extent) using standard QGIS vector rendering capabilities.

Something I'm not clear on -- would this be done using iterators? If so, would a new type of feature request be added? Can you expand further on the proposed API here?

Most of the time, QGIS vector rendering works fine if the input data are tiled and the results look the same as if they weren't tiled.

Possibly for points/lines, but only very very basic polygon styling (solid fill without stroke) would work for tiled vectors

Question -- for rendering, would a similar approach as WMTS be used, where existing cached tile levels are used while the correct tile level is being fetched in the background?

[wonder-sk]

@elpaso I'm curious about the GPU rendering (for the future), can you please elaborate on that idea and how it crosses with the vector tiles concept?

Generally the idea is to use the similar approach like in Mapbox GL or in Tangram web map clients: have all of the map rendered on GPU (talking about OpenGL or one of its successors) - a bit similar to how 3D rendering is done. QGIS styling is quite complex compared to those web map clients and many things are hard to do entirely on the GPU, so we may need to still do some parts of the rendering on the CPU and use those partial outputs as textures. It would be a long way there, with lots of tears, but in the end we could have a 2D map renderer capable of 60 frames per second with super smooth zooming/panning etc. :-)

[nyalldawson]

It would be a long way there, with lots of tears, but in the end we could have a 2D map renderer capable of 60 frames per second with super smooth zooming/panning etc. :-)

It's certainly a desirable endpoint, but we've got to be careful not to lose the ability to export full vector versions of layers too (e.g. pdf output of layouts). We have the biggest demands out of any project here -- a super fast interactive canvas renderer + extremely high quality vector based print outputs :scream:

[tomchadwin]

we've got to be careful not to lose the ability to export full vector versions of layers

Could the proposed "merge features" option be used in such instances?

[tomchadwin]

Most of the time, QGIS vector rendering works fine if the input data are tiled and the results look the same as if they weren't tiled.

Possibly for points/lines, but only very very basic polygon styling (solid fill without stroke) would work for tiled vectors

I wasn't aware of this issue. Is this why Mapbox GL doesn't support both fill and outline on polygons? I believe one has to duplicate the layer if one needs both.

[tomchadwin]

@mnboos @sfkeller

[wonder-sk]

@nyalldawson

So to confirm -- these would be blocked for use from tools like processing?

Yes. I mean blocked to be used interchangeably with vector layers. But of course if some tools would explicitly support vector tile layers, then there's no problem. I can also see a processing tool "extract from vector tiles" that would take a particular sub-layer and create a regular vector layer for it.

Also would need to consider labeling here

Yes - in fact I have labeling included in my prototype already, just forgotten to add it to the QEP. Every sub-layer should be also allowed to have labeling assigned in addition to the feature renderer.

Something I'm not clear on -- would this be done using iterators? If so, would a new type of feature request be added? Can you expand further on the proposed API here?

So far I was thinking about something very simple. Very roughly, if each decoded tile would be something like QHash<QString, QVector > where the hash table keys are sub-layer names, we could even directly return a vector of features for simplicity. But having a feature iterator can't hurt...

Possibly for points/lines, but only very very basic polygon styling (solid fill without stroke) would work for tiled vectors

Actually it works with polygons with outline too. Normally the generators of tiles do not cut polygons right at the tile boundary - they let geometries bleed out of the tile because there's the assumption that rendering will be clipped to the vector tile's extent. So it is really just the advanced rendering styles where the tiling would cause issues.

Question -- for rendering, would a similar approach as WMTS be used, where existing cached tile levels are used while the correct tile level is being fetched in the background?

Ideally yes. Maybe the initial implementation will not be that smart, but there is nothing that would stop us to do that...

[nyalldawson]

I can also see a processing tool "extract from vector tiles" that would take a particular sub-layer and create a regular vector layer for it.

Cool idea!

Actually it works with polygons with outline too. Normally the generators of tiles do not cut polygons right at the tile boundary - they let geometries bleed out of the tile because there's the assumption that rendering will be clipped to the vector tile's extent. So it is really just the advanced rendering styles where the tiling would cause issues.

Wouldn't that mean that you'd potentially get overlapping features? Say a polygon crossing the tile boundary, styled with a 50% opacity. If we aren't clipping them then you'd be rendering two features when there should only be one, and if we DO clip them to the tile rect then you'd get the phantom outlines/rendering artefacts...

[wonder-sk]

Wouldn't that mean that you'd potentially get overlapping features? Say a polygon crossing the tile boundary, styled with a 50% opacity. If we aren't clipping them then you'd be rendering two features when there should only be one, and if we DO clip them to the tile rect then you'd get the phantom outlines/rendering artefacts...

In my prototype I have been using QPainter's setClipRect or setClipRegion (if tiles were reprojected) and haven't seen any artifacts so far, even for non-square tiles with reprojection. Both versions of clipping functions used map coordinates rounded to whole numbers, I have not tried with QRectF. It was my hope that Qt would have consistent clipping, so if a clipping line passes through a pixel, we would get the correct amount of color on each side, adding up to the correct final color value.

[wonder-sk]

@tomchadwin Is this why Mapbox GL doesn't support both fill and outline on polygons? I believe one has to duplicate the layer if one needs both.

Not really. That's simply the way how you define rendering rules in Mapbox GL styling. For example, rendering a road linestring with an outline would also require two separate rules in Mapbox GL styling - in QGIS one just defines a single symbol with two symbol layers.

[anvpires]

+1 Great proposal!

[luipir]

My concerns are:

• how much time will pass to have user issues trying to create buffers of network analysis on vector tiles! there are already user trying to do raster processing on WMS services :/ e.g. should be clear that is not a format for (local) analysis but only for representation and map generation. Or if a processing capability would be available should be details in the QEP.
• Another important architectural point (at least for me), If a representation layer is introduced should be abstracted. Now we have mapbox, tomorrow what else?

I would more in favour to remove limitation to the current plugin and leave it outside the core. BTW because I'm more focused on qgis for analysis more than data representation (that is a great % of the use of qgis) I leave to them the evaluation of the impact and importance of this new feature.

[wonder-sk]

@luipir

how much time will pass to have user issues trying to create buffers of network analysis on vector tiles! there are already user trying to do raster processing on WMS services :/ e.g. should be clear that is not a format for (local) analysis but only for representation and map generation. Or if a processing capability would be available should be details in the QEP.

The QEP suggests that a new layer type would be introduced, so users would not be allowed to run ordinary processing tools on vector tiles. So this should not really be a problem.

Another important architectural point (at least for me), If a representation layer is introduced should be abstracted. Now we have mapbox, tomorrow what else?

I have tried to separate the concepts of vector tiles and anything related to Mapbox specs. To clarify - the abstraction is that a single vector tile = a collection of features (ID+geometry+attributes) from multiple vector sub-layers. This abstraction seems to be generally accepted and I have not seen any other abstractions / interpretations anywhere else. If there would be other standards appearing, e.g. a different encoding would be preferred instead of MVT, or a GeoPackage would become a standard container for vector tiles, there's nothing that would stop us using them instead of whatever we use now...

[wonder-sk]

@luipir I would more in favour to remove limitation to the current plugin and leave it outside the core.

Do you have any thoughts on how to remove the limitations of the current plugin? Because I can't really see a way how to fix them without adding the support directly to qgis_core and rendering using a single map layer...

[luipir]

Do you have any thoughts on how to remove the limitations of the current plugin?

nope! there weren't reference to the complexity to focus on the plugin. I never read it, so mine was just a speculation. I've no evaluated how an API can be introduced to the plugin (without having multi thread rendering issues)

[mtravis]

Really good proposal. Good luck!

On Wed, 22 Jan 2020, 12:34 Luigi Pirelli, notifications@github.com wrote:

Do you have any thoughts on how to remove the limitations of the current plugin?

nope! there weren't reference to the complexity to focus on the plugin. I never read it, so mine was just a speculation. I've no evaluated how an API can be introduced to the plugin (without having multi thread rendering issues)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/qgis/QGIS-Enhancement-Proposals/issues/162?email_source=notifications&email_token=AANXIRQGROYZ2IVJQMVGPM3Q7A4MHA5CNFSM4KKCLAX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJTMGNI#issuecomment-577159989, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANXIRSBXPUYYCO7XFQWLM3Q7A4MHANCNFSM4KKCLAXQ .

[nyalldawson]

In my prototype I have been using QPainter's setClipRect or setClipRegion (if tiles were reprojected) and haven't seen any artifacts so far, even for non-square tiles with reprojection. Both versions of clipping functions used map coordinates rounded to whole numbers, I have not tried with QRectF. It was my hope that Qt would have consistent clipping, so if a clipping line passes through a pixel, we would get the correct amount of color on each side, adding up to the correct final color value.

Ah - my misunderstanding, sorry. I thought you were referring to clipping the geometries, not the paint surface. I can't see any issues with this approach.

One thought -- maybe we should make the geometry unioning the automatic default behaviour whenever a layout is exported to a vector based format?

[timlinux]

I would more in favour to remove limitation to the current plugin and leave it outside the core.

@luipir the problem with that is we miss out on one of the strongest use cases for vector tiles - providing tiles for mobile apps (QField / Input).

[timlinux]

@wonder-sk What a great proposal!

I have a bunch of questions:

1) For supported data sources - what about being able to retrieve tiles straight from PostGIS? Or do we already get that with the list of data source schemes you propose? 2) For ingesting MapBox GL Styles - could we also export styles? I've used Maputnik and I think (totally biased of course) that QGIS is a far nicer way to define styles... 3) For vector renderers like classified renderers won't there also be some limitations since we can't easily know what the domain list for a feature type is in order to assemble a set of unique values? 4) For the vector tile sub layers, we should expect the list of sublayers to change as you zoom in and out right? How will we plan our cartography rules? Will we be able to see a list of all the potential sublayers and their geometry types? 5) Regarding you comments on attributes etc. Can we assume that identifying features will be supported, attribute tables not? 6) Do I remember correctly that for 3D stuff we need vector tiles in TopoJSON format? Is it worth considering supporting TopoJSON (maybe in the future) to support that use case?

Anyway your QEP fills me with happiness, can't wait to try this out!

@elpaso BTW for server component what about integrating Tegola (https://tegola.io) some how?

[nyalldawson]

BTW for server component what about integrating Tegola (https://tegola.io) some how

It's quite possibly nih syndrome, but I really don't think we want to introduce another language (go) into the qgis stack...

[mnboos]

Do you have any thoughts on how to remove the limitations of the current plugin?

nope! there weren't reference to the complexity to focus on the plugin. I never read it, so mine was just a speculation. I've no evaluated how an API can be introduced to the plugin (without having multi thread rendering issues)

@wonder-sk Nice proposal!

Hi all, I'm the auther or the plugin and am excited to see this QEP! The plugin has indeed it's limitations, some of them which one could probably solve, but also some quirks, especially in the interaction between the plugin and QGIS, i.e. unexpected crashes related to multi-threading, which are "somewhat hard" to debug.

By developing the plugin we brought a web-technology available to desktop clients and from what it seems, there is some demand in that. In my opinion now would be a good time to make a transition from the plugin to a QGIS integrated solution.

However, I'm asking myself what would be a good solution to handle the styling. The current style convert which I developed is far from complete but I assume, you don't want to build some Mapbox GL Style converter, which would probably have to be updated quite regularly, into QGIS. Maybe make QGIS fully capable of handling Vector Tiles and leave the styling to other tools?

[wonder-sk]

@nyalldawson One thought -- maybe we should make the geometry unioning the automatic default behaviour whenever a layout is exported to a vector based format?

That sounds very reasonable!

[tomchadwin]

For ingesting MapBox GL Styles - could we also export styles?

This is what Victor's bridge-style does - it's what qgis2web uses to convert QGIS styles to Mapbox styles. https://github.com/GeoCat/bridge-style

I'm asking myself what would be a good solution to handle the styling. The current style convert which I developed is far from complete but I assume, you don't want to build some Mapbox GL Style converter, which would probably have to be updated quite regularly, into QGIS. Maybe make QGIS fully capable of handling Vector Tiles and leave the styling to other tools?

Importing style could be out of scope. Esri styles aren't imported when one open a shapefile, and one has to use a dedicated tool such as Nyall's SLYR. I wouldn't say that the Mapbox style spec changes "quite regularly", though, so if this proposal or another tool implemented style import/conversion, I don't think it would be a huge burden to keep up-to-date.

bridge-syle has been built to support MB-to-QGIS conversion, but it's only stubbed out at the moment. That does mean it could be an alternative to implementing the style import in core.

[wonder-sk]

@timlinux

1. For supported data sources - what about being able to retrieve tiles straight from PostGIS? Or do we already get that with the list of data source schemes you propose?

I have not thought about such feature. I have planned for local and remote (HTTP/S) access through XYZ template + local access through MBTiles. If you have a PostGIS database available, it's probably easier to directly load the vector layers. But I may be missing your point :)

2. For ingesting MapBox GL Styles - could we also export styles? I've used Maputnik and I think (totally biased of course) that QGIS is a far nicer way to define styles...

In theory yes, but I have not planned that (at least not for the initial implementation). My main goal with support of Mapbox GL import was to have some nice styles to start with when dealing with vector tiles as starting from scratch is a lot of work. But I can see the value of export as well, in case people want to use the styling on the web...

3. For vector renderers like classified renderers won't there also be some limitations since we can't easily know what the domain list for a feature type is in order to assemble a set of unique values?

Possibly yes. Maybe we could simply use a random sample of vector tiles or the ones currently displayed in the main map canvas and extract unique values / ranges just from those.

4. For the vector tile sub layers, we should expect the list of sublayers to change as you zoom in and out right? How will we plan our cartography rules? Will we be able to see a list of all the potential sublayers and their geometry types?

Right, not all sub-layers may be available on all zoom levels. I would suggest that we would still see a list of all sub-layers ever seen and possibly even allow users to define styling for sub-layers we have not seen yet.

5. Regarding you comments on attributes etc. Can we assume that identifying features will be supported, attribute tables not?

Identifying features can be done, display of attribute table could be allowed too, but with some limitations - e.g. you would have to pick which sub-layer to use and probably the attribute table would be limited to the content of tiles currently displayed...

6. Do I remember correctly that for 3D stuff we need vector tiles in TopoJSON format? Is it worth considering supporting TopoJSON (maybe in the future) to support that use case?

Not really, we do not use TopoJSON anywhere for 3D stuff. In my opinion it is most useful in a limited set of cases (e.g. when you have polygons with shared borders) and it's text-based format so currently I do not see a lot of value in adding support for it...

[timlinux]

Not really, we do not use TopoJSON anywhere for 3D stuff. In my opinion it is most useful in a limited set of cases (e.g. when you have polygons with shared borders) and it's text-based format so currently I do not see a lot of value in adding support for it...

Sorry I didnt express myself well. I meant "isnt topojson used for 3D stuff by web clients like ol-cesium"...So I was trying to say that it may be a commonly used format for people doing 3D viz on teh web and potentially interesting to be able to use with QGIS 3D stuff in the future.

Thanks for your other responses!

[strk]

1. For supported data sources - what about being able to retrieve tiles straight from PostGIS? Or do we already get that with the list of data source schemes you propose?

I have not thought about such feature. I have planned for local and remote (HTTP/S) access through XYZ template + local access through MBTiles. If you have a PostGIS database available, it's probably easier to directly load the vector layers. But I may be missing your point :)

I don't think there's support in QGIS to directly load topologically defined layers, other than via TopoGeometry::Geometry casts (that is, transferring duplicated common edges) or via loading primitives (loosing the "areal" features bound by those edges).

The TopoJSON format is an available output for TopoGeometry objects, which would allow transferring areal features w/out duplicated edges.

If the ability to reconstruct areas on the client side will be implemented in QGIS it'd be great to allow different formats to populate such structures (so we can do TopoJSON, direct PostGIS, direct GRASS etc..)

[wonder-sk]

@timlinux

Sorry I didnt express myself well. I meant "isnt topojson used for 3D stuff by web clients like ol-cesium"...So I was trying to say that it may be a commonly used format for people doing 3D viz on teh web and potentially interesting to be able to use with QGIS 3D stuff in the future.

I am not aware of topojson being used for 3D stuff. In 3D worlds there is a nice standard called 3D Tiles coming from Cesium community (see https://cesium.com/blog/2015/08/10/introducing-3d-tiles/ ). It can be thought as a generalization of vector tiles into 3D space, it is optimized towards use in 3D engines and allowing a heterogeneous mixture of input data (e.g. models of 3D buildings, point clouds and others). At some point I would like to have support for import/export of it in QGIS 3D, but we already digress :-)

[wonder-sk]

@mnboos Great to hear you are happy with the proposal!

However, I'm asking myself what would be a good solution to handle the styling. The current style convert which I developed is far from complete but I assume, you don't want to build some Mapbox GL Style converter, which would probably have to be updated quite regularly, into QGIS. Maybe make QGIS fully capable of handling Vector Tiles and leave the styling to other tools?

Yeah as also @tomchadwin says, maybe it will be best to leave the conversions between different styling formats to a dedicated library such as https://github.com/GeoCat/bridge-style and only integrate it through a 3rd party plugin.

The only reason I wanted to support Mapbox GL style was to help people reuse existing styles so they do not have to start from scratch. But I am happy to drop that if we can have 3rd party tools for that. I have not looked into bridge-style in more detail yet to understand how well it can handle mapbox style to qgis style conversion.

[sfkeller]

As the advisor of the initial implementation by @mnboos as QGIS plugin:

1. I'm in favour of MB Vector Tile specs. over TileJSON because it's dissemination and support (indirectly e.g. by OpenMapTiles) .

2. I'd vote for MB GL styling language over others (except QGIS's own of course) again just because of it's dissemination and support (like the independent Maputnik style editor).

Just my 2 cents.

[jerstlouis]

Check out the Open Portrayal Framework from OGC Testbed 15: Styles API, styles in Geopackages; multiple styles encoding options around a conceptual model.

http://docs.opengeospatial.org/per/19-018.html Video demonstrations: https://www.youtube.com/playlist?list=PLQsQNjNIDU85HBDZWc8aE7EvQKE5nIedK

[X3msnake]

Why is the minimum plege 100.eur? A bit high if you are a individual wanting to contribute

[saberraz]

@X3msnake see: https://twitter.com/lutraconsulting/status/1231978711806873600

[Jnb2387]

This will be Awesome! Can't wait. For importing Mapbox styles is it possible to read the whole style including expressions? This seems to be limited in maputnik.

[saberraz]

@Jnb2387 see https://github.com/qgis/QGIS-Enhancement-Proposals/issues/162#issuecomment-577586096

[Komzpa]

There is MVT driver that lets you open MVT datasets already: https://gdal.org/drivers/vector/mvt.html that wasn't mentioned in prior art section. May it happen that task can be solved by finetuning integration with ogr driver instead of a completely new thing?

[wonder-sk]

@Komzpa the bulk of the work is not in the actual reading of encoded vector tile data. There is a lot of work involved in creation of a new map layer type, adding GUIs to define connections, rendering etc. In theory OGR driver could be used for reading of tile data, but it is probably not going to be most efficient. There are various other cases in QGIS where gdal/ogr is not directly used - to either improve the UX (e.g. WMTS provider that can display raster tiles as soon as they arrive) or to support features that are not supported through ogr interface (e.g. postgis provider or other DB providers).

[sfkeller]

Thanks @wonder-sk for explaining the challenges.

In order to understand the challenges implementing a MVT reader, I can only encourage visting the Python code the of https://github.com/geometalab/Vector-Tiles-Reader-QGIS-Plugin where we faced exactly some design decisions mentioned above, like

• adding GUI to define connections and to set parameters
• being (potentially) more efficient than a OGR driver
• improving UX to render tiles from the beginning
• giving feedback about the loading progress
• having access to intermediate data structures (like transformed style and cached vector tiles)

[jaakla]

There is one more open source cross-platform vector tile reader and renderer, written in C++/OpenGL : https://github.com/CartoDB/mobile-sdk . Despite of name it is ported not only to Android and iOS, but also Windows 10. Renderer styling works with 2 languages: CartoCSS or Mapnik XML.