search

qgis / QGIS-Enhancement-Proposals

QEP's (QGIS Enhancement Proposals) are used in the process of creating and discussing new enhancements for QGIS
117 stars 37 forks source link

Developing a QGIS style for OSM-based XYZ tiles #94

Open shenriod opened 7 years ago

shenriod commented 7 years ago

QGIS Enhancement: Developing a QGIS style for OSM-based XYZ tiles

Date 2017/06/06

Author Stéphane Henriod (@shenriod)

Contact s@henriod.info

maintainer ??

Version QGIS 3.0

Summary

A new feature is being developed to make it easier for users to display web basemaps in QGIS: https://issues.qgis.org/issues/16493 Under the current implementation, tiles would be pulled from the main OSM tile server. A few potential issues with this approach have been identified:

We thus would like to explore the possibility to build a custom QGIS solution:

Brainstorming

For now I suggest to brainstorm on this Gdoc: https://docs.google.com/document/d/1f1OJtshHCPcLVEVqZjTEgI47zn61Plv5I-Yc68d2b64/edit?usp=sharing

The results of this common brainstorming should determine if it makes sense or not to move ahead with a more concrete proposal

Proposed Solution

(required)

Example(s)

(optional)

Affected Files

(required if applicable)

Performance Implications

(required if known at design time)

Further Considerations/Improvements

(optional)

Backwards Compatibility

(required)

Issue Tracking ID(s)

(optional)

Votes

(required)

elpaso commented 7 years ago

Sorry, but I think that this is not something that the QGIS project should handle, I believe that we should focus all our resources on the development of QGIS and try to keep the related (web) services at the bare minimum (like the official website, the plugin repo and a few others). So, it's a strong -1 from me.

klokan commented 7 years ago

What about to integrate into QGIS instead a vector global basemap - powered by OpenStreetMap vector tiles from http://www.openmaptiles.org/. We offer a free hosting - and the QGIS team can host the data easily by themselfs - independetly of us. It is a single MBTiles file of the size of about 50 Gbytes - to be exposed from a server. Scalability is not an issue either.

This way the style can be made in QGIS native styling language - and applied on the client side. It would make a unique service in the application - not just another "load these raster tiles" which is offered by several plugins already.

It would require improvement of the parsing of PBF / MVT tiles - and loading these as zoomable vector service. The initial implementation is done a Python plugin in https://plugins.qgis.org/plugins/vector_tiles_reader/ - but it would require more work and improvement of performance (probably rewrite into C++).

https://github.com/geometalab/Vector-Tiles-Reader-QGIS-Plugin @sfkeller

sfkeller commented 7 years ago

I support the proposal of @klokan .

This astonishing experimental Python plugin I'm supervising will get a last update now before final release. This includes most probably "loading/flushing tiles" much like raster tiles in "Base map" mode (there are two other modes: Analysis and Inspection).

As @klokan wrote, the "base map" mode of this Vector Tiles Reader should actually be migrated to a C++ plugin in order to become really fast. Unfortunately I don't have the resources in short terms to accomplish that.

rduivenvoorde commented 7 years ago

@klokan @sfkeller I'm in general in favour of starting to use vector tiles (and sharing QGIS styles around those), next to image tiles.

But I agree with @elpaso that we (as QGIS community) should not start our own tile server (with all dangers of bandwidth/speed/uptime troubles). Nor image tiles (as that was the initial question/idea), nor vectortiles. Or at least not at this moment.

But if the Klokan is providing a free tile/download(?) service I think we should bring this idea to the PSC/Community. (It is not clear to if you mean hosting the 50Gb MBTiles file (as a download), or a server which serves out the vector tiles)

I installed the vector_tiles_reader plugin, but I could not find the right url to use for openmaptiles.org. I don't think an average user should request a key/token is it? It should just work out of the box.

Tried the plugin:

Some general questions (but maybe I should address them at the plugin github repo?):

sfkeller commented 7 years ago

@rduivenvoorde : You wrote

I don't think an average user should request a key/token is it? It should just work out 
of the box.

The plugin works out of the box with the predefined connection to OpenMapTiles.com which is a service offered by Klokan. In addition the plugin allows to configure other/own connections to vector tiles services.

But if the Klokan is providing a free tile/download(?) service I think we should bring this 
idea to the PSC/Community. (It is not clear to if you mean hosting the 50Gb MBTiles file 
(as a download), or a server which serves out the vector tiles)

@klokan : Can you clarify?

Some general questions (but maybe I should address them at the plugin github repo?):

Yes please; enter them also in the plugin github repo as mentioned below. Just answering here quickly nevertheless:

are the tiles only available at streetlevel?

No; level 14 represents the "street level" and get's overzoomed. Then, in vector tiles there's a new geometry generated for each levels 14,13,.... up to 1.

should there not be a warning if you try to load the data in a project which is not 
epsg:3857?

There should be a warning and turning on OTF would help too (like QuickMapServices?). Could you pls. enter this as an issue in https://github.com/geometalab/Vector-Tiles-Reader-QGIS-Plugin/issues/ with a description what you exactly did when you got an error?

I think we need some good blogging about vector tiles too first

Indeed. See my public scratch pad as a starting point: https://giswiki.hsr.ch/Vector_Tiles

klokan commented 7 years ago

We are willing to provide a free vector tile hosting from our global infrastructure for the QGIS project - if there is a proper visible attribution with links to both OpenStreetMap and OpenMapTiles projects.

End users would not need any access key - it should run out of the box (we could probably limit access only to the QGIS user agent submitted in the HTTP headers).

We operate a high-availability tile hosting infrastructure on two continents (Europe, North America) with global CDN and over 100 POPs all over the planet including Oceania, Japan, New Zealand, Africa, etc. Our vector tiles are updated weekly with latest changes from the OpenStreetMap data.

The tile hosting service I talk about delivers individual vector tiles - MVT PBFs - via XYZ service. Try it by getting a free key at https://openmaptiles.com/hosting/. For QGIS users we would make a special setup.

Next to that, we have a free bulk download service https://openmaptiles.org/downloads/ of the pre-generated MBTiles files, supported by the Swiss academia (HSR and SWITCHengines academic cloud). QGIS users who are offline (or on a bad Intenet connection) should be able to use a downloaded MBTiles and get the same functionality as with the online tilehosting service.

There is no vendor lock-in on the OpenMapTiles format or map styles - it is possible to migrate to another hosting provider or self-host. OpenMapTiles is open-source project driven by the community - and used already by Siemens, IBM, Bosch, Amazon, Carto, etc. I believe it is a great candidate for the QGIS vector base map.

We would offer the hosting for free, but in case QGIS PSC/Community wants to host the vector tiles on their own infrastructure now or in future - it is possible too - of course.

Hosting practically means exposing the pre-generated PBFs stored in the MBTiles on a ZXY endpoint. Tiles can be even unpacked into a folder or on Amazon S3 - and hosted completely without a server - but this is not practical for updates and for very large tilesets (such as the whole planet). Basic hosting can be done with a 50-line FastCGI in C/Python or with our TileServer PHP or TileServer GL Light (Node.js).

Naturally, a production tile hosting service requires much more - including regular updates, CDN with enough bandwidth, proper caching setup, high-availability configuration and load balancers, geo-distribution, logging, monitoring, elasticity on increased load, backups, rate limiting, access keys, etc.

Our vector tile hosting can be available soon - or even right now - but QGIS needs to be able to load the vector tiles and display these efficiently...

It means parsing the MVT PBFs and converting them into native QGIS features on panning and zooming - as if the features would come from PostGIS or another vector source.

This is where the core QGIS developers can put the hands on... Maybe it could be done fast if something like https://github.com/TimSC/mbtiles-cpp is used?

A single QGIS native style for the beginning would be enough - probably a light background style such as ISC/CC-BY http://maps.stamen.com/toner-lite/#12/37.8068/-122.3149 or Carto Positron available under BSD/CC-BY https://openmaptiles.github.io/positron-gl-style/#0.87/11/15 or perhaps a port QML style sheets available at https://github.com/charleyglynn/OSM-Shapefile-QGIS-to the layers present in our OpenMapTiles vector tile schema?

IMHO It would not be hard to write a converter for the existing JSON GL styles to QML - which would bring all the open-source map styles made for OpenMapTiles (Positron, Bright, Dark Matter, KT Basic, Toner, etc) into QGIS natively...

The OpenMapTiles open-source project is going to be presented at FOSS4G in Boston - so there may be a chance to meet personally with QGIS PSC and discuss the next related steps there - or anytime sooner?