Support GeoServer Vector tile

[sunshine0576]

I want add Geoserver Vector tile layer to QGIS, How should I add a connection?

[sfkeller]

We did not yet test the plugin with a GeoServer connection. Can you provide us with remotely accessible source. A Vector Tile connection requires either a mbtiles file or a TileJSON file typically in the form https://yourdomain.xyz/data/v3.json according to TileJSON 2 spec: is here https://github.com/mapbox/tilejson-spec/tree/master/2.2.0 (which actually still has to be extended in order to reflect the add-ons Mapbox made in order to support Mapbox Vector Tiles, .mvt).

[sunshine0576]

The format of the request for Geoserver. http://localhost:8080/geoserver/gwc/service/wmts?REQUEST=GetTile&SERVICE=WMTS&VERSION=1.0.0&LAYER=sf:roads&STYLE=&TILEMATRIX=EPSG:900913:{z}&TILEMATRIXSET=EPSG:900913&FORMAT=application/x-protobuf;type=mapbox-vector&TILECOL={x}&TILEROW={y}

[mnboos]

It could work if a TileJSON file is created and the tile url set to the link above. Additionally the required TileJSON values would have to be present (i.e. "vector_layers", "minzoom" and "maxzoom").

[sunshine0576]

Expect to be able to support Geoserver new features.

[mnboos]

@sfkeller Should we implement GeoServer support?

[sfkeller]

Yes, but not "directly" accessing zxy but as data source via the TileJSON metadata file.

[sunshine0576]

@sfkeller Geoserver support TileJSON metadata file.

[sfkeller]

We can't test GeoServer Support without an example of a remote server.

[tomchadwin]

@mtravis Are any of our Geoservers on versions recent enough to support vector tiles?

[mtravis]

I don't believe so. Let me check.

On 17 Nov 2017 11:13 am, "Tom Chadwin" notifications@github.com wrote:

@mtravis https://github.com/mtravis Are any of our Geoservers on versions recent enough to support vector tiles?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/geometalab/Vector-Tiles-Reader-QGIS-Plugin/issues/112#issuecomment-345216235, or mute the thread https://github.com/notifications/unsubscribe-auth/ABt0RgMHrjp7a_cTwdNxARi5JbQx9t6-ks5s3WpLgaJpZM4QV6pE .

[tomchadwin]

@sunshine0576 Do you know which Geoserver version introduced VT support?

[mtravis]

Its seems that the vt extension is only available with 2.11 onwards.

https://sourceforge.net/projects/geoserver/files/GeoServer/2.11.0/extensions/

@tomchadwin the shared geoserver is running 2.9

On Fri, Nov 17, 2017 at 11:18 AM, Tom Chadwin notifications@github.com wrote:

@sunshine0576 https://github.com/sunshine0576 Do you know which Geoserver version introduced VT support?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/geometalab/Vector-Tiles-Reader-QGIS-Plugin/issues/112#issuecomment-345217393, or mute the thread https://github.com/notifications/unsubscribe-auth/ABt0RpGmPGH2Dy3rXSQveY7aN9gfP1-Aks5s3WuWgaJpZM4QV6pE .

[tomchadwin]

And we can't use yours? @tomchadwin gives @mtravis a winningly persuasive smile

[mtravis]

Shamefully we are even further behind (need to upgrade ubunutu first). I've only just seen the whole thread so see that you need one for testing.

[tomchadwin]

And for fun...

[mtravis]

of course...

[sfkeller]

Dear @flippmoke : Pls. allow me to bump on this again since here at Vector Tiles Reader for QGIS project (to be released end of 2017) we are heavily relying on remote TileJSON and mbtiles metadata, while TileJSON spec. still does not even mention Vector Tiles.

I'd really like to know now which metadata is going to be there - mandatory or optional.

Saying "we" I dare to include OpenMapTiles by @klokan, MapZen by @nvkelso, t-rex by @pka, as well as TileMaker by @systemed and others like GeoServer by @aaime (?) and PostGIS i.e. by @pnorman ...

[pnorman]

fwiw, I don't have any plans on including tilejson data with the projects I'm currently working on.

[sfkeller]

@pnorman Thanks for the info (BTW you probably have been the first realizing this issue in 2014: https://github.com/mapbox/tilejson-spec/issues/14 !)

[systemed]

Yes, would be good to get this finally sorted out - .mvt is in real danger of turning into the new Markdown (everyone uses it, no two clients parse/write it the same way).

[aaime]

Hi, I had no part in adding MVT support for GeoServer, let me add into the loop @dblasby, @jodygarnett and @tbarsballe (sorry for the long lineup, I'm not sure who's maintaining vector tiles these days)

[DBlasby]

The easiest way to connect to Geoserver vector tiles is via the standard OGC WMTS - you can use the GetCapabilies to get any meta data you need.

You can connect to it as a XYZ server, but (I believe) you have to use -Y ordinates.

There is a trivial example in the geoserver documentation - for OL3 (more on the OL3 site). You will need a recent geoserver, with the vector tiles plugin (extention) installed.

I don't think there is a tilejson meta data file - but not sure (that would be part of GeowebCache).

[flippmoke]

I just wanted to let everyone know that we are working internally at Mapbox to update both the mbtiles and tilejson specs to be correct. Thanks you everyone for your notes about this, we definitely want to make sure it is all correct. If you have concerns for the individuals specs please add issues to the tilejson-spec (or a pull request is even better!). For mbtiles-spec the pull request that is listed above is being reviewed internally and we hope to release soon.

[sfkeller]

@DBlasby Thanks for you answer.

Our VT QGIS client consumes three sources: Online (=TileJSON), File (mbtiles) and a local directory of unpacked vector tiles (still zipped PDFs but being in subdirectories according to the XYZ index).

The TileJSON contains metadata similar to WMTS GetCapabilies but not really. And I really prefer to build on a spec. which is aware of vector tiles - which is why I pushed tilejson-spec.

AFAIK GeowebCache has no support yet of TileJSON but probably can generate a local directory as described above. => Do you know who is the maintainer of GeoWebCache?

[DBlasby]

Hi - @smithkm (kevin smith) is the GWC lead. You can contact him on the Geoserver-devel mailing list. http://geoserver.org/comm/

[sfkeller]

That's the the current metadata read by this MBVT Reader QGIS plugin:

• tiles: "https://domain.org/data/v3/{z}/{x}/{y}.pbf"
• minzoom: 0
• maxzoom: 14
• bounds: [-180,-85.0511,180,85.0511] -- Synonym extent.
• attribution: "" -- optional, maybe empty string
• crs: 3857 -- optional EPSG code; 3857 by default
• vector_layers: -- List of layers, each described with id, description, minzoom, maxzoom and fields. For example (JSON): [{id: "road", description: "", minzoom: 7, maxzoom: 14, fields: {type:String, name:String}}, {id: "water", description: "", minzoom: 0, maxzoom: 14, fields: {name:String}}, ...]

[smithkm]

I'm not sure what kind of vector tile awareness you need in your protocol. WMTS doesn't require interaction with the content of the tiles so all it needs to support a format is the MIME type. GeoWebCache is able to handle all of its supported vector tile formats through WMTS. New tile protocols are fairly easy to plug in to GWC so if you want to implement this TileJSON protocol you are welcome to do so.

At the moment we recommend using the KVP profile of WMTS as the best tile protocol for most situations regardless of tile format. If you want to add additional metadata to WMTS capabilities, there is an extension point for that.

I'm not even sure what is meant by TileJSON being vector tile aware, the spec website just seems to be a lightly commented example and never even mentions formats other than that UTFGrid and GeoJSON for the interactivity and data sections. I'm not sure where vector_layers is coming from. I assume this is just something MapBox software uses but is not included in their published "spec"?

[sfkeller]

Thanks @smithkm for your comments. You wrote:

I'm not sure what kind of vector tile awareness you need in your protocol.

It's currently really these attributes I listed before your comment:

• tiles refers to the vector tile data (compressed pbf).
• minzoom and maxzoom announce in which zoom levels there are tiles.
• bounds and attribution (while attribution is IMHO the only optional attribute).
• vector_layers contains the layers and their field names. This schema is used to prepare the layers e.g. of QGIS before vector tiles data is loaded.

I'm not even sure what is meant by TileJSON being vector tile aware ... I assume this is just something MapBox software uses but is not included in their published "spec"?

Exactly - and I hope Mapbox reads this ;-) That's what I'm saying since a long time! See https://github.com/geometalab/Vector-Tiles-Reader-QGIS-Plugin/issues/112#issuecomment-345535770 . Actually @flippmoke answered recently (see above) that they ...

are working internally at Mapbox to update both the mbtiles and tilejson specs to be correct.

[smithkm]

The WMTS capabilities document currently provides all of those except the attribute names. That could probably be added via the WMTS extension mechanism although it would require a mechanism for setting it (this would also be a problem for implementing TileJSON). This would be a somewhat similar issue to the handling of legend graphics. The underlying data for a layer might vary with zoom level or other things. I'm not sure but the details of what's included in a vector tile might also vary with the particular tile format. MapBox PBF vs GeoJSON, vs TopoJSON vs whatever. Also you can set the STYLE(S) parameter which at least in the case of GeoServer's vector tile implementation can alter the set of attributes returned. So a GWC TileLayer wouldn't have just one fixed set of attributes for its vector tiles.

[sfkeller]

Yes, the nature of vector tiles (VT) is that the data and most probably the schema too changes at every zoom level. And the data and schema coming from different VT providers varies.

As said, vector_layers in de-facto TileJSON contains those field names for each zoom level. See e.g. https://gist.github.com/jatorre/6212354d5e023076797db7ea18540c33 . Without knowing these attributes a client has a hard time to prepare it's layers to load the data.

Regarding TileJSON the format is set/announced - most probably Mapbox pbf for Mapbox Vector Tiles (MVT) .

You wrote

Also you can set the STYLE(S) parameter which at least in the case of GeoServer's vector tile implementation can alter the set of attributes returned.

I can't follow here: Vector Tiles are not styled; styling is done by the client.

[smithkm]

GeoServer uses a style to simplify the data before packaging it up as a vector tile. In particular it reduces the attribute set based on the style. GeoWebCache layers are configured with parameter filters that allow a layer to vary based on style, time, elevation etc. So a GWC cached GeoServer layer might end up giving different attribute sets depending on that parameter.

[DBlasby]

Geoserver goes through a "rendering" process to create the vector tiles. For example, zoomed out, you might only want to put in HWY roads, or lakes bigger than a certain area, or turn layers on/off. You control that with styles in Geoserver.

You can see more about VT control in GS in the talks linked here;

http://docs.geoserver.org/latest/en/user/extensions/vectortiles/index.html

[smithkm]

Ah right I think I was conflating attributes being removed with features being removed, Thanks Dave.

[sfkeller]

This reminds me to the confusion Mapnik introduced when they used the name "schema style" to produce VT. At least they mention (hopefully) "schema". But even cartographers call this view/filter step "database generalization". I hope VT documentation makes this clear to the VT data generating and consuming users.

So, coming back to my initial intention to reach de-facto interoperability with Mapbox Vector Tiles (MVT), I see at least following issues:

• What are the mandatory parts of VT metadata (crucial: vector_layers)? That's to be specified for MVT in TileJSON and MBTiles. I hope @flippmoke can help us here by making a proposal of the respective Mapbox specs.
  • Can GWC/Geoserver be extended according to this TileJSON spec. so that it optionally generates a minimal TileJSON file (probably similar to WMTSCapabilities.xml and like e.g. t-rex from @pka does https://github.com/t-rex-tileserver/t-rex/blob/master/CHANGELOG.md ).

[flippmoke]

What are the mandatory parts of VT metadata (crucial: vector_layers)? That's to be specified for MVT in TileJSON and MBTiles. I hope @flippmoke can help us here by making a proposal of the respective Mapbox specs.

@sfkeller The mandatory parts of the vector tiles are set out explicitly in the VT spec. It is not an easy read, but if you want to know definitively what is required that is the most important location.

As for metadata that might be used in TileJSON or MBTiles, what is considered critical metadata might differ based on how VTs are used. Honestly, @ericfischer might be better to answer this question.

[e-n-f]

My best attempt to describe what is necessary to create a working mbtiles file of vector tiles is in https://github.com/mapbox/mbtiles-spec/pull/47 (spec.md).

I am not sure if vector_layers is actually necessary for rendering, but it is necessary for using Mapbox Studio to do the styling.

[sfkeller]

vector_tiles with fields (announced as part of TileJSON) seems to me necessary to almost all VT consumers which are not purely rendering oriented. I'm thinking of databases (inlcuding MBTiles) and specifically GIS with layers etc. like QGIS.

@ericfischer: Your attempt to describe what is necessary for mbtiles spec. for VT looks good to me.

If this could be aligned with the TileJSON spec., and if both updated specs. would be released after some consultation, I'd be happy.

[tomchadwin]

I find this thread really encouraging - thanks to @sfkeller for bringing a lot of interested parties together, and thanks to you all for accepting that improvement can be made. It would certainly help a MBVT beginner like me to navigate your different implementations if standards were a little clearer.

I'm sure this is not the place to mention it, but is a mandatory id field unique per feature a possibility in future iterations of the spec? Without it, labelling is extremely difficult.

[e-n-f]

I think in the Vector Tile spec id will remain optional because it adds a lot of overhead for tilesets that don't need it. The future direction that we have been thinking about is an additional field to identify features that have been split across tiles so they can be rejoined for rendering or matching.

[sfkeller]

@tomchadwin : Thanks for testing. My strong worries are not about the Mapbox vector tile (MVT) spec. but about the metadata of Vector Tiles (as encoded in TileJSON and MBTiles)! As you realize from the discussion above, GeoServer and ArcGIS Pro devs seem not to be aware of the need of such metadata besides ZXY service/data!

You refer to the MVT spec. and this seems to me somehow stable: The spec. https://github.com/mapbox/vector-tile-spec/tree/master/2.1#42-features says "A feature MAY contain an id field. If a feature has an id field, the value of the id SHOULD be unique among the features of the parent layer." So it's up to the VT producer to offer an id field in the data. Mapzen MVTs deliver it while MB and OpenMapTiles have chosen not in their main VT data products.

AFAIK there's still an id field in the protobuf format which could be exposed from a reader like the QGIS MVT reader, but have to verify this.

[tomchadwin]

@ericfischer, @sfkeller Thanks, both. Apologies for confusing metadata and data. I'm with you now.

[jaakla]

It seems that the vector_layers tag in mbtiles/tilejson is mapbox-specific not documented extension. You can consider it proprietary, and hope that their lawyers do not chime in if you use it. Requiring such metadata to be omnipresent by other vendors vector tile implementations sounds like slippery slope to me.

This metadata is very useful indeed for many use cases, without it you may need to scan through all tiles to reverse-engineer data schema. Note: MVT was designed first of all for map rendering, so with using it beyond it brings you to many tech challenges. So any motion to standardize it would be very welcome. But as long as the whole MVT stack is fully under single company control, then there is not much what outsiders can do about it.

p.s. Regarding object IDs in actual data there is some activity in OMT side: https://github.com/openmaptiles/openmaptiles/issues/303

[sfkeller]

It seems that the vector_layers tag in mbtiles/tilejson is mapbox-specific not documented extension. You can consider it proprietary, and hope that their lawyers do not chime in if you use it. Requiring such metadata to be omnipresent by other vendors vector tile implementations sounds like slippery slope to me.

Actually @flippmoke (see above) gave us hope that no lawyers but techies chime in order to update the spec.

[flippmoke]

@jaakla

We are 100% committed to open source and open standards at Mapbox.

https://www.mapbox.com/about/open/

We work hard to keep open standards and open source. This is backed up by our use of Creative Commons license on our specifications. In fact I just added a license file to make it even more clear on the Vector Tile Specification.

Not only do provide these licenses in an open way, we go out of our way to make sure that we spend time to listen to the community as a whole prior to making changes to the specification. While these specs are often important to Mapbox's strategic goals, we also want to spend the time to provide any changes that are needed externally to Mapbox.

As the maintainer at Mapbox of the Vector Tile Spec, I spend quite a bit of time reading and looking at requests for the spec and implementations of the specification. Just like any other body that controls a specification we spend time to ensure that proper time is given for input from others prior to publishing and collect important ideas and comments that determine how the standard should move forward.

But as long as the whole MVT stack is fully under single company control, then there is not much what outsiders can do about it.

This is simply not true -- @sfkeller is a great example of this through this ticket and other efforts. We at Mapbox had not updated some of our standards up to this point and he is calling us out for it. This is greatly appreciated. I have already had several internal meetings about supporting the community needs here and updating items as necessary. The fault here is our own because we did not properly assign a maintainer of some specs as we had some individuals leave Mapbox.

Now that we are attempting to consider how to update these documents we are reviewing community input and issues that have been linked from the specification. We encourage pull requests on all of our standards and we want to be notified when we are not supporting the community properly.

[jaakla]

Thank you for the explanation. By "single company control" I mean different from how e.g. OGC and many other standard setting groups are designed - you have open independent group of people from various backgrounds, who decide about some standard, there is no one big entity who would have final decision for any detail. "Open standard from a company" by definition is different - in addition to the community you also need to consider company interests, and you have every right to keep some things you've invested in or what helps to increase revenues, closed. For example vector data schema or web map style editing tools. It is not special for mapbox, this is how any sensible commercial company does and there is nothing bad about that really. Being too zen with too much openness and forgetting about business realities is dangerous also.

Mapbox has been above average open to share useful things, and to accept community contributions, big kudos to them for this. However, as there has been at least one case where someone has naively tried to hack into tech details which were not explicitly open and they got their fingers burnt. So instead of reverse-engineering I would expect that these de-facto things are specified clearly in the open specification first, to be sure that it is all legal. Also to be sure that it has minimal technical stability. Is see the vector-revision draft is on the way, that's correct?

Going with fast-developing tech like vector tiles were few years back to an organization like OGC can easily kill technical edge, so in short term one strong commercial leader is better. But once they are stable (and vector tiles and mbtiles for example are already 'establishment', have not changed much anymore in last few years), then for long-term safety I hope Mapbox considers handing them over to a neutral standard body, or create new if there is no suitable one.

[gannebamm]

Are there any news how to load by Geoserver served vector tiles by the plugin?

[sfkeller]

No, sorry that I can't report from any activities related to this issue. Perhaps others listening here can correct me.

As this is an underfunded, volunteered project I beg you patience. And to encourage contributors I put a new label "contribute or fund me" on it.

[mnboos]

Apologies for all the delays. We just moved to another town and will soon go to our honeymoon for 3 weeks. I hope I can do something after that.

[GjueAtGit]

I made some attempts to integrate Vector Tiles from GeoServer into QGIS via the Vector Tiles Reader plugin. I was not successful with TMS. However, the tiles can be read via WMTS. A code example and further information (currently only in German) I have described here.

[sfkeller]

Hi/Hallo Gerd.

Thank you very much for your information about your successful approach connecting QGIS MVTs with GeoServer via WMTS (GeoServer 2.14. + VectorTilesPlugin, QGIS 2.18.24 + Vector Tiles Reader QGIS-Plugin 2.0.3).

If you want to help, it would be great when you could deploy a GeoServer on the internet (and send us the URL to the TileJSON file), so that we can test the VT Reader.

It's not yet clear when we have time since, as said above, @mnboos is away for some weeks, and the VT Reader is a hobby project of him and me in our leisure time.