Adding vector tile service from ArcGIS / Esri not working

[StefanJaq]

Trying to add a vector tile service from ArcGIS to QGIS results in the following Error.

We are using QGIS Las Palmas with Extenstion Vector Tiles Reader QGIS Plugin.

Using the style URL https://tiles.arcgis.com/tiles/Ok7pjj1jT9rEneC7/arcgis/rest/services/Vector_tiles/VectorTileServer/resources/styles/root.json

Our setting are as in following screenshot (default): This operation results in the following error:

We adopted the style-json: http://austria.maps.arcgis.com/sharing/rest/content/items/e01226ca419642f08a929974adcff3a1/resources/styles/root.json After doing this and the same default setting in QGIS the following error occurs:

1. It looks like the plug-in is requiring a tilejson-spec extension named vector_layers, despite it isn't part of the current documentation (I didn't find it). Why is this needed? https://github.com/mapbox/tilejson-spec/issues/14

2. Why is it neccessary to enter two URLS? Why especially the TileJSON-URL? Usually the Tile-Url is linked in the styles - JSON.

Any ideas?

[mnboos]

1. We're requiring the _vectorlayers field in order to build a tree hierarchy in QGIS (and to show a preview of the available layers so that the user can make a selection of those to only load the selected layers).
2. You entered the StyleJSON URL in the field for the TileJSON, that's not working. Due to the current implementation of the reader, it's only capable of reading from one source (i.e. one TileJSON address) at a time. As there may be several TileJSON URLs per StyleJSON, that's not supported at the moment.

[mnboos]

The one with the "int() argument must be a string or a number, not 'NoneType'" was clearly an error and has been fixed. The fix will be included in the next release, which will be in the next few days.

[StefanJaq]

:: _vectorlayers field - Could you make this an optional parameter?

:: TileJSON - There exists NO TileJSON Url for the Esri vector tiles, as far as I know. The mapbox spec https://www.mapbox.com/mapbox-gl-js/style-spec/#sources allows 3 different implementations. Esri choose supplying TileJSON properties such as "tiles", "minzoom", and"maxzoom" directly in the source. Could the plugin support this one?

:: "int() argument..." - We will try this with the new release.

Thanks for your answers.

[mnboos]

:: vector_layers field - Could you make this an optional parameter?

Technically, this would be possible.

:: TileJSON - There exists NO TileJSON Url for the Esri vector tiles, as far as I know. The mapbox spec https://www.mapbox.com/mapbox-gl-js/style-spec/#sources allows 3 different implementations. Esri choose supplying TileJSON properties such as "tiles", "minzoom", and"maxzoom" directly in the source. Could the plugin support this one?

We use the StlyeJSON URL only for styling, nothing else. Due to this, I don't think there will be a change soon in this matter in the implementation of the plugin.

@sfkeller Would you mind to comment on our StyleJSON usage, especially, that we use it only for styling, even though there are TileJSON URLs in the StyleJSON?

[StefanJaq]

The "int() Argument error..." is fixed, thanks.

Summarizing, Esri vector tiles can't be used with this plugin in QGIS as the plugin supports a different implementation than Esri. Both implementations are valid.

@mnboos Thanks for your help.

[mnboos]

Happy to help.

[sfkeller]

@mboos wrote

@sfkeller Would you mind to comment on our StyleJSON usage, especially, that we use it only for styling, even though there are TileJSON URLs in the StyleJSON?

For a GIS usability point-of-view it's obvious: Data/layer/source are loaded, styled, then save as a GIS project. So, a project acts as "grouping concept". A TileJSON represents a Mapbox VT (metadata) data/layer/source without style (like a vector GeoPackage).

A MB GL JS style (style.json), as it is handled in Mapbox tools (like Studio) in fact can contain several sources and maps them to a style descriptor. So here, the style acts as style config. and as grouping container, aka "project" file (=> "project" style.json with many data-source refs plus many style-descriptors).

Currently the plugin manages one VT source and allows to assign optionally one style (ignoring other sources there) and persists this in the QGIS project file (=> one data-source plus many style-descriptors).

Now given TileJSON/mbtiles is not even standardized yet, it makes sense to wait for a maturation of the specs.

[sfkeller]

@StefanJaq wrote

Summarizing, Esri vector tiles can't be used with this plugin in QGIS as the plugin supports a different implementation than Esri. Both implementations are valid.

I'd rather say that I still hope, that it's possible to generate a minimal TileJSON metadata file. This is needed by Maputnik as source and by Mapbox Studio too. Given this TileJSON it would be possible at least to load it as data source/layer in this plugin too.

@mboos: Any ideas to preprocess any VT source (may it come from ArcGIS Pro or tilemaker or so) and generate this TileJSON metadata file?

[mnboos]

@sfkeller I think the generation/creation of a TileJSON is not even the problem. The reader is implemented this way, that it reads from a single source (=TileJSON) at a time. A StyleJSON on the other hand, consists of multiple TileJSON sources and that's the problem.

I'm thinking about the following change: The StyleJSON input is removed and the only remaining input field then accepts both type (i.e. StyleJSON and TileJSON). The support of a StyleJSON as source, would be a nice enhancement of the plugin.

[Moult]

Sorry to bump this very old thread, but I am currently trying to get this resource showing in QGIS:

https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer

The conclusion I got from the various bug reports is that it is still currently not possible, even with a local metadata.json. Has there been any developments on this, or have I misunderstood the conclusion?

[mnboos]

@Moult No worries, I'm happy about feedback. The resource you linked here, won't work out of the box, due to several missing properties. What you can do is create a metadata.json like described here: https://github.com/geometalab/Vector-Tiles-Reader-QGIS-Plugin/wiki/Help#metadatajson

The file can be on your local disk. Instead of using a URL in the connections dialog, just enter the full filepath.

If you need further assistance, please let me know!

[Moult]

Thanks @mnboos - I've already tried to create a metadata.json file as shown below - I copy pasted the layers json and renamed it as vector_layers without a full understanding of what it is expecting.

Please see the attached file (called .txt because that's what github will allow me to upload, but of course in real life it is a json file).

With this file, it can connect and load layers, but I only see black and no outcome in QGIS. Do you mind investigating my metadata.json for me?

metadata.txt

[mnboos]

Sorry for the delay, I'm currently on vacation. As far as I can see on my mobile, you copied the style json into the vector layers property, what either won't work or at least won't do anything. I'll have an in depth look at this scenario, as soon as I'm back.

[Moult]

No worries - take your time, no rush :) The style json is there, but doesn't seem to do anything. It certainly succeeds to connect and lists the layers though.

[mnboos]

Ok, I'm currently looking at it. The good news is, I can see something:

The bad news: An error occurs, due to invalid filenames, which I create in the temp folders (see #240).

I first have to fix this bug and hopefulle, it then works.

[mnboos]

I release version v3.0.5 which should work with this TileJSON: https://github.com/geometalab/Vector-Tiles-Reader-QGIS-Plugin/files/3269379/arcgis_tilejson.json.txt

[Moult]

Thanks @mnboos - can you give step by step instructions on how to get output? Here's what I have tried:

1. Update to v3.0.5 of the plugin.
2. Add Vector Tiles Layer -> Directory tab -> Browse path to local directory which has a metadata.json with the content of that textfile you just linked to.
3. GL style JSON URL is blank.
4. Press refresh
5. Select all layers
6. Press add.

Result is that I have a whole bunch of layers, but can't see anything.

[mnboos]

Add Vector Tiles Layer -> Directory tab -> Browse path to local directory which has a metadata.json with the content of that textfile you just linked to.

Not quite;

Select the server tab and just insert the whole filepath to the metadata.json

Result is that I have a whole bunch of layers, but can't see anything.

That's because they're read from the metadata, not from the actual server.

So the whole process would be:

1. Update to v3.0.5 of the plugin.
2. Add Vector Tiles Layer -> Server tab -> New -> Insert full path to local metadata.json
3. GL style JSON URL can stay blank.
4. Press Save
5. Press Connect (layers are listed, they are read from the json file you just specified, NOT from the server)
6. Press Add

[Moult]

Sorry, @mnboos I still cannot get your result. I've added it as a server as you've specified, with that json file having the contents of what you uploaded. It detects the layers of "foobar", which I can add.

However here is my map with OSM tiles as another layer to show how I am hovering around Shenzhen. As you can see apart from the OSM tiles basemap no vector tiles are loaded.

I have also tried setting the project CRS to EPSG 4214.

In the log messages I see this, suggesting that the X Y Z is incorrect:

2019-06-12T15:11:34     WARNING    Vector Tiles Reader 3.0.5 (Python 3.6.0)
2019-06-12T15:11:43     WARNING    Native decoding supported!!! (Windows, 64bit)
2019-06-12T15:11:45     WARNING    Bounds of source: {'zoom': 8, 'x_min': 0, 'x_max': 255, 'y_min': 0, 'y_max': 255, 'width': 256, 'height': 256, 'scheme': 'xyz'}
2019-06-12T15:11:45     WARNING    Loading zoom level '8', bounds: {'zoom': 8, 'x_min': 127, 'x_max': 128, 'y_min': 127, 'y_max': 128, 'width': 2, 'height': 2, 'scheme': 'xyz'}
2019-06-12T15:11:45     WARNING    Tile limit enabled: True (32)
2019-06-12T15:11:45     WARNING    0 tiles in cache. Max. 4 will be loaded additionally.
2019-06-12T15:11:45     WARNING    Getting 4 tiles from source...
2019-06-12T15:11:46     WARNING    Error during network request: Error transferring https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/8/128/127.pbf - server replied: Not Found, https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/8/128/127.pbf
2019-06-12T15:11:46     WARNING    Error during network request: Error transferring https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/8/127/127.pbf - server replied: Not Found, https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/8/127/127.pbf
2019-06-12T15:11:46     WARNING    Error during network request: Error transferring https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/8/128/128.pbf - server replied: Not Found, https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/8/128/128.pbf
2019-06-12T15:11:46     WARNING    Error during network request: Error transferring https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/8/127/128.pbf - server replied: Not Found, https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/8/127/128.pbf
2019-06-12T15:11:46     WARNING    Creating 0 layers...
2019-06-12T15:11:46     WARNING    Refreshing layers...
2019-06-12T15:11:46     WARNING    Adding 0 new layers to layer group...
2019-06-12T15:11:46     WARNING    Layer creation complete
2019-06-12T15:11:46     WARNING    Import complete
2019-06-12T15:11:46     WARNING    Loading of zoom level 8 complete! No extent loaded.
2019-06-12T15:11:55     WARNING    Reloading due to scale change from '970603' (zoom 8) to '183434673' (zoom 2)
2019-06-12T15:11:55     WARNING    Loading aborted as there are no layers of the current connection already loaded.
2019-06-12T15:18:22     WARNING    Bounds of source: {'zoom': 12, 'x_min': 0, 'x_max': 4095, 'y_min': 0, 'y_max': 4095, 'width': 4096, 'height': 4096, 'scheme': 'xyz'}
2019-06-12T15:18:22     WARNING    Loading zoom level '12', bounds: {'zoom': 12, 'x_min': 3344, 'x_max': 3346, 'y_min': 1783, 'y_max': 1785, 'width': 3, 'height': 3, 'scheme': 'xyz'}
2019-06-12T15:18:22     WARNING    Tile limit enabled: True (32)
2019-06-12T15:18:22     WARNING    0 tiles in cache. Max. 9 will be loaded additionally.
2019-06-12T15:18:22     WARNING    Getting 9 tiles from source...
2019-06-12T15:18:23     WARNING    Error during network request: Error transferring https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/12/1785/3346.pbf - server replied: Not Found, https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/12/1785/3346.pbf
2019-06-12T15:18:23     WARNING    Error during network request: Error transferring https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/12/1784/3346.pbf - server replied: Not Found, https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/12/1784/3346.pbf
2019-06-12T15:18:23     WARNING    Error during network request: Error transferring https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/12/1783/3344.pbf - server replied: Not Found, https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/12/1783/3344.pbf
2019-06-12T15:18:23     WARNING    Error during network request: Error transferring https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/12/1784/3345.pbf - server replied: Not Found, https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/12/1784/3345.pbf
2019-06-12T15:18:23     WARNING    Error during network request: Error transferring https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/12/1785/3345.pbf - server replied: Not Found, https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/12/1785/3345.pbf
2019-06-12T15:18:23     WARNING    Error during network request: Error transferring https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/12/1783/3345.pbf - server replied: Not Found, https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/12/1783/3345.pbf
2019-06-12T15:18:23     WARNING    Error during network request: Error transferring https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/12/1783/3346.pbf - server replied: Not Found, https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/12/1783/3346.pbf
2019-06-12T15:18:23     WARNING    Error during network request: Error transferring https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/12/1785/3344.pbf - server replied: Not Found, https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/12/1785/3344.pbf
2019-06-12T15:18:23     WARNING    Error during network request: Error transferring https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/12/1784/3344.pbf - server replied: Not Found, https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/12/1784/3344.pbf
2019-06-12T15:18:23     WARNING    Creating 0 layers...
2019-06-12T15:18:23     WARNING    Refreshing layers...
2019-06-12T15:18:23     WARNING    Adding 0 new layers to layer group...
2019-06-12T15:18:23     WARNING    Layer creation complete
2019-06-12T15:18:23     WARNING    Import complete
2019-06-12T15:18:23     WARNING    Loading of zoom level 12 complete! No extent loaded.
2019-06-12T15:19:40     WARNING    Reloading due to scale change from '67785' (zoom 12) to '1084557' (zoom 8)
2019-06-12T15:19:40     WARNING    Loading aborted as there are no layers of the current connection already loaded.

What am I missing?

[mnboos]

Can you do the following:

1. Create a new project in QGIS
2. Connect to the previously created connection
3. Add

And then copy logs and a screenshot. At this moment, for me it looks like this:

See the little purple area in the middle? That's your actual data.

And the logs:

2019-06-12T07:27:24     WARNING    Native decoding not supported. (Windows, 64bit)
2019-06-12T07:27:26     WARNING    Bounds of source: {'zoom': 1, 'x_min': 0, 'x_max': 1, 'y_min': 0, 'y_max': 1, 'width': 2, 'height': 2, 'scheme': 'xyz'}
2019-06-12T07:27:26     WARNING    Loading zoom level '1', bounds: {'zoom': 1, 'x_min': 0, 'x_max': 1, 'y_min': 0, 'y_max': 1, 'width': 2, 'height': 2, 'scheme': 'xyz'}
2019-06-12T07:27:26     WARNING    Tile limit enabled: True (32)
2019-06-12T07:27:26     WARNING    1 tiles in cache. Max. 3 will be loaded additionally.
2019-06-12T07:27:26     WARNING    Processing features of 1 tiles...
2019-06-12T07:27:26     WARNING    Getting 3 tiles from source...
2019-06-12T07:27:26     WARNING    Error during network request: Error transferring https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/1/1/0.pbf - server replied: Not Found, https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/1/1/0.pbf
2019-06-12T07:27:26     WARNING    Error during network request: Error transferring https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/1/0/0.pbf - server replied: Not Found, https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/1/0/0.pbf
2019-06-12T07:27:26     WARNING    Error during network request: Error transferring https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/1/1/1.pbf - server replied: Not Found, https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/1/1/1.pbf
2019-06-12T07:27:26     WARNING    Creating 5 layers...
2019-06-12T07:27:26     WARNING    Refreshing layers...
2019-06-12T07:27:26     WARNING    Adding new layers...
2019-06-12T07:27:26     WARNING    Adding 5 layers to QGIS...
2019-06-12T07:27:26     WARNING    Adding 5 new layers to layer group...
2019-06-12T07:27:26     WARNING    Layer creation complete
2019-06-12T07:27:26     WARNING    Import complete
2019-06-12T07:27:26     WARNING    Loading of zoom level 1 complete! Loaded extent: {'zoom': 1, 'x_min': 1, 'x_max': 1, 'y_min': 0, 'y_max': 0, 'width': 1, 'height': 1, 'scheme': 'xyz'}

[Moult]

Results:

I also notice that after pressing add, my EPSG changes to 4326 automatically.

Log:

2019-06-12T16:10:52     WARNING    Loading native dll...
2019-06-12T16:10:53     WARNING    Vector Tiles Reader 3.0.5 (Python 3.6.0)
2019-06-12T16:11:20     WARNING    Native decoding supported!!! (Windows, 64bit)
2019-06-12T16:11:21     WARNING    Bounds of source: {'zoom': 7, 'x_min': 0, 'x_max': 127, 'y_min': 0, 'y_max': 127, 'width': 128, 'height': 128, 'scheme': 'xyz'}
2019-06-12T16:11:21     WARNING    Loading zoom level '7', bounds: {'zoom': 7, 'x_min': 62, 'x_max': 65, 'y_min': 63, 'y_max': 64, 'width': 4, 'height': 2, 'scheme': 'xyz'}
2019-06-12T16:11:21     WARNING    Tile limit enabled: True (32)
2019-06-12T16:11:21     WARNING    0 tiles in cache. Max. 8 will be loaded additionally.
2019-06-12T16:11:21     WARNING    Getting 8 tiles from source...
2019-06-12T16:11:22     WARNING    Error during network request: Error transferring https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/7/64/62.pbf - server replied: Not Found, https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/7/64/62.pbf
2019-06-12T16:11:22     WARNING    Error during network request: Error transferring https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/7/63/62.pbf - server replied: Not Found, https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/7/63/62.pbf
2019-06-12T16:11:22     WARNING    Error during network request: Error transferring https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/7/63/64.pbf - server replied: Not Found, https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/7/63/64.pbf
2019-06-12T16:11:22     WARNING    Error during network request: Error transferring https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/7/64/65.pbf - server replied: Not Found, https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/7/64/65.pbf
2019-06-12T16:11:22     WARNING    Error during network request: Error transferring https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/7/64/63.pbf - server replied: Not Found, https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/7/64/63.pbf
2019-06-12T16:11:22     WARNING    Error during network request: Error transferring https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/7/63/63.pbf - server replied: Not Found, https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/7/63/63.pbf
2019-06-12T16:11:22     WARNING    Error during network request: Error transferring https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/7/63/65.pbf - server replied: Not Found, https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/7/63/65.pbf
2019-06-12T16:11:22     WARNING    Error during network request: Error transferring https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/7/64/64.pbf - server replied: Not Found, https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer/tile/7/64/64.pbf
2019-06-12T16:11:22     WARNING    Creating 0 layers...
2019-06-12T16:11:22     WARNING    Refreshing layers...
2019-06-12T16:11:22     WARNING    Adding 0 new layers to layer group...
2019-06-12T16:11:22     WARNING    Layer creation complete
2019-06-12T16:11:22     WARNING    Import complete
2019-06-12T16:11:22     WARNING    Loading of zoom level 7 complete! No extent loaded.

Our logs seem quite different? Perhaps the JSON isn't the same?

[mnboos]

It tries to load zoom level 7 for you. And all the network requests fails. Therefore, I assume your QGIS map location is outside of the bounds of the available data. Could you try to scroll out stepwise. Maybe, at one point you'll reach zoom level and it should load something.

[Moult]

Thanks @mnboos I can now see vector tiles. However, they don't seem to be geographically located in the right place? I can see both vector tiles, as well as an OSM base map somewhere between Italy and Spain in EPSG 3857.

[mnboos]

Yes, I realize that now too.

@sfkeller : Do you think it might be related to something visible here: https://tiles.arcgis.com/tiles/XYvGSMHmK4EU0DNo/arcgis/rest/services/BaseMap_withoutDEM/VectorTileServer

{
    "currentVersion": 10.61,
    "name": "BaseMap_withoutDEM",
    "capabilities": "TilesOnly,Tilemap",
    "type": "indexedVector",
    "tileMap": "tilemap",
    "defaultStyles": "resources/styles",
    "tiles": ["tile/{z}/{y}/{x}.pbf"],
    "exportTilesAllowed": false,
    "initialExtent": {
        "xmin": 53766.609476385944,
        "ymin": -1804.5673782131707,
        "xmax": 199102.52514821733,
        "ymax": 63838.688908299424,
        "spatialReference": {
            "wkt": "PROJCS[\"Shenzhen_Local\",GEOGCS[\"GCS_Beijing_1954\",DATUM[\"D_Beijing_1954\",SPHEROID[\"Krasovsky_1940\",6378245.0,298.3]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],PROJECTION[\"Gauss_Kruger\"],PARAMETER[\"False_Easting\",-199250.645],PARAMETER[\"False_Northing\",-2477994.076],PARAMETER[\"Central_Meridian\",111.0],PARAMETER[\"Scale_Factor\",1.0],PARAMETER[\"Latitude_Of_Origin\",0.0],UNIT[\"Meter\",1.0]]"
        }
    },
    "fullExtent": {
        "xmin": 53766.609476385944,
        "ymin": -1804.5673782131707,
        "xmax": 199102.52514821733,
        "ymax": 63838.688908299424,
        "spatialReference": {
            "wkt": "PROJCS[\"Shenzhen_Local\",GEOGCS[\"GCS_Beijing_1954\",DATUM[\"D_Beijing_1954\",SPHEROID[\"Krasovsky_1940\",6378245.0,298.3]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],PROJECTION[\"Gauss_Kruger\"],PARAMETER[\"False_Easting\",-199250.645],PARAMETER[\"False_Northing\",-2477994.076],PARAMETER[\"Central_Meridian\",111.0],PARAMETER[\"Scale_Factor\",1.0],PARAMETER[\"Latitude_Of_Origin\",0.0],UNIT[\"Meter\",1.0]]"
        }
    },
    "minScale": 1.476693380542385E8,
    "maxScale": 563.3138200921574,
    "tileInfo": {
        "rows": 512,
        "cols": 512,
        "dpi": 96,
        "format": "pbf",
        "origin": {
            "x": -1.0201387142540421E7,
            "y": 7524142.421540422
        },
        "spatialReference": {
            "wkt": "PROJCS[\"Shenzhen_Local\",GEOGCS[\"GCS_Beijing_1954\",DATUM[\"D_Beijing_1954\",SPHEROID[\"Krasovsky_1940\",6378245.0,298.3]],PRIMEM[\"Greenwich\",0.0],UNIT[\"Degree\",0.0174532925199433]],PROJECTION[\"Gauss_Kruger\"],PARAMETER[\"False_Easting\",-199250.645],PARAMETER[\"False_Northing\",-2477994.076],PARAMETER[\"Central_Meridian\",111.0],PARAMETER[\"Scale_Factor\",1.0],PARAMETER[\"Latitude_Of_Origin\",0.0],UNIT[\"Meter\",1.0]]"
        },
        "lods": [{
                "level": 0,
                "resolution": 39070.84569351727,
                "scale": 1.476693380542385E8
            }, {
                "level": 1,
                "resolution": 19535.422846758636,
                "scale": 7.383466902711925E7
            }, {
                "level": 2,
                "resolution": 9767.711423379318,
                "scale": 3.6917334513559625E7
            }, {
                "level": 3,
                "resolution": 4883.855711689659,
                "scale": 1.8458667256779812E7
            }, {
                "level": 4,
                "resolution": 2441.9278558448295,
                "scale": 9229333.628389906
            }, {
                "level": 5,
                "resolution": 1220.9639279224148,
                "scale": 4614666.814194953
            }, {
                "level": 6,
                "resolution": 610.4819639612074,
                "scale": 2307333.4070974765
            }, {
                "level": 7,
                "resolution": 305.2409819806037,
                "scale": 1153666.7035487383
            }, {
                "level": 8,
                "resolution": 152.62049099030185,
                "scale": 576833.3517743691
            }, {
                "level": 9,
                "resolution": 76.31024549515092,
                "scale": 288416.67588718457
            }, {
                "level": 10,
                "resolution": 38.15512274757546,
                "scale": 144208.33794359228
            }, {
                "level": 11,
                "resolution": 19.07756137378773,
                "scale": 72104.16897179614
            }, {
                "level": 12,
                "resolution": 9.538780686893865,
                "scale": 36052.08448589807
            }, {
                "level": 13,
                "resolution": 4.769390343446933,
                "scale": 18026.042242949035
            }, {
                "level": 14,
                "resolution": 2.3846951717234663,
                "scale": 9013.021121474518
            }, {
                "level": 15,
                "resolution": 1.1923475858617332,
                "scale": 4506.510560737259
            }, {
                "level": 16,
                "resolution": 0.5961737929308666,
                "scale": 2253.2552803686294
            }, {
                "level": 17,
                "resolution": 0.2980868964654333,
                "scale": 1126.6276401843147
            }, {
                "level": 18,
                "resolution": 0.14904344823271665,
                "scale": 563.3138200921574
            }
        ]
    },
    "maxzoom": 18,
    "resourceInfo": {
        "styleVersion": 8,
        "tileCompression": "gzip",
        "cacheInfo": {
            "storageInfo": {
                "packetSize": 128,
                "storageFormat": "compactV2"
            }
        }
    },
    "serviceItemId": "527e0a10a0dd4c9a9cecdc430a35e220",
    "maxExportTilesCount": 100000,
    "minLOD": 0,
    "maxLOD": 18
}

[sfkeller]

@StefanJaq: Mapbox Vector Tiles (MVT) "MAY be used to represent data with any projection and tile extent scheme." See https://github.com/mapbox/vector-tile-spec/tree/master/2.1#3-projection-and-bounds: "Web Mercator is the projection of reference and the Google tile scheme is the tile extent convention of reference". Even forthcoming MVT version 3 sticks to Web Mercator and leaves open any standardization on CRS and tile extent scheme.

Esri has built into the vector tile generator (e.g. of ArcGIS Pro) many extensions (like CRS) and optimizations (like a different scheme and like object generalization). So AFAIK there are some options to override in ArcGIS Pro in order to generate MVT 2 compatible tiles.

So, before continuing to resolve this issue, I strongly recommend to make sure that the tiles created have Web Mercator (EPSG:3857) and stick to the Goog tile scheme.

[ninitartaruga]

@sfkeller @mnboos Thanks, we’re very grateful that you are working on it.

We’re already testing for a while the integration from vector tiles in QGIS. The vector tile should look like in the screenshot:

This is the url to vector tile server in ArcGIS Online: (This link will be only temporarly published) https://tiles.arcgis.com/tiles/Ok7pjj1jT9rEneC7/arcgis/rest/services/testvector_20190429/VectorTileServer

First of all, we have modified the style file, as described above. (see attached root.json) root.json.txt

Afterwards we added it with Vector Tiles Reader to QGIS:

The tiles are created with ArcGIS Online / Bing Maps / Google Maps Tiling scheme and are Web Mercator with EPSG: 3857.

Do you have any idea why the vector tiles look different and with less features?

[ninitartaruga]

Hi @mnboos,

would it be possible that you have a look at the issue above? Or have you already done that? Do you have new information about that?

[mnboos]

Ah sorry.

I did not debug it yet.

Esri has built into the vector tile generator (e.g. of ArcGIS Pro) many extensions (like CRS) and optimizations (like a different scheme and like object generalization). So AFAIK there are some options to override in ArcGIS Pro in order to generate MVT 2 compatible tiles. So, before continuing to resolve this issue, I strongly recommend to make sure that the tiles created have Web Mercator (EPSG:3857) and stick to the Goog tile scheme.

So according to your information, this requirements should be met. I'll have to debug it. I'll try to look at it on Sunday.

[sfkeller]

@ninitartaruga: Just wondering: Do you think that Esri or someone could sponsor this feature? In any case, I assume nobody refuses inkind payments :-).

[mnboos]

@ninitartaruga From what I can tell, it seems to work as expected. There are no errors in the plugins log and the features map OSM.

Why there are less features I don't know. But since the plugin is showing some content, I would assume there might be a misconfiguration on the server side.

[kommen]

I’ve used the information from this issue to get maps.wien.gv.at vector basemap, an ESRI Vector TileCache, loaded in QGIS. I’ve written it up here: https://nextjournal.com/kommen/use-wien-vector-basemap-with-qgis

Still needs some work, but it’s at least something.

[wilhelmberg]

@mnboos thanks for this awesome plugin.

I've created a proxy that generates (hopefully proper) style/tile.json on the fly from the files provided by the ArcGIS Vector Tile Cache for Austrian basemap VT

Details here: https://bergwerkgis.github.io/vtbasemapat/

Unfortunately I'm seeing inconsistent behavior using the plugin, especially with regards to styling.

Also from time to time I get

TypeError: '>' not supported between instances of 'int' and 'NoneType'

here at line 527 where max_zoom doesn't have a value:

https://github.com/geometalab/Vector-Tiles-Reader-QGIS-Plugin/blob/d8666a219cedd798f1d74795e9248a50498fec8c/plugin/vtr_plugin.py#L522-L532

Not sure where these problems come from (my style/tile.json, plugin, data, ...). A demo map using Mapbox GL JS seems to work fine tough: https://bergwerkgis.github.io/vtbasemapat/map.html

Any way I can help debug/solve this?

Btw: fields is empty in my tile.json because the ArcGIS service doesn't provide that information - or I just haven't found it yet. Probably would have to extract that information from the style or the tiles themselves - but that's a task for another weekend.

- EDIT - As a side note: some tiles are a bit heavy:

/cc @ninitartaruga @kommen

[sfkeller]

Wait a moment: Does this mean that ArcGIS (Pro) generates MVT compatible data - against all assumptions - and the core compatibility issues lie 1. in the metadata and 2. in the styling (style.json)?

[kommen]

@sfkeller That is how I—bearing my very limited understanding and GIS experience—understood it, yes. At least when the export from ArcGIS (Pro) is done using the EXPLODED setting. If this setting is not used, ArcGIS will produce .bundle files, which seem to be Esri Compact Cache V2 and from which the tiles have to be extracted first.

[wilhelmberg]

I don't own or have access to any ESRI products so these are all my assumptions or things I've read somewhere on the internet (don't know if they are still true, or were ever true):

• ESRI MVTs use spec 2.0 not 2.1
• ESRI's default tile cache uses tiles of different size at the same zoom level, depending of how much data they hold
• you can export (explode) an ESRI tile cache to be in a OSM/Google compatible structure

The orginal tile.json of the Austrian data does have the exploded flag: https://maps.wien.gv.at/basemapv/bmapv/3857/

Yes, if the cache is exploded I think it's just a metadata issue.

[wilhelmberg]

@kommen 😄 you beat me by seconds 😂

[mnboos]

@BergWerkGIS Thanks for your efforts! The problem is that the style converter is pretty incomplete and a bit error prone. The MB GL style specification is to complex in order for me to maintain the style converter properly. Of course, errors like the above one shouldn't happen.

edit: Oh wait, sorry. I still seem to be a bit tired, the bug you reported comes from the plugin and not the style converter, which is indeed bad.

[wilhelmberg]

@mnboos I totally understand that a full blown style converter would be a huge undertaking.

Let me know if you need anything for debugging the other issue - might be an error on my side as well when mangling the ESRI JSONs.

[williamscraigm]

Got pulled into this via twitter. A few comments / questions: A) The Esri implementation uses the vector tile spec and the mapbox-gl style spec. There is not a tile json representation for tiles B) The difference between MVT 2.0 and 2.1 is just documentation. Where is this causing issues? C) "ESRI's default tile cache uses tiles of different size at the same zoom level, depending of how much data they hold" - think this is referring to overzoom of tiles? Can you confirm? D) From a tile request standpoint, you shouldn't see CompactCacheV2. The tile handler returns a vector tile.

[wilhelmberg]

@williamscraigm

A) The Esri implementation uses the vector tile spec and the mapbox-gl style spec. There is not a tile json representation for tiles

Exactly, that's the problem. Where a tilejson should be linked there's some other json that throws off other tools.

EDIT ---> from the Mapbox Style Spec: https://docs.mapbox.com/mapbox-gl-js/style-spec/#sources

Tiled sources (vector and raster) must specify their details in terms of the TileJSON specification.

<--- EDIT

B) The difference between MVT 2.0 and 2.1 is just documentation. Where is this causing issues?

No issues that I am aware of. Just wanted to make the authors of this plugin aware that ESRI is not using the latest version number just in case that might be checked somewhere in the code.

C) "ESRI's default tile cache uses tiles of different size at the same zoom level, depending of how much data they hold" - think this is referring to overzoom of tiles? Can you confirm?

No, not referring to overzoom.

As I said above

I don't own or have access to any ESRI products so these are all my assumptions or things I've read somewhere on the internet (don't know if they are still true, or were ever true)

So, no way for me to say with certainty.

But I was thinking about a presentation I've seen years ago, similar to this which clearly suggests (at least to me) that tiles have a smaller extent, where there is more data

D) From a tile request standpoint, you shouldn't see CompactCacheV2. The tile handler returns a vector tile.

That's perfect 👍

And that's also why this plugin, my Mapbox GL JS demo and maputnik are working with the ArcGIS tile cache - after creating a proper tilejson.

[sfkeller]

Hi Craig. Thanks for joining this issue. Ad A) Yes, that tilejson is missing. At least this plugin needs it in order to know max_zoom and other metadata. But this is something we can work around. Ad B) Don't think so. Ad C) I'm pretty sure that MVT assumes that all tiles at same zoom level have same (spatial) size - and there's no tile index. Overzooming is just the fact that the client can show tiles at zoom levels > maz_zoom (often z14) by taking the tile at z14 scaling it up. Ad D) don't know, and even can't yet differenciate "bundle" from "Vector Tile Packages".

My intention here is to find out how ArcGIS (Pro, or other Esri SW) can be configured so that it generates vector tiles that are compatible to MVT and can be consumed by a client like this QGIS plugin. That's what I'm aware of what ArcGIS Pro users must configure:

1. Projection: Web Mercator
2. Tiling Schema: Google/Bing Maps/ArcGIS Online Schema
3. Indexed tiling: No

Something missing?

[williamscraigm]

a) This TileJSON specific addition to source is new to me. Versioning of the style spec has been an ongoing problem with Mapbox as they don't explicitly manage versions since they merged the spec repo into their implementation repo. That said, minzoom / maxzoom has defaults if not specified. As stated above, TileJSON is insufficient for many needs and full spec is not required for most mapbox-gl compatible clients. Mapbox themselves has created their mbtiles metadata spec to makeup for many such inefficiencies. b) Version gets written to the tiles as an unsigned integer. So 2.0 vs 2.1 is not able to be represented. c) The example @BergWerkGIS points to is the indexed overzoom. Basically, overzoom starts earlier than z14. This isn't a "different size" thing - it's a "kick your overzoom implementation in sooner" thing. d) VTPK is a zip file of bundles and the service data (root json, style json etc)

For maximum interchangeability produced from ArcGIS Pro, it's:

1. Projection: Web Mercator (3857)
2. Tiling Schema: Google/Bing Maps/ArcGIS Online Scheme
3. Indexed tiling: No

We have R&D work going on WRT supporting more clients even with Indexed tiles.

[wilhelmberg]

thanks for you speedy answer @williamscraigm

a) This TileJSON specific addition to source is new to me.

As I understand it, the first explicit mention of TileJSON appeared in style spec v5, dating 2014:

either TileJSON or a URL to a TileJSON must be provided

https://github.com/mapbox/mapbox-gl-style-spec/blob/24790118b768c03680d4b16820aa849b271fe824/reference/v5.json#L49

b) Version gets written to the tiles as an unsigned integer. So 2.0 vs 2.1 is not able to be represented.

Oh yes, you are right. I had forgotten that just the major version is epxressed in the layer.

c) The example @BergWerkGIS points to is the indexed overzoom

Thanks for the clarification. Was a misunderstanding as neither index nor overzoom appear in the context of the slide.

We have R&D work going on WRT supporting more clients even with Indexed tiles.

👌

[sfkeller]

"indexed overzoom" is interesting, but I assume you understand, that adding yet another complexitiy to the VT spec. has lower priority than interoperability and compatibility.

Regarding TileJSON spec. A (vector tiles) client in a loosely coupled, open environment, needs some metadata. minzoom / maxzoom may have defaults, but 'bounds' and the (source) 'json' are required.

IMHO TileJSON (metadata.json, https://github.com/mapbox/tilejson-spec ) and MBTiles (metadata table, https://github.com/mapbox/mbtiles-spec/blob/master/1.3/spec.md ) must be consistent.

Citing MBTiles spec.: "In a future revision of this specification, the bounds, minzoom, and maxzoom rows of the metadata table will be mandatory."

=> Assuming you/ArcGIS are'nt going to support MBTiles soon, I'd like to focus on interoperability with TileJSON/MBTiles compatible metadata json (and vector tile handler servers returning tiles in the usual tiling schema).

Just for testing I'd suggest to look at how t-rex (https://t-rex.tileserver.ch/ ) and especially how GDAL (https://gdal.org/drivers/vector/mvt.html ) are handling TileJSON metadata.json .

P.S. I wish there could be a meeting about VT with you, Blake (@flippmoke) and Jeff (@slarjy) at FOSS4G end of August...

[wilhelmberg]

@sfkeller

"indexed overzoom" is interesting, but I assume you understand, that adding yet another complexitiy to the VT spec. has lower priority than interoperability and compatibility.

That's nothing the client would have to worry about. The server has to handle that transparently.

[StefanJaq]

As member of the team realising the basemap.at-VTC and as I started the discussion here, I'm really happy for your inputs and discussion.

Sorry for not responding earlier, but at the moment I'm "out-of-office" and will be back during next week.

Nevertheless some short comments:

a) TileJSON We had some discussion about the tilejson allready at the beginning of this thread. The Esri implementation doesn't provide such a file at the moment. See: https://github.com/geometalab/Vector-Tiles-Reader-QGIS-Plugin/issues/170#issuecomment-354437916 We could request Esri that it might be included in a future release. If it is mandatory in the future I'm sure that Esri will do this. It might be possible to provide the tilejson.style to the basemap.at manually, but this will have to be decided by the project team.

b) At the moment the basemap.at VTC uses the "Indexed" option which is a Esri specification so that the tilesize is dependent on the amount of features. We (the basemap.at group) decided for using this in a first step. It might be changed, it is part of the discussion. ArcGIS can create non-indexed tiles.

c) @BergWerkGIS at the moment there are no fields (attributes) included. This might be included in a further release.

d) @BergWerkGIS There are other tools except ArcGIS Pro that can allready use the basemap.at Vektor. ;-) See: https://weboffice.mysynergis.com/en/news/article/id/basemapat-in-vector-tile-format-gain-first-insight-with-the-weboffice-flex-client/

https://woportal.mysynergis.com/BasemapAT/synserver?project=BasemapAT&client=flexjs&language=en

f) Again, thanks a lot for your work and input.

I want to add that I can't speak for the basemap.at team.

[sfkeller]

@BergWerkGIS : Happy to hear this that a client doesn't need the overzoom index(?). There was a similar but simpler trick I've seen in MVT (Mapbox vector tile) sources, that some single tiles aren't existing/served (between default minzoom z8 and maxzoom z14), being NULL in MBTiles and/or giving error 404 over http. Clients then instead take a tile at z8 by default. This saves much data/space e.g. in monotone sea, forest or snowy areas.

But what we should focus here, is

1. achieve compatible tiles format => seems feasible looking at Craig's recommendations above.
2. agree on a minimal set of metadata (bounds, minzoom, maxzoom, source/'json') => seems close to an agreement(?)
3. agree that MVT servers should generate and publish a metadata.json (TileJSON). See the nice documentation and implementation here https://gdal.org/drivers/vector/mvt.html

[wilhelmberg]

👋 @StefanJaq thanks for chiming in.

If it is mandatory in the future

Has been since 2014 https://github.com/geometalab/Vector-Tiles-Reader-QGIS-Plugin/issues/170#issuecomment-520720629

It might be possible to provide the tilejson.style to the basemap.at manually

Not urgent, my tool takes care of that on the fly.

but this will have to be decided by the project team.

I think project team is more than happy to finally share their great effort with a broader audience, not just ~ArcGIS Pro~ ESRI based software users.

At least that's what I get form my little tool being linked to on the https://basemap.at/ main page already.

Really looking forward to more interoperability in the near(?) future 🚀

---

@sfkeller

Happy to hear this that a client doesn't need the overzoom index(?).

Yes, at least that's how I understand it should work.

[StefanJaq]

Thanks to your hints we found errors in the basemap.at VTC implementation regarding the mapbox specification. E.g.: Layernames appearing multiple times, missing attributes at some text-nodes, furthermore the allready discussed missing TileJSON.

We have adjusted those settings at our development area. I don't know by now when those new files will be public available. Hopefully soon.

Thanks again.

[kommen]

@StefanJaq On basemap.at in the News section it says the root.json of the vector basemap.at now is adheres the Mapbox 2.0 spec. So I assume this is live now?