WMS/WMTS Endpoint

[reyemtm]

Is there any thought to adding a WMS/WMTS endpoint? There is currently a bug in ArcGIS Online/Server where it will not print past zoom level 17 when using the "Add Tile Layer" option. I am using with this server to serve out several years of imagery, however I need print functionality to work properly. I have found that a WMTS service does not suffer from this bug in AGOL. There are a few examples of WMTS endpoints in GitHub, with one example being https://github.com/DenisCarriere/mbtiles-server/. I am attempting to get this JS based server to work since I know JS but not GO. However this project has more contributors and would love to see WMTS in this project.

[brendan-ward]

I'm not entirely sure I follow the bug with ArcGIS here, so I want to learn a bit more about that specifically (which should be in a separate issue if it is a bug on our end). Are you serving tiles on mbtileserver but consuming those in ArcGIS Online / ArcGIS server using the ArcGIS API provided by mbtileserver? Have you ruled out any bug with that API here?

It looks like WMTS has a fairly large API surface area: it allows tileset metadata and tile requests in Key/Value Pair, SOAP, and RESTful API structures (with recommendations in the spec to at least support Key/Value Pair and RESTful APIs). Which of those specifically do you need here?

It looks like we could use the mbtiles-server as inspiration for how to implement the WMTS API.

[reyemtm]

First off, I am using this project to serve out imagery that I consume in ArcGIS Online and Mapbox GL JS. So far this has worked fairly well. In AGOL you can add services in a variety of ways -

I have been using "A Tile Layer". I could not get the ArcGIS API to work - I have the endpoint running but I cannot add tiles to AGOL from this endpoint. So, when maps are then printed with the AGOL print function using this tile layer method, the tiles are only pulled from a max zoom level of 17. No doubt this is a bug on the Esri end, but they will have not recognized it as such, since the other methods work and since any tile layer added this way would not be created from an Esri product, most likely. There are also some issues on getting the xyz endpoint to work directly in ArcGIS Pro (same issue, tiles will not load past zoom 17 or 18).

Here is a map where you can see what I mean - it will work for now but may not stay up forever.

https://lancastergis.maps.arcgis.com/home/webmap/viewer.html?webmap=9d2c02ea2e9e49a3ad248df1da8d14b1

Simply print and you can see what is going on.

I have successfully added a third-party WMS service and printed from that service and that works - it is on the map.

The WMTS XML seems to be very finicky. I have a version of the mbtiles-server running, but cant seem to get the layer to show in AGOL, even though it shows up in ArcGIS Pro. There is something up with the xml most likely but I have not figured it out yet. Anyway, having a WMS endpoint would make this project even more useful as WMS/WMTS is an OGC standard and seems to be more universally accepted than simply the xyz endpoint.

It looks like there have been several attempts at this in the past and all the projects seem to have faded away.

Some I have looked into:

tileserver.php AppGeo/WMS mbtiles-server

[reyemtm]

https://www2.ci.lancaster.oh.us/tileserver/arcgis/rest/services/2020_fairfield_3in_z21/MapServer

https://www2.ci.lancaster.oh.us/tileserver/services

https://www2.ci.lancaster.oh.us/wmts-server/2020_fairfield_3in_z21/WMTS this is the output from mbtiles-server

[reyemtm]

In terms of which of the spec I need to add data to AGOL...I can tell you that this service works -

https://sampleserver6.arcgisonline.com/arcgis/rest/services/WorldTimeZones/MapServer/WMTS/1.0.0/WMTSCapabilities.xml

And the WMS endpoint that is in the map works, but it's not clear what all urls are needed to get the layers onto the map.

[brendan-ward]

We should not support WMS; mbtileserver doesn't provide server-side rendering (expected by that API). WMTS is appropriate for pre-rendered tiles, so I'm not opposed to that but it would be a bit of an implementation lift. First, I want to make sure that it would solve this specific issue.

Thanks for sharing examples we can use for exploring this. In your [AGOL example], I believe that the WMS layer ("Stream Raster WMS Works") works at all zooms because it is providing dynamic rendering for all zooms. The WMTS layer associated with the WMS layer ("StreamRaster Tile Layer Does Not Work") does not, because the tiles return HTTP 204 / no data beyond a certain zoom level (example). I'm not able to quickly find the tileset info for the WMTS service to find out what zoom levels are supported. Can you share the root WMTS URL you used when you added this to AGOL?

The ArcGIS APIs that mbtileserver provides not working in AGOL could be a bug. We only minimally built those APIs out and tested in other ESRI JSAPI viewers, not AGOL. AGOL might be expecting something we do not provide here. In theory, it seems like the cleanest way to make sure that tiles served up from mbtileserver work for printing in AGOL is to make sure they can first be added there (opened #114 for this).

Since the World Time Zones WMTS source is published on an ArcGIS server, I'd hope that AGOL would also be able to consume that. However, it looks like maybe this was added to your map as an ArcGIS layer instead of a WMTS layer? The link / description in your map for this one is an ArcGIS API endpoint. I can't seem to add the WMTS endpoint specifically as a WMTS layer; the add layer panel rewrites the URL and then adds it as an ArcGIS layer.

It may be possible to compare the XML from the WMTS endpoint for this layer to that provided by mbtiles-server to identify what is missing / different that AGOL expects in order to work correctly. One of the problems with OGC specs is that they are too open-ended, and more narrowly supported in certain clients (e.g., in AGOL). Thus implementing WMTS in mbtileserver does not necessarily ensure that it will work in AGOL until we know what it is that AGOL expects (requires investigation).

I was able to add one of your tilesets to AGOL and I think I reproduced the issue. I can zoom to level 19 and it displays sharply in the interactive map, but in the print version, it uses a lower zoom level and uses lower resolution image from your tileset. No good! It calls a backend service inside AGOL to make the image for printing, but doesn't pass in parameters to specify what zoom level to use or what zoom levels are supported from the service. That points to a bug on the AGOL side.

I think in order to be sure that WMTS sidesteps this issue, you'd need a WMTS working in AGOL that has zoom levels beyond 17. I don't trust the World Time Zones example here because even though it claims to go through level 18, when you zoom in that far in the map, it is actually fetching tiles at level 8(?!). And eyeballing the results in the print image, it looks lower resolution that when I'm seeing at level 18 for that layer in the map.

[reyemtm]

Ok, great. There's a lot to unpack here. In terms of the World Time Zone WMTS, that's just the first one I found but yes we need a better example here.

I'll respond to your other questions in time.

Thanks!

On Fri, Jul 30, 2021, 5:41 PM Brendan Ward @.***> wrote:

We should not support WMS; mbtileserver doesn't provide server-side rendering (expected by that API). WMTS is appropriate for pre-rendered tiles, so I'm not opposed to that but it would be a bit of an implementation lift. First, I want to make sure that it would solve this specific issue.

Thanks for sharing examples we can use for exploring this. In your [AGOL example], I believe that the WMS layer ("Stream Raster WMS Works") works at all zooms because it is providing dynamic rendering for all zooms. The WMTS layer associated with the WMS layer ("StreamRaster Tile Layer Does Not Work") does not, because the tiles return HTTP 204 / no data beyond a certain zoom level (example https://raster.stream.woolpert.io/layers/3jAnYdujBfWE4KAjPW9c2C/tiles/22/1134503/1592225?key=AIzaSyA0SlUTOqihxCWvSIDxWQmj3Hzd7KQeYEA). I'm not able to quickly find the tileset info for the WMTS service to find out what zoom levels are supported. Can you share the root WMTS URL you used when you added this to AGOL?

The ArcGIS APIs that mbtileserver provides not working in AGOL could be a bug. We only minimally built those APIs out and tested in other ESRI JSAPI viewers, not AGOL. AGOL might be expecting something we do not provide here. In theory, it seems like the cleanest way to make sure that tiles served up from mbtileserver work for printing in AGOL is to make sure they can first be added there (opened #114 https://github.com/consbio/mbtileserver/issues/114 for this).

Since the World Time Zones WMTS source is published on an ArcGIS server, I'd hope that AGOL would also be able to consume that. However, it looks like maybe this was added to your map as an ArcGIS layer instead of a WMTS layer? The link / description in your map for this one is an ArcGIS API endpoint. I can't seem to add the WMTS endpoint https://sampleserver6.arcgisonline.com/arcgis/rest/services/WorldTimeZones/MapServer/WMTS/1.0.0/WMTSCapabilities.xml specifically as a WMTS layer; the add layer panel rewrites the URL and then adds it as an ArcGIS layer.

It may be possible to compare the XML from the WMTS endpoint for this layer to that provided by mbtiles-server to identify what is missing / different that AGOL expects in order to work correctly. One of the problems with OGC specs is that they are too open-ended, and more narrowly supported in certain clients (e.g., in AGOL). Thus implementing WMTS in mbtileserver does not necessarily ensure that it will work in AGOL until we know what it is that AGOL expects (requires investigation).

I was able to add one of your tilesets https://www2.ci.lancaster.oh.us/tileserver/services/2020_fairfield_3in_z21 to AGOL and I think I reproduced the issue. I can zoom to level 19 and it displays sharply in the interactive map, but in the print version, it uses a lower zoom level and uses lower resolution image from your tileset. No good! It calls a backend service inside AGOL to make the image for printing, but doesn't pass in parameters to specify what zoom level to use or what zoom levels are supported from the service. That points to a bug on the AGOL side.

I think in order to be sure that WMTS sidesteps this issue, you'd need a WMTS working in AGOL that has zoom levels beyond 17. I don't trust the World Time Zones example here because even though it claims to go through level 18, when you zoom in that far in the map, it is actually fetching tiles at level 8(?!). And eyeballing the results in the print image, it looks lower resolution that when I'm seeing at level 18 for that layer in the map.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/consbio/mbtileserver/issues/113#issuecomment-890168703, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQ2HULAJGA6AAXDTB2QNTDT2ML75ANCNFSM5BIQTSFA .

[reyemtm]

I'm going to try and reply to your questions one by one.

I believe stream raster is tiled for this service to zoom 21.

Agree with your point about the ArcGIS endpoint, getting that working would be great.

On Fri, Jul 30, 2021, 5:41 PM Brendan Ward @.***> wrote:

We should not support WMS; mbtileserver doesn't provide server-side rendering (expected by that API). WMTS is appropriate for pre-rendered tiles, so I'm not opposed to that but it would be a bit of an implementation lift. First, I want to make sure that it would solve this specific issue.

Thanks for sharing examples we can use for exploring this. In your [AGOL example], I believe that the WMS layer ("Stream Raster WMS Works") works at all zooms because it is providing dynamic rendering for all zooms. The WMTS layer associated with the WMS layer ("StreamRaster Tile Layer Does Not Work") does not, because the tiles return HTTP 204 / no data beyond a certain zoom level (example https://raster.stream.woolpert.io/layers/3jAnYdujBfWE4KAjPW9c2C/tiles/22/1134503/1592225?key=AIzaSyA0SlUTOqihxCWvSIDxWQmj3Hzd7KQeYEA). I'm not able to quickly find the tileset info for the WMTS service to find out what zoom levels are supported. Can you share the root WMTS URL you used when you added this to AGOL?

The ArcGIS APIs that mbtileserver provides not working in AGOL could be a bug. We only minimally built those APIs out and tested in other ESRI JSAPI viewers, not AGOL. AGOL might be expecting something we do not provide here. In theory, it seems like the cleanest way to make sure that tiles served up from mbtileserver work for printing in AGOL is to make sure they can first be added there (opened #114 https://github.com/consbio/mbtileserver/issues/114 for this).

Since the World Time Zones WMTS source is published on an ArcGIS server, I'd hope that AGOL would also be able to consume that. However, it looks like maybe this was added to your map as an ArcGIS layer instead of a WMTS layer? The link / description in your map for this one is an ArcGIS API endpoint. I can't seem to add the WMTS endpoint https://sampleserver6.arcgisonline.com/arcgis/rest/services/WorldTimeZones/MapServer/WMTS/1.0.0/WMTSCapabilities.xml specifically as a WMTS layer; the add layer panel rewrites the URL and then adds it as an ArcGIS layer.

It may be possible to compare the XML from the WMTS endpoint for this layer to that provided by mbtiles-server to identify what is missing / different that AGOL expects in order to work correctly. One of the problems with OGC specs is that they are too open-ended, and more narrowly supported in certain clients (e.g., in AGOL). Thus implementing WMTS in mbtileserver does not necessarily ensure that it will work in AGOL until we know what it is that AGOL expects (requires investigation).

I was able to add one of your tilesets https://www2.ci.lancaster.oh.us/tileserver/services/2020_fairfield_3in_z21 to AGOL and I think I reproduced the issue. I can zoom to level 19 and it displays sharply in the interactive map, but in the print version, it uses a lower zoom level and uses lower resolution image from your tileset. No good! It calls a backend service inside AGOL to make the image for printing, but doesn't pass in parameters to specify what zoom level to use or what zoom levels are supported from the service. That points to a bug on the AGOL side.

I think in order to be sure that WMTS sidesteps this issue, you'd need a WMTS working in AGOL that has zoom levels beyond 17. I don't trust the World Time Zones example here because even though it claims to go through level 18, when you zoom in that far in the map, it is actually fetching tiles at level 8(?!). And eyeballing the results in the print image, it looks lower resolution that when I'm seeing at level 18 for that layer in the map.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/consbio/mbtileserver/issues/113#issuecomment-890168703, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQ2HULAJGA6AAXDTB2QNTDT2ML75ANCNFSM5BIQTSFA .

[reyemtm]

FYI it looks like one of my issues with the ArcGIS MapServer endpoint is CORS related, so I'm trying to get that figured out and will report back if I resolve the CORS issue.

Edit - Fixed CORS issue, working on the AGOL issue

[reyemtm]

A quick update on the tile layers not printing at high zoom levels on AGOL, Esri is looking into it. I will report back with any information.

[reyemtm]

I have a slightly modified version of https://github.com/DenisCarriere/mbtiles-server working in AGOL with print and no shift. I'm thinking maybe it's better to target this endpoint than the elusive ArcGIS MapServer (https://github.com/consbio/mbtileserver/issues/114#) endpoint? The XML in the mbtiles-server is formatted fine, which is actually coming from here. I just made a few changes to the mbtiles-server dependencies so it would actually run in a modern version of Node. I would much rather run a GO script than run this node script with a very shaky foundation.

The links that AGOL pulls for the WMTS service are -

root + / layer + /WMTS/1.0.0/WMTSCapabilities.xml

with the tiles served at

root + / layer + /WMTS/tile/1.0.0/2020_fairfield_3in_z21/default/GoogleMapsCompatible/z/y/x

[reyemtm]

Looks like there is a GO utility for WMTS - https://github.com/PDOK/ogc-specifications. I'm wondering if we can use this to implement a very basic WMTS endpoint.

[brendan-ward]

I'd like to avoid extra dependencies, but we should be able to take some inspiration from that.

[reyemtm]

I've started to try and tackle the WMTS endpoint and had some issues with the deeply nested fields but am making progress.

On Fri, Aug 6, 2021, 6:28 PM Brendan Ward @.***> wrote:

I'd like to avoid extra dependencies, but we should be able to take some inspiration from that.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/consbio/mbtileserver/issues/113#issuecomment-894548647, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQ2HUL4KOTXM2U6VQAESHLT3ROX5ANCNFSM5BIQTSFA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

[Techmaven-geospatial]

You can add WMTS support without writing any code Just take your XYZ url paste it into This site https://wmts.maptiler.com/ Download the XML file generated (GET CAPABILITIES) Host that file on your server Give that URL to map client wanting WMTS