WMTS in WorldMap

[capooti]

@blewis @wendy-guan as per last email: GeoExplorer right now don't support WMTS layers. In order to display WMTS layers a WMTS plugin must be implemented, which may not be a really easy operation. I am filing this ticket in order to update you on this issue.

[wendy-guan]

Is this related to, or the same as, #10 ?

[blewis]

I'm removing #10 and bringing those options I described here to consolidate:

The idea is that we are looking for the most sensible approach and if that means finding a TMS version or converting WMTS to TMS that is fine. TMS is supported in GeoNode, right?

1) Find TMS or other version of these same layers that work. Can GeoTIFF versions of these be found or created? 2) Can MapProxy be used to cascade a WMTS into a TMS or WMS which can be handled normally as a base map or regular layer? 3) Write a WMTS plugin for Geoexplorer, load layers manually as base layers. 4) Write a WMTS plugin for Geoexplorer, add WMTS harvesting to HHypermap. Add WMTS layer loading to Map composition client. NOTE: Is new plugin needed for map composer or layer page or both?

[blewis]

These are key layers for Japan. Perhaps we bring them in another way if WMTS doesn't work?

1. the layer in Japanese: https://maps.gsi.go.jp/xyz/std/{z}/{x}/{y}.png
2. the satellite layer: https://maps.gsi.go.jp/xyz/seamlessphoto/{z}/{x}/{y}.jpg
3. the boundary layer: https://maps.gsi.go.jp/xyz/blank/{z}/{x}/{y}.png

They are famous and widely used so I think that they are most suitable for base layer. I guess that it is quite useful if we can chose them as one.

==== These also might be relevant for Japan

1. https://gsi-cyberjapan.github.io/experimental_wmts/gsitiles_wmts.xml
2. https://gsi-cyberjapan.github.io/experimental_wmts/gsitiles_wmts_light.xml

==== Here is the China endpoint: http://gis.sinica.edu.tw/ccts/wmts

[capooti]

if they are WMTS they need a WMTS source in GeoExplorer I can have a try proxying them in MapProxy and see if they can be loaded as WMS-C or TMS in the map composer. Keep in mind that if we proxy them there will be a much larger load on Hypermap if using them as base layers.

[capooti]

These are key layers for Japan. Perhaps we bring them in another way if WMTS doesn't work?

1. the layer in Japanese: https://maps.gsi.go.jp/xyz/std/{z}/{x}/{y}.png
2. the satellite layer: https://maps.gsi.go.jp/xyz/seamlessphoto/{z}/{x}/{y}.jpg
3. the boundary layer: https://maps.gsi.go.jp/xyz/blank/{z}/{x}/{y}.png

which is the standard or format used by these services?

[wendy-guan]

Please provide instructions to Wang Lijun on how to add any of the maps in http://gis.sinica.edu.tw/ccts/wmt. Thanks!

[wendy-guan]

In http://gis.sinica.edu.tw/ccts/wmts, only the first layer (the one now added to staging) is in jpeg format. All other layers are in png format. So the solution now can only work on the first layer but not the rest of the layers. Please demonstrate the ability for adding png format layers using 清代歷史地圖(Qing), id=ad1820.

[capooti]

@wendy-guan png is supported. I have just added a layer from this service: https://services.arcgisonline.com/arcgis/rest/services/Demographics/USA_Population_Density/MapServer/WMTS/ and you can notice that is succesfully displayed in GeoExplorer.

I spent a couple of hours to figure out why that layer (ad1820) isn't displayed by GeoExplorer. It looks like the sinaca server is returning an error (500) when sending requests for tiles from that layer, for example:

http://gis.sinica.edu.tw/ccts/wmts?SERVICE=WMTS&REQUEST=GetTile&VERSION=1.0.0&LAYER=ad1820&STYLE=default&TILEMATRIXSET=GoogleMapsCompatible&TILEMATRIX=4&TILEROW=7&TILECOL=11&FORMAT=image%2Fpng

This is instead not happening with the other layer, AMS L500 (sinica server correctly returns a valid tile):

http://gis.sinica.edu.tw/ccts/wmts?SERVICE=WMTS&REQUEST=GetTile&VERSION=1.0.0&LAYER=AMS_L500&STYLE=default&TILEMATRIXSET=GoogleMapsCompatible&TILEMATRIX=6&TILEROW=25&TILECOL=50&FORMAT=image%2Fjpeg

I'd suggest to check with sinica staff (if possible) if they have something in the logs which can be used to address the error.

[wendy-guan]

Please try the following and see if you can add any one of them successfully. These are the historical maps Prof. Xu cares the most. It would be surprising if none of them works. Thanks!

西漢歷史地圖(W. Han)bc0007 東漢歷史地圖(E. Han)ad0140 三國歷史地圖(Sanguo)ad0262 西晉歷史地圖(W. Jing)ad0281 東晉歷史地圖(E. Jing)ad0382 南北朝歷史地圖(South and North)ad0497 隋代歷史地圖(Sui)ad0612 唐代歷史地圖(Tang)ad0741 北宋歷史地圖(N. Song)ad1111 南宋歷史地圖(S. Song)ad1208 元代歷史地圖(Yuan)ad1330 明代歷史地圖(Ming)ad1582 清代歷史地圖(Qing)ad1820

[capooti]

Unfortunately no one of them seem to work using GeoExplorer which relies on OpenLayers (an old version, maybe now it was improved). I see that they are working on QGIS, I would be curious to check what is the QGIS request format to see what could be the difference, let me check if there is a way.

[capooti]

@wendy-guan would it be possible to ask sinica why one request works and the other not? That would help

[wendy-guan]

Sure. You may write to them directly. The contact info is here: https://ccts.sinica.edu.tw/contact.php?lang=en. Thanks!

[capooti]

@wendy-guan @blewis I added WMTS support in Hypermap Registry today. Now it is possible to register a WMTS endpoint in Registry and use it from the WorldMap search interface based on Solr.

For example in CAMP create a new map, and search for the Esri demographics layer (in Registry it is here: http://128.31.22.46:8001/registry/hypermap/service/bd6b5a3c-1e5b-4e29-beea-01855c163791/):

Add the layer in the map and it should be displayed as in this picture:

[wendy-guan]

Can you show how to add one of the historical maps from the list I sent earlier (Prof. Xu's list)? I tried to Submit a Map Service URL but got "ValueError at /maps/add_endpoint" http://128.31.22.46/maps/add_endpoint

@wendy-guan I will check why this raises an error. creating a new ticket for it.

[capooti]

There is a user guide for Hypermap: https://cga-harvard.github.io/Hypermap-Registry/ Anyway I can add it for you and see what happens...

[wendy-guan]

Why did add_endpoint gave me error even before I entered any URL? (Even though I know now that is not the right place to add WMTS endpoints.)

If you could add all these layers to Hypermap and make them searchable that would be great:

西漢歷史地圖(W. Han)bc0007 東漢歷史地圖(E. Han)ad0140 三國歷史地圖(Sanguo)ad0262 西晉歷史地圖(W. Jing)ad0281 東晉歷史地圖(E. Jing)ad0382 南北朝歷史地圖(South and North)ad0497 隋代歷史地圖(Sui)ad0612 唐代歷史地圖(Tang)ad0741 北宋歷史地圖(N. Song)ad1111 南宋歷史地圖(S. Song)ad1208 元代歷史地圖(Yuan)ad1330 明代歷史地圖(Ming)ad1582 清代歷史地圖(Qing)ad1820

[capooti]

@wendy-guan the problem with this service (sinica) is that all of the layers in hypermap are marked as invalid as they get a server error (500) for tile 0/0/0

In hypermap: http://128.31.22.46:8001/registry/hypermap/service/95d6ddd9-34fb-4f38-bb8e-c22baa86338f/

You can check the error for one of the layer here: http://128.31.22.46:8001/registry/hypermap/layer/cb416348-85c4-4926-ad0c-a2a4a0fd305e/checks/

Therefore, being invalid, Hypermap doesn't index the layers in Solr, thus they are not displayed in the WM search interface

[wendy-guan]

The Sinica server is working fine since I can add it to ArcMap. This screenshot shows the ad1820 layer.

[cid:image002.jpg@01D585B0.EEFA7F70]

[capooti]

@wendy-guan I didn't write is not working, I said that for Hypermap they are not as the tiles 0/0/0 is not available, therefore they are considered invalid and not usable in WM search interface. This is the current behaviour of Hypermap

[wendy-guan]

Sorry I misunderstood it earlier. In this case Sinica people may not be able to help us (their system isn't broken). Is there any workaround from Hypermap side? If tile 0/0/0 is not available, is tile 1/1/1 available?

[capooti]

This is the corresponding line in Hypermap - which uses OWSLib to check the status of a layer using tile 0/0/0 (https://github.com/cga-harvard/Hypermap-Registry/blob/master/hypermap/aggregator/models.py#L679).

I would be interested in the opinion of @tomkralidis which is the maintainer of the OSWLib and involved in OGC for standards definition. Tom, could we consider a service which returns 500 for tile 0/0/0 not respecting the standard?

For the service in question, tile 0/0/0 request results in error 500: http://gis.sinica.edu.tw/ccts/wmts?SERVICE=WMTS&REQUEST=GetTile&VERSION=1.0.0&LAYER=AMS_L500&STYLE=default&TILEMATRIXSET=GoogleMapsCompatible&TILEMATRIX=0&TILEROW=0&TILECOL=0&FORMAT=image%2Fjpeg

While other tiles seem to correctly work, for example for tile 4/6/11:

http://gis.sinica.edu.tw/ccts/wmts?SERVICE=WMTS&REQUEST=GetTile&VERSION=1.0.0&LAYER=AMS_L500&STYLE=default&TILEMATRIXSET=GoogleMapsCompatible&TILEMATRIX=4&TILEROW=6&TILECOL=11&FORMAT=image%2Fjpeg

@wendy-guan regarding your question, I cannot modify the behavior of Hypermap just for one service which has this kind of issue. In my tests I have seen that there are other tiles returning error (500), including tile 1/1/1. I would suggest you to speak with @bglewis for figuring out possible options.

[wendy-guan]

How about these layers for Japan? Are they also returning error (500)? the layer in Japanese: https://maps.gsi.go.jp/xyz/std/{z}/{x}/{y}.png the satellite layer: https://maps.gsi.go.jp/xyz/seamlessphoto/{z}/{x}/{y}.jpg the boundary layer: https://maps.gsi.go.jp/xyz/blank/{z}/{x}/{y}.png https://gsi-cyberjapan.github.io/experimental_wmts/gsitiles_wmts.xml https://gsi-cyberjapan.github.io/experimental_wmts/gsitiles_wmts_light.xml

[capooti]

@wendy-guan I will have a look at them on Tuesday and let you know

[tomkralidis]

@capooti as per http://schemas.opengis.net/wmts/1.0/wmtsGetTile_request.xsd:

<element name="TileRow" type="nonNegativeInteger">
    <annotation>
        <documentation>Row index of tile matrix</documentation>
    </annotation>
</element>
<element name="TileCol" type="nonNegativeInteger">
    <annotation>
        <documentation>Column index of tile matrix</documentation>
    </annotation>
</element>

so nonNegativeInteger is defined in https://www.w3.org/TR/xmlschema11-2/#nonNegativeInteger as 0 or greater. So I suspect the server should not be returning 500 based on the request.

[capooti]

thanks @tomkralidis

[capooti]

hi @wendy-guan

1) Regarding these layers:

the layer in Japanese: https://maps.gsi.go.jp/xyz/std/{z}/{x}/{y}.png
the satellite layer: https://maps.gsi.go.jp/xyz/seamlessphoto/{z}/{x}/{y}.jpg
the boundary layer: https://maps.gsi.go.jp/xyz/blank/{z}/{x}/{y}.png

Could you provide the capabilities links?

2) Regarding this layer:

https://gsi-cyberjapan.github.io/experimental_wmts/gsitiles_wmts.xml

it returns an error when adding it in Hypermap. The error is caused by the underlying OWSLib library. I have created a ticket for this error here:

https://github.com/geopython/OWSLib/issues/622

3) Last layer:

https://gsi-cyberjapan.github.io/experimental_wmts/gsitiles_wmts_light.xml

Unfortunately still a problem with OWSLib, I still didn't create a ticket - I am first checking with developers. I will keep you posted...

[capooti]

@wendy-guan I checked with OWSLib developers, and unfortunately support for Python 2.7 has been recently dropped: https://github.com/geopython/OWSLib/commit/430342c024f74d70a6cf78d1af810e803822997d

Hypermap relies on Python 2.7, therefore it must be migrated to Python 3 if we want to use fixes in OWSLib (or request new fixes) which addresses our issues - one has been already merge by Tom Kralidis in OWSLib master.

Also, note that Python 2 is at end of life in 2 months - so it may be worth to start planning an Hypermap upgrade to Python 3 if needed in the future.

[wendy-guan]

I will defer to @blewis for providing the capabilities links for these layers: the layer in Japanese: https://maps.gsi.go.jp/xyz/std/{z}/{x}/{y}.png the satellite layer: https://maps.gsi.go.jp/xyz/seamlessphoto/{z}/{x}/{y}.jpg the boundary layer: https://maps.gsi.go.jp/xyz/blank/{z}/{x}/{y}.png

[wendy-guan]

@capooti By your estimation, how much work is it to upgrade to Python 3 for Hypermap, and who has the skill to do it if not you? Do you think both sinica and cyberjapan problems are likely due to OWSLib issues which will be fixed once we upgrade Hypermap to Python 3? Of course you never know for sure, but what is your educated guess? Is there any other way to (even manually by admin) add sinica and cyberjapan layers to WM as either base maps or savable layers in a template map? Thanks!

[capooti]

• how much work is it to upgrade to Python 3 for Hypermap, and who has the skill to do it if not you?

It is difficult to say, it can be a few hours or a few days depending on a few things. For GeoNode we are experiencing problems mainly because many libraries we are using with GeoNode - which uses Python 2.7 - were still not ported to Python 3. For Hypermap here are the libraries we are using: https://github.com/cga-harvard/Hypermap-Registry/blob/master/requirements.txt First thing to do would be to check if all of these libraries were ported to Python 3. If not it can be a big problem as Hypermap code must be rewritten using other libraries or - as an alternative - forking existing unported libraries and maybe send PRs to have them ported by the dev team.

I think Devika and Tao could work on this: what is required is mainly knowledge of Python and Django/web developments

• Do you think both sinica and cyberjapan problems are likely due to OWSLib issues which will be fixed once we upgrade Hypermap to Python 3?

For cyberjapan it is very likely. I tried OWSLib new version with that service and OWSLib parsed correctly. You can test it on your own by using the same code provided for the WMTS OWSLib usage here: https://geopython.github.io/OWSLib/ For seneca I believe is rather a problem of the implementation of the WMTS service endpoint. But to know this for sure some study must be done, which unfortunately is time consuming :(

• Is there any other way to (even manually by admin) add sinica and cyberjapan layers to WM as either base maps or savable layers in a template map?

Unfortunately not. I didn't try with cyberjapan, I can give a try. For Seneca, as you noticed, only one layer seems to work

[blewis]

Regarding Japan WMTS endpoints, lets use this: https://gsi-cyberjapan.github.io/experimental_wmts/gsitiles_wmts.xml More info here: https://github.com/gsi-cyberjapan/experimental_wmts

[capooti]

@blewis as per my previous message (22 October) and following discussion with @wendy-guan that wmts cannot be used in Hypermap/GeoExplorer right now unless we migrate Hypermap Registry to Python 3

[blewis]

Thanks Paolo. I don't mean harvesting a WMTS endpoint and loading to hhypermap registry. I mean having a manual procedure for adding individual WMTS layers as base layers in CAMP maps.

[capooti]

Hi Ben, could you provide a layer from this service which works for you in QGIS/ArcGIS? I tried adding a few ones, I don't get errors but I don't see any tiles using QGIS Then I will try to add that in GeoExplorer as a WMTS layer

[blewis]

Here is a layer for high res imagery

地震、台風・豪雨等、火山_地震_平成28年熊本地震_写真_写真（2007年～）_ort20160414image/jpegz2to18

Here is a screenshot

[capooti]

I don't see a layer named ort20160414 using the service at https://gsi-cyberjapan.github.io/experimental_wmts/gsitiles_wmts.xml

this is what I see:

[capooti]

@blewis I have added that layer in CAMP (not sure why on QGIS didn's show up in layers list). Now on CAMP we have 3 different WMTS:

1) Demographics_USA_Population_Density from services.arcgisonline.com this correctly works and it was added like this in GeoNode:

{
        "source": {
            "url": "https://services.arcgisonline.com/arcgis/rest/services/Demographics/USA_Population_Density/MapServer/WMTS/",
            "ptype": "gxp_wmtssource",
            "title": "Demographics_USA_Population_Density",
            },
        "group": "background",
        "name": "Demographics_USA_Population_Density",
        "visibility": False,
        "fixed": True,
    }

2. 二十五萬分一中國圖(AMS L500) from http://gis.sinica.edu.tw/ccts/wmts this partially work (not only tiles are rendered) and this is how was added in GeoNode

{
        "source": {
            "url": "http://gis.sinica.edu.tw/ccts/wmts",
            "ptype": "gxp_wmtssource",
            "title": "二十五萬分一中國圖(AMS L500)",
            },
        "group": "background",
        "name": "AMS_L500",
        "visibility": False,
        "fixed": True,
    }

3) ort_old10 from https://gsi-cyberjapan.github.io/experimental_wmts/gsitiles_wmts.xml this doesn't work and was added in GeoNode in this way:

{
        "source": {
            "url": "https://gsi-cyberjapan.github.io/experimental_wmts/gsitiles_wmts.xml",
            "ptype": "gxp_wmtssource",
            "title": "ort_old10",
            },
        "group": "background",
        "name": "ort_old10",
        "visibility": False,
        "fixed": True,
    }

Looking at the tile errors in GeoExplorer it looks like response is truncated (maybe because is too long)?

[capooti]

This is the request GeoExplorer is sending for a specific tile:

https://gsi-cyberjapan.github.io/experimental_wmts/gsitiles_wmts.xml?SERVICE=WMTS&REQUEST=GetTile&VERSION=1.0.0&LAYER=ort_old10&STYLE=default&TILEMATRIXSET=z2to17&TILEMATRIX=4&TILEROW=7&TILECOL=7&FORMAT=image%2Fpng

It looks like it is taking many seconds (try opening the above link on your browser) and at some point the browser stops it. This is why tiles are not rendered. It would be interesting to know how QGIS handles this and how its request differs from GeoExplorer...

[blewis]

Use the layer "ort" just above the one in your screen shot

[capooti]

This is how GeoExplorer sends the request to the working ESRI WMTS endpoint:

https://services.arcgisonline.com/arcgis/rest/services/Demographics/USA_Population_Density/MapServer/WMTS/?SERVICE=WMTS&REQUEST=GetTile&VERSION=1.0.0&LAYER=Demographics_USA_Population_Density&STYLE=default&TILEMATRIXSET=default028mm&TILEMATRIX=4&TILEROW=5&TILECOL=3&FORMAT=image%2Fpng

it could be a problem of the TILEMATRIXSET parameter, maybe the exposed ones in the Japanase layer are not compatible with the tilematrixset OpenLayers is looking for... So no idea how can I help with this

[blewis]

It is not an esri service. Here is URL https://maps.gsi.go.jp/xyz/seamlessphoto/{TileMatrix}/{TileCol}/{TileRow}.jpg

[capooti]

It is not an esri service. Here is URL https://maps.gsi.go.jp/xyz/seamlessphoto/{TileMatrix}/{TileCol}/{TileRow}.jpg

you may have lost a few of my comments. I am saying that the ESRI WMTS endpoint is the only one properly working in the three layers I have added using the GeoExplorer WMTS plugin.

[blewis]

Oh sorry, so geoexplorer WMTS doesn't handle the WMTS coming from the https://maps.gsi.go.jp server ? QGIS handles it but geoexplorer only works for Esri type WMTS?

[capooti]

@blewis I am saying that the WMTS plugin for GeoExplorer doesn't seem to like all of the WMTS endpoint. In the tests I did only the ESRI one is working properly, while the other ones don't work (Japan) or work partially (Sinica). So I am afraid that I can't help too much here. I have already spent hours and hours of development (for adding WMTS support in Hypermap and GeoExplorer) and debugging. I believe this ticket should be closed as the fact that we are getting issues to show all of the WMTS layers doesn't depend on my code but on problems on underlying libraries.

[blewis]

Thank you Paolo for all your work on this. Any thoughts on how to best implement robust WMTS (maybe using QGIS as standard) in this version of WorldMap?

[capooti]

@blewis there are two distinct things which can be definitely improved for WMTS in WorldMap:

1) GeoExplorer client in WorldMap relies for WMTS on a very old version of the WMTS plugin of OpenLayers (7/8 years ago). Running the client on a new version of OpenLayers would make things more solid but this mean to migrate the WM client to GeoExt 3 [1]. Also using MapStore - the new GeoNode client - could be an option but not sure about the WMTS support there

2) Hypermap: OWSLib, the library which we use to harvest web services has now been discontinued for Python 2, so even if we report bugs and they fix them we can't benefit from it. I'd recommend to move Hypermap to Python 3, this way it will be possible to use very latest version of OWSLib (and other libraries)

[1] https://geoext.github.io/geoext3/