mbjones
commented
1 year ago And here's the same layer viewed in QGIS against the high-res (50cm) RGB imagery form the state of alaska WMTS server (rather than OSM):
Open mbjones opened 1 year ago
mbjones
commented
1 year ago And here's the same layer viewed in QGIS against the high-res (50cm) RGB imagery form the state of alaska WMTS server (rather than OSM):
mbjones
commented
10 months ago While we can produce a WMTS for the entire globe, another consideration might be to create a subsetted WMTS focused on a particular area of interest (AOI) which the user might specify with a polygon in the MetacatUI tool. This would, for example, generate a derived WMTS document that constrains the data layer to the AOI while providing the metadata needed to load the data at mutliple zoom levels in mapping software. This would be a "virtual" subset of the data layer, and might be useful if people want to focus on a specific region and not default to the entire global layer.
yvonnesjy
commented
7 months ago Hey Matt! I've read through the WMTSCapabilities document you linked, with the question of defining an area of interest in mind. Here are some questions:
mbjones
commented
7 months ago @yvonnesjy -- yes, by derived, I simply mean another WMTS that is generated from the full dataset WMTS description and only includes a subset of the data -- that can be via changing the extent, reducing the features list, the tile zoom levels supported, and reducing the formats, among other things, which makes the dataset appear smaller than the original, and constrains the choices.
As I generated the original WMTS by hand just as an example of the format, I might not have matched the smallest bounding box for the IWP data. I was hoping we would automate the production of WMTS metadata during our workflow runs and publish them with the tilesets (probably in the root of the tileset directory).
yvonnesjy
commented
7 months ago Thanks Matt- that makes sense! I also downloaded QGIS and have successfully imported iwp_high through the https://arcticdata.io capabilities url.
My next step is to test out some xml changes locally to understand the schema better. You mentioned a local download of WMTSCapabilities.xml should work equally, but I haven't found a way to specify it as a WMTS connection. Can you show me how?
yvonnesjy
commented
7 months ago Reading QGIS's documentation, I also noticed a potential issue-
WGS 84 Bounding Box — ... Some WMS servers do not set this correctly (e.g., UTM coordinates are used instead). If this is the case, then the initial view of this layer may be rendered with a very ‘zoomed-out’ appearance by QGIS. ...
I wonder if this means that the client (at least QGIS) won't strictly follow a smaller bounding box set by a derived capabilities file, which would explain why I still see the dataset outside of the WGS84BoundingBox you set @mbjones
mbjones
commented
7 months ago @yvonnesjy Could be, I'm not sure. But note the quote talks about WMS services, and not WMTS. WMS is an older specification (still in widespread use) and behaves somewhat differently from WMTS, so I wonder if the same limitations apply there. WMS isn't really a tiling standard (requests are specified in geocoordinates, and return a server-provided map image). SO has an overview of the differences beween WMS and WMTS..
Regarding the larger display extent, that could be a simple artifact of the tiling approach. If the bbox requested 1) doesn't exactly match the boundaries of tiles, or 2) is not the same aspect ratio as the viewport being rendered, then I think most WMTS clients will retrieve all of the tiles needed to cover the bounding box and the viewport, which can be a fair bit larger than the bbox itself. Could this account for what you are seeing?
yvonnesjy
commented
7 months ago AH I have a hypothesis! For your bounding box, @mbjones , I think it actually wraps around the world from -179 to 179, instead of crossing the antimeridian like I assumed. So showing the full dataset is expected:) (But then that begs the question: how do I define the bounding box if I do want it to cross the antimeridian?)
I've verified with a modified capabilities file (through url, haven't found a way to use local capabilities.xml yet) that QGIS handles changes to extent (through ows:WGS84BoundingBox), and zoom levels (through TileMatrixSet) correctly. Here are clarifications to the remainder of your comment:
ianguerin
commented
6 months ago I would be very curious to see a working example of a GetCapabilities response that works with GeoTIFF image type instead of PNGs. So far I've had no luck loading some from a GitHub search, or trying to write my own. I'd like to be able to visualize it in QGIS, and I'm assuming it will be loaded in much the same way as the WMTSGetCapabilities.xml file you, @mbjones , posted above.
I have gotten to the point in QGIS where there is a logged error message
Returned image is flawed [Content-Type: image/tiff; URL: https://arcticdata.io/data/10.18739/A2KW57K57/iwp_geotiff_high/WGS1984Quad/1/3/0.tif]Which points to a valid tif file, which I can load into QGIS. So it's clearly trying to load various GeoTIFF tiles, but seems to fail in doing so
mbjones
commented
6 months ago @ianguerin The error you cite seems to indicate the TIFF file is not valid. I downloaded 0.tif from the URL you provided at the ADC, and I also have trouble opening it in any application. It is possible the file is not valid, we should explore that with @julietcohen and @robyngit.
I went and looked up a WMTS tileset that supports multiple formats. This example is from SentinelHub (see their API examples). I was able to use QGIS to access their capabilities URL and load a GeoTIFF layer in QGIS, and recolorize it with a palette of my choice.
SentinelHub doesn't use the RESTFUL http encoding that we do in our WMTS, and instead use KVP encoding for service access, but it does show how to support multiple formats. The main difference is we need to provide the proper ResourceURL template for our REST approach too. To access SentinelHub, you need an API key to access the service (free, just create an account and a use the SimpleWMS configuration instance id everywhere below where I mention <INSTANCE_ID>).
Here's the capabilities document, which I retrieved with curl but edited out a lot of the details so I could paste it in this issue (there are more layers and TileMatrixSets than I am showing here):
curl -s "https://services.sentinel-hub.com/ogc/wmts/<INSTANCE_ID>?REQUEST=GetCapabilities" > capabilities.xmlNote how each Layer has a list of formats that are supported for that Layer:
<Format>image/png</Format>
<Format>image/jpeg</Format>
<Format>image/jpg</Format>
<Format>image/tiff</Format>
<Format>application/vnd.google-earth.kmz+xml</Format>
<Format>image/tiff;depth=8</Format>
<Format>image/tiff;depth=16</Format>
<Format>image/tiff;depth=32f</Format>
<Format>application/vnd.google-earth.kmz+xml;image_type=image/jpeg</Format>
<Format>application/vnd.google-earth.kmz+xml;image_type=image/png</Format>
<Format>application/x-esri-shape</Format>
<Format>application/json</Format>You can get a specific tile in a specific format by providing the FORMAT=image/tiff parameter to the GetTile request, like this:
curl -s "https://services.sentinel-hub.com/ogc/wmts/<INSTANCE_ID>?REQUEST=GetTile&BBOX=3238005,5039853,3244050,5045897&RESOLUTION=10&TILEMATRIXSET=PopularWebMercator512&LAYER=FALSE-COLOR&MAXCC=20&TILEMATRIX=14&TILEROW=3065&TILECOL=4758&TIME=2018-03-29/2018-05-29&FORMAT=image/tiff" -o tile.tiffThat command saves a single tile in a file named tile.tiff. You can pick any of the formats from the supported list in the WMTS capabilties document. Again, that works with a KVP service, but for our case of using REST URIs, we'd need to provide format-specific ResourceURL templates for each format.
ianguerin
commented
6 months ago I may have shared a bad example, another tile mentioned in the QGIS logs is https://arcticdata.io/data/10.18739/A2KW57K57/iwp_geotiff_high/WGS1984Quad/2/0/0.tif which I can load as an independent Raster in QGIS, but doesn't work when trying to load the WMTS layer.
Here's the slimmed down GetCapabilities response I've been working with:
Where file:///home/path/to/file is the actual filepath
ianguerin
commented
6 months ago After much trial and error, I think @yvonnesjy and I have figured out the issue. It seems that GeoTIFFS that has an interleave metadata option of "BAND" are not rendering properly through WMTS in QGIS, but interleave metadata option of "PIXEL" is working fine when loaded as WMS/WMTS.
Here's a link to GDAL docs about creation options. Notably:
... but some applications only support pixel interleaved TIFF files.I was able to convert a band-interleaved file to pixel-interleaved and QGIS displayed it when loaded as WMS/WMTS. I think the next step is to run our viz pipeline on a subset of one of our datasets (Ice-wedge polygons is what I've been looking at) and change the GeoTIFF creation to override the default interleave creation option (which is band by default).
yvonnesjy
commented
6 months ago Today's update:
Interleave turned out to be a correlation, not a cause. By updating the band data type to int32 (instead of float64) (thanks to @ianguerin !), QGIS downloads the tifs successfully. Yesterday when we exported the file as pixel interleaved, it changed the data type to byte, which turned out to also be supported.
This thread suggests there are limitations to the data types WMTS can support. Both that and its linked thread suggest WCS is the way to go. The description of "an open specification for sharing raster datasets on the web" does fit our use case.
@mbjones Is WCS actually what we want, rather than WMTS?
@ianguerin also posted a question to continue exploring the WMTS path.
mbjones
commented
6 months ago WCS may be the way to go, but I haven't looked at it in years. It is from the older family of OGC specifications, and would likely require a dedicated WCS server IIRC (whereas WMTS can be served from any web location, no special server needed) -- but we'd need to explore that further.
mbjones
commented
6 months ago Reading the threads you linked, I see they are talking about WMS and not about WMTS -- WMS is the older family spec as well, and we've not been using it.
mbjones
commented
6 months ago One last comment (sorry for the stream)... WMTS enables serving arbitrary formats, and uses different identifiers for different types. For example, here's the list from SentinelHub for the example I posted the other day:
<Format>image/png</Format>
<Format>image/jpeg</Format>
<Format>image/jpg</Format>
<Format>image/tiff</Format>
<Format>application/vnd.google-earth.kmz+xml</Format>
<Format>image/tiff;depth=8</Format>
<Format>image/tiff;depth=16</Format>
<Format>image/tiff;depth=32f</Format>
<Format>application/vnd.google-earth.kmz+xml;image_type=image/jpeg</Format>
<Format>application/vnd.google-earth.kmz+xml;image_type=image/png</Format>
<Format>application/x-esri-shape</Format>
<Format>application/json</Format>Notice that the tiff formats include a depth parameter. Have you tried setting that to match our file format?
ianguerin
commented
6 months ago Is there a way to determine what "depth" our GeoTIFFs are? I tried various values, mostly anytime I added ;depth=X to the Format string I started seeing 301 HTTP errors in the network tab in QGIS.
mbjones
commented
6 months ago The GDAL GTiff driver (https://gdal.org/drivers/raster/gtiff.html) is the defacto format definition as it is used under the hood for most geo tools, including QGIS it seems. That documentation says:
Currently band types of Byte, UInt16, Int16, UInt32, Int32, Float32, Float64, CInt16, CInt32, CFloat32 and CFloat64 are supported for reading and writing
How that maps to the depth string in WMTS is unknown to me, but I'll bet depth=16 corresponds to a GTiff drive pixel data type of Int16, and depth=32f corresponds to Float32. QGIS should be able to open any file that was created with the GDAL GTiff driver.
Each band in the raster can have its own data type, and gdalinfo will report the actual datatype of each band -- see this example showing a band with type Byte: https://gdal.org/programs/gdalinfo.html#example I'm not sure how those map onto the Format string options in WMTS. Hope some of this is useful to you.
ianguerin
commented
6 months ago That was my assumption as well. No luck using the depth parameter, though. QGIS has no problem opening the GeoTIFF; to be clear it is just when I try to load it via WMTS. I'll keep exploring though. The response I got on my stackexchange question seems to think that GeoTIFFs are not the primary use case for WMTS. Quote: "It is recommended that servers offer Tiles in the image/png and image/jpeg file formats"
yvonnesjy
commented
6 months ago Some thoughts and findings:
We found the qgis line that throws the error, which points to QImage from qt gui not able to parse the image data (line 4859). Further researching on QT says that it likely can’t read float tiffs and accepts a finite range of format encodings: https://stackoverflow.com/questions/35404348/how-assert-that-a-tiff-file-conforms-to-standard
I also did more reading on WCS, and think that it likely does not support tiled images, but behaves more like WMS (returns only one .tif). See screenshot from a super old random pdf that explains the service
robyngit
commented
3 weeks ago Document summarizing some experiments with this: https://docs.google.com/document/d/1pVGlacPihHXCqO9451bVh68VeIyHC5erPx2g0RD2UQY/edit?tab=t.0
The tiled data products and map tile products produced from viz-workflow conform to the WorldCRS84Quad TileMatrixSet, and so can be described as a [WMTS]()-compliant tileset that can be loaded easily into any GIS system that supports WMTS, such as QGIS and various python libraries. As our tiles are structured and being delivered through a REST API endpoint on our web server, all we need to provide is the
GetCapabilitiesresponse document that describes the tileset and its layers. These can be provided both for the map tiles in png format, but also for our tiled Geotiff, geopackage, and other format files. WMTS even supports a syntax to describe the same data product with different formats using different resource url formats, so we should discuss how to best arrange our file hierarchies for this.Attached is a proof of concept for the PNG tiles from the Witharana et al. (https://doi.org/10.18739/A2KW57K57) ice wedge polygon dataset. This response document is installed on our servers at the following url location:
That URL is all that is needed to load the data layer into QGIS or another software platform.
And here is a screenshot showing the layer loaded in QGIS automatically:
TODO:
@julietcohen @robyngit @iannesbitt Feedback and other ideas appreciated. Thanks, and let's discuss.