Add more templating parameters to raster XYZ url schemes: datetime and subdomain

[jo-chemla]

Feature description

It can be useful to add more templating strings to XYZ/TMS raster sources.

• subdomains: to alleviate the burden on tile servers, often several subdomains can be used to retrieve each tile. OpenLayers supports tile patterns like {0-3} or {a-c} to that end as shown on this OL issue
• temporal map_start_time: for TMS/XYZ servers which do support a temporal dimension (a bit like WMS-T servers, but only changing the xyz tile URL scheme to embed year, month or other), tying the temporal controller slider to xyz parameters could be of great help. Supporting templating strings which would replace expressions like format_date(@map_start_time, 'yyyy-MM') could be useful.

Probable place where this would be implemented: qgswmsprovider.cpp

Additional context

For context, a successfully implemented workaround is to create a macro which, at project opening, would get the xyz tile layer, as well as the temporal controller, and update the tile source on the temporal controller update by connecting the update to the

def update_query_layer():
        tct = tc.dateTimeRangeForFrameNumber(tc.currentFrameNumber()).begin().toPyDateTime()
        newSource = f'type=xyz&url=https://server.com/{tct.year}_{tct.month:02d}' + '/{z}/{x}/{y}.png'
        layer.setDataSource(newSource, f'{tct.year}_{tct.month:02d}', layer.dataProvider().name())

tc = iface.mapCanvas().temporalController()
tc.updateTemporalRange.connect(update_query_layer)

[nyalldawson]

TMS/XYZ servers which do support a temporal dimension

Interesting... Do you have a public example of these servers?

[jo-chemla]

public example of these servers?

Hi Nyall - first, huge thanks for all your work, especially regarding the qgis temporal control which is tremendously helpful!

Some examples

• provided by RealEarth: earthquakes, VIIRS-Fire and Hydro Estimator Rainfall. Some vector sources also exist for these data (rather than using raster tiles), but depending on the use the temporal raster tiles can be useful.
• some landsat historical imagery collections also use year/monh in the TMS tile url scheme like this resourcewatch historical imagery
• NSW Historical
• not public: Planet also offers a TMS service for their imagery basemaps, for those that prefer tiles over their qgis plugin.

Useful resource for Geoserver temporal support for WMS sources

example schemes: 
https://realearth.ssec.wisc.edu/tiles/Earthquake-mag/{yyyy}{MM}{dd}/{hh}{mm}{ss}/{z}/{x}/{y}.png
https://realearth.ssec.wisc.edu/api/image?products=Earthquake-mag&time={yyyy}{MM}{dd}+{hh}{mm}{ss}&x={x}&y={y}&z={z}
https://storage.googleapis.com/landsat-cache/{yyyy}/{z}/{x}/{y}.png
https://tiles0.planet.com/basemaps/v1/planet-tiles/global_monthly_{yyyy}_{MM}_mosaic/gmap/{z}/{x}/{y}.png?api_key={api-key}
https://portal.spatial.nsw.gov.au/tileservices/Hosted/HistoricalImagery{yyyy}/MapServer/tile/14/9829/15069

[jo-chemla]

Another (meteo) example are the TMS raster layers displayed on zoom.earth with these URLS:

// high-res zoom.earth custom merged tiles, combines NOAA, EUMETSAT, NESDIS, METEOSAT etc
https://tiles.zoom.earth/geocolor/msg-zero/2023-05-12/0745/5/15/17.jpg 

// more real-time less hd: NASA Worldview WMS endpoint can be seen as a TMS url
https://gibs-c.earthdata.nasa.gov/wmts/epsg4326/best/wmts.cgi?TIME=2023-05-12T00:00:00Z&layer=MODIS_Terra_CorrectedReflectance_TrueColor&style=default&tilematrixset=250m&Service=WMTS&Request=GetTile&Version=1.0.0&Format=image%2Fjpeg&TileMatrix=3&TileCol=7&TileRow=0

[jo-chemla]

Two workarounds:

• a python script to add one layer per date (per year or per month in range) and set its temporalProperties FixedTemporalRange via layer.temporalProperties().setFixedTemporalRange(QgsDateTimeRange(QDateTime.fromString(f"{year}-{month}-01", "yyyy-MM-dd", QDateTime.fromString(f'{year}-{month}-30', "yyyy-MM-dd"))).
• a better one: connect the temporalController.updateTemporalRange with a function that updates the source TMS url

Temporal control impacting TMS layer

Linked QGIS Feature Request issue: https://github.com/qgis/QGIS/issues/51939

from qgis.utils import iface from datetime import datetime from qgis.core import ( QgsProject, QgsRasterLayer, )

def get_tmsname(tct): return f'TemporalTMS{tct.year}_{tct.month:02d}'

def get_tms_url(tct):

Working

# return f'https://tiles0.planet.com/basemaps/v1/planet-tiles/global_monthly_{tct.year}_{tct.month:02d}_mosaic/gmap/{{z}}/{{x}}/{{y}}.png?api_key={planetApiKey}' 
return f'https://gibs.earthdata.nasa.gov/wmts/epsg3857/all/MODIS_Aqua_CorrectedReflectance_TrueColor/default/{tct.year}-{tct.month:02d}-{tct.day:02d}/GoogleMapsCompatible_Level9/{{z}}/{{y}}/{{x}}.jpg'
# Note for the below sources, data is only available for the past 2 days, so set the temporal controller accordingly
# return f'http://realearth.ssec.wisc.edu/tiles/NESDIS-GHE-HourlyRainfall_{tct.year}{tct.month:02d}{tct.day:02d}_{tct.hour:02d}{(int(tct.minute / 15) * 15):02d}00/{{z}}/{{x}}/{{y}}.png'
# return f'http://realearth.ssec.wisc.edu/tiles/Earthquake-mag_{tct.year}{tct.month:02d}{tct.day:02d}_{tct.hour:02d}{(int(tct.minute / 15) * 15):02d}00/{{z}}/{{x}}/{{y}}.png'
# return f'http://realearth.ssec.wisc.edu/tiles/NESDIS-SST_{tct.year}{tct.month:02d}{tct.day:02d}_{tct.hour:02d}0000/{{z}}/{{x}}/{{y}}.png'
# Not working
# return f'https://realearth.ssec.wisc.edu/api/image?products=globalvis-tsp_{tct.year}{tct.month:02d}{tct.day:02d}_{tct.hour:02d}{(int(tct.minute / 15) * 15):02d}00&x={{x}}&y={{y}}&z={{z}}&client=RealEarth&device=Browser'
# return f'https://tiles.zoom.earth/geocolor/goes-east/{tct.year}-{tct.month:02d}-{tct.day:02d}/1000/{{z}}/{{x}}/{{y}}.jpg' # or goes-east or himawari or msg-zero

tct = datetime(2023, 7, 1) layer = QgsRasterLayer(f'type=xyz&url={get_tms_url(tct)}', get_tms_name(tct), 'wms') QgsProject.instance().addMapLayer(layer, True)

def update_query_layer(): tct = tc.dateTimeRangeForFrameNumber(tc.currentFrameNumber()).begin().toPyDateTime() layer.setDataSource(f'type=xyz&url={get_tms_url(tct)}', get_tms_name(tct), layer.dataProvider().name())

tc = iface.mapCanvas().temporalController() tc.updateTemporalRange.connect(update_query_layer) update_query_layer()

Note some of the above temporal TMS URLs are not working anymore. Planet and NASA MODIS are still working, as well as the Sea-Surface Temp, Rainfall and Earthquakes for the last two days. 

[mueckl]

Here is an example WMTS-capabilities for multiple resourceurls resp. subdomains for resourceType="tile"

https://geoservices.bayern.de/od/wmts/geobasis/v1/1.0.0/WMTSCapabilities.xml

[jo-chemla]

Hi @nyalldawson would you have some pointers as to how to implement this feature, especially to add datetime templating to XYZ url schemes? I've never built qgis, but after parsing through the codebase it seem that interesting bits reside here: src/providers/wms/

• especially ~generic qgswmsprovider.cpp which parses strings like {x} within a function createTileRequestsXYZ with signature void QgsWmsProvider::createTileRequestsXYZ( const QgsWmtsTileMatrix *tm, const QgsWmsProvider::TilePositions &tiles, QgsWmsProvider::TileRequests &requests, QgsRasterBlockFeedback *feedback ))
• and specific qgsxyzconnection.cpp. which is specific to the XYZ source, but simply seem to prepend options like zmin etc to a url.
• The file where both connection implementations are used is within qgswmsdataitems.cpp

I'm not entirely sure how the temporal controller datetime could be passed to that createTileRequestsXYZ function made to replace formatting strings with tile index etc. For formatter litterals, we should probably rely on the official date_time_string_format reference, something along parsing each template litteral of this format: YYYY-MM-DDTHH:mm:ss.sssZ