[BUG] Custom tileset urls with {s} and {ext} values are not properly parsed

[harabat]

Description of problem Base maps can be generated with custom tilesets through add_tile(XXXXTileLayer(url, attribution)). Most urls for custom tilesets are of the type https://tiles.yourtileprovider.come/{z}/{x}/{y}.png, but many are like https://{s}.yourtileprovider.com/{z}/{x}/{y}.png or https://{s}.yourtileprovider.com/{z}/{x}/{y}.{ext} (where {s} is the subdomain and {ext} is the format), which breaks add_tile(): instead of showing a basemap, it will show an empty grid. The way I had reimplemented this feature in issue #10057, I went with a hacky workaround where:

• I replace {s} by 'a', not sure how hacky it is though, as folium's code seems to be doing the same thing by default, even though subdomains can be specified (the relevant code is in class TileLayer in raster_layers.py in folium, if you're interested)
• I replace {ext} by png (I noticed that both jpg and png work). Folium itself is broken by leaving the {ext} though (I'll raise an issue there)

Possible solutions Specifying a url with {s} and {exp} should work straight away. Maybe add_tile() could try to replace {s} and {ext} with default values ('a' and 'png') and the docstring could mention something along the lines of If you specify a url for a custom tileset, it needs to be of the type https://tiles.yourtileprovider.come/{z}/{x}/{y}.png, so that at least people will know where to look for the problem if their url doesn't work.

Alternatives considered People could try and find clean urls, but this is not straightforward, as (i) the intricacies of custom tiles are not known by most people who want to work with maps and (ii) most resources, like OpenStreetMap's list of tile servers and this list of free tile providers, don't consider {s} to be a problem.

Additional context The description of the subdomains on OSM's tile servers page:

In the URLS to Carto.com and some others there is also a {s} which should be replaced by the subdomain[1]. Subdomains are used to help with browser parallel requests per domain limitation. These subdomains are typically single letters 'a', 'b' and 'c'. Refer to the provider documentation for details of available subdomains.

This works:

from bokeh.models import WMTSTileSource

p = figure(x_range=(-2000000, 6000000), y_range=(-1000000, 7000000),
           x_axis_type="mercator", y_axis_type="mercator", width=500, height=500)
p.add_tile(
    WMTSTileSource(
        url='https://stamen-tiles-a.a.ssl.fastly.net/watercolor/{z}/{x}/{y}.jpg',
        attribution='Map tiles by <a href="http://stamen.com">Stamen Design</a>, <a href="http://creativecommons.org/licenses/by/3.0">CC BY 3.0</a> &mdash; Map data &copy; <a href="https://www.openstreetmap.org/copyright">OpenStreetMap</a> contributors'
    )
)
show(p)

Replacing the clean url above by the original url (https://stamen-tiles-{s}.a.ssl.fastly.net/watercolor/{z}/{x}/{y}.{ext} or https://stamen-tiles-{s}.a.ssl.fastly.net/watercolor/{z}/{x}/{y}.png) willl return this:

[bryevdv]

@harabat I don't understand why the extension would need to be templatized? A given tile provider renderer will only ever load one kind of image type. It seems like it should always be specified explicity in the URL that is configured on a tile provider.

For the subdomain, that's a more interesting case, but you need to clarify what the desired behaviour is. Are you suggesting that TileProvider get a new subdomains property that is a list of strings, and that whenever BokehJS fetches a tile it should round-robin through that list, replacing {s} with each one of the strings in turn?

[philippjfr]

Also having trouble understanding why you'd ever template the {ext}. The subdomains a, b and c are often used to get around browser parallel requests per domain limitations, but I'm not sure that's a problem I've ever run into. We could try to experiment by cycling through potential subdomains as @bryevdv suggested and see if that speeds up the requests.

[harabat]

@bryevdv: I don't understand either, but the fact is that image types sometimes are templatized. So while I agree with you that {ext} should not be a thing, the url that I have in my code is one that I have come across without intentionally looking for it, and Stamen is a well-known tile provider as well. Two examples: https://stamen-tiles-{s}.a.ssl.fastly.net/terrain-background/{z}/{x}/{y}.{ext} https://stamen-tiles-{s}.a.ssl.fastly.net/watercolor/{z}/{x}/{y}.{ext}

Now, in the urls above, I'm getting the tiles to render even when replacing {ext} with random text (I've checked, it's not a cache thing), so maybe urls with templatized {ext} are not looking at the extensions anyway, in which case I assume it's safe to replace {ext} with png by default. I don't know whether it's a case we want Bokeh to cover though, given neither of you two has run into this.

@philippjfr: As I mentioned in my question, folium seems to have the subdomains property of their TileLayer class set to 'abc' by default, I don't know how they're cycling through these in the background, but you seem to be knowledgeable about this kind of thing. Would they be looking at requests speeds or whether the requests return an error? Is the suggested default behaviour of making add_tiles() replace {s} with 'a' sufficient?

The simplest workaround would be to just have some explanation of custom tileset urls in the relevant part of Bokeh docs that copies the OSM Wiki paragraph, along the lines of:

Custom tileset urls sometimes include an {s} template, which is the subdomain and helps with browser parallel requests per domain limitation. Subdomains generally equal 'a', 'b', or 'c'. Sometimes the urls come with other templates, like {ext} or {r}. Bokeh can only process urls with {x}, {y}, and {z} templates, so please refer to the provider's documentation to find out appropriate values for the other templates.

[bryevdv]

I'm going to go ahead and close this. Bokeh has since delegated most tile provider handing to the external xyzservices library, which at least at a glance, may seem to support more template options.