openstreetmap / operations

OSMF Operations Working Group issue tracking
https://operations.osmfoundation.org/
98 stars 13 forks source link

Use preferred tile.openstreetmap.org URL #737

Open Firefishy opened 2 years ago

Firefishy commented 2 years ago

TL;DR: use https://tile.openstreetmap.org, no subdomains, no short names.

Many projects, examples, and documentation still use old URLs to access the tile.openstreetmap.org service. The URLs should be updated as follows:

tile.openstreetmap.org supports HTTP/2 and HTTP/3 which no longer require the old (a|b|c) aliases to increase browser connection concurrency. Using a single URL improves performance and ability to cache.

The old URLs have a maintenance burden and add complexity. The old URLs may stop working in the future. Please switch to the preferred URL now.

(This issue is for linking 3rd party projects on Github and elsewhere which are being asked to update tile.openstreetmap.org URLs)

wrobell commented 2 years ago

@Firefishy Thanks for reporting this for GeoTiler. Do you mind if I ask you to link to the relevant OSM documentation?

Firefishy commented 2 years ago

@Firefishy Thanks for reporting this for GeoTiler. Do you mind if I ask you to link to the relevant OSM documentation?

This ticket is the documentation ;-) What would you like to see? Add something to https://operations.osmfoundation.org/ or on the blog? https://blog.openstreetmap.org/2022/05/01/meet-grant-slater-the-openstreetmap-foundation-s-new-senior-site-reliability-engineer/

grischard commented 2 years ago

@wrobell you can also see the recommended tile URL at https://wiki.openstreetmap.org/wiki/Tiles and https://wiki.openstreetmap.org/wiki/Tile_servers

wrobell commented 2 years ago

@grischard Thank you

ztNFny commented 2 years ago

Due to this request we discoverd that the tile source for "OSM France" stated on https://wiki.openstreetmap.org/wiki/Tile_servers is wrong. Current: https://tile.openstreetmap.fr/osmfr/${z}/${x}/${y}.png Correct: https://{a|b|c}.tile.openstreetmap.fr/osmfr/${z}/${x}/${y}.png

wrobell commented 2 years ago

Do you mind if I ask for clarification about tile usage policy?

Is "Maximum of 2 download connections" restriction set per URL? If so, using out-of-date URL scheme allowed us to create 6 connections at the same time. Would it be possible to increase this limit for the new scheme?

HolgerJeromin commented 2 years ago

Due to this request we discoverd that the tile source for "OSM France" stated on https://wiki.openstreetmap.org/wiki/Tile_servers is wrong. Correct: https://{a|b|c}.tile.openstreetmap.fr/osmfr/${z}/${x}/${y}.png

@ztNFny I "cleaned up" the wikipage yesterday. I checked if the new url supports http/2 (and therefore do not need a,b,c) but I missed checking the delivered content. 🤦‍♂️ Thanks a lot! Reverted.

Strubbl commented 1 year ago

Matekarte needs an update, too. https://gitlab.com/Strubbl/matekate/-/issues/2

pnorman commented 1 year ago

Current numbers are 82% {s}.tile.openstreetmap.org, 14% tile.openstreetmap.org, 4% {s}.tile.osm.org, and negligible tile.osm.org.

image
Athena query used ```sql SELECT CAST(year AS varchar) || '-' || CAST(month AS varchar) || '-' || CAST(day AS varchar) AS "date", ROUND(CAST("tile.openstreetmap.org" AS REAL)/total, 4) AS "tile.openstreetmap.org", ROUND(CAST("{s}.tile.openstreetmap.org" AS REAL)/total, 4) AS "tile.openstreetmap.org", ROUND(CAST("tile.osm.org" AS REAL)/total, 4) AS "tile.osm.org", ROUND(CAST("{s}.tile.osm.org" AS REAL)/total, 4) AS "{s}.tile.osm.org" FROM (SELECT year, month, day, COUNT(*) AS "total", COUNT(*) FILTER (WHERE host = 'tile.openstreetmap.org') AS "tile.openstreetmap.org", COUNT(*) FILTER (WHERE host LIKE '%.tile.openstreetmap.org') AS "{s}.tile.openstreetmap.org", COUNT(*) FILTER (WHERE host = 'tile.osm.org') AS "tile.osm.org", COUNT(*) FILTER (WHERE host LIKE '%.tile.osm.org') AS "{s}.tile.osm.org" FROM fastly_success_logs_v1 WHERE year = 2022 AND month IN (8, 9) GROUP BY year, month, day ) AS q ORDER BY year, month, day ```
pnorman commented 1 year ago

image

There's been a bit of movement since opened this issue, but this is obviously a long-term move.

Even though we still would like people to use https://tile.openstreetmap.org/, I don't see anything for operations to do at this time.

If you've got a website or app, use https://tile.openstreetmap.org/. With HTTP/2 or HTTP/3, it is significantly faster than the old method with multiple domains. This issue can still be referenced for any tickets in other repos for the move.

wrobell commented 1 year ago

Do you mind if I ask for clarification about tile usage policy?

Is "Maximum of 2 download connections" restriction set per URL? If so, using out-of-date URL scheme allowed us to create 6 connections at the same time. Would it be possible to increase this limit for the new scheme?

Do you mind if I remind about the question above?

tomhughes commented 1 year ago

The limit was never two per domain, so nothing has changed. If you thought a, b and c somehow each got their own limit then that was never our intention - those names all went to the same hosts and only existed to allow increased parallelisation in browsers.

pnorman commented 1 year ago

With HTTP/2 or HTTP3 and multiplexing, you should use one connection and let the server dictate the multiplexing.

otbutz commented 1 year ago

@pnorman Would it be possible to check the Referer header of the clients accessing the old URLs? That would make it easier to identify and contact bigger sites still using the old schemes.

Firefishy commented 1 year ago

tile.openstreetmap.de (note .de) is retiring their a., b. and c. aliases on the 1st of August 2023: https://community.openstreetmap.org/t/a-b-c-tile-openstreetmap-de-subdomains-von-tile-openstreetmap-de-werden-aufgehoben/100830

pnorman commented 1 year ago

I was of the opinion that this doesn't matter, but a forum thread has made me reconsider.

When browsing and fetching tiles, a browser will do a DNS lookup which points to the Fastly CNAME, which then points at the Fastly IPs, behind which are multiple servers. In a browser this process is consistent, i.e. if you open osm.org, view tiles, then close the window and open a new one, you'll get the same cache server. This holds true even if you visit a different site using the same hostname. If they completely quit the browser and re-open it, they will start getting a different cache server.

This is not true for something like curl, so for i inseq 1 10; do curl -sI https://tile.openstreetmap.org/0/0/0.png | grep x-served-by; done will return different cache servers.

When a user goes to a site using a different domain name, it uses a different cache server. This, in some conditions, can cause different content to be displayed.

pnorman commented 1 year ago

bc42abf5-864b-4b00-bb51-ea8a6a8b93b9.csv

I did a search of logs for top websites not using the tile.openstreetmap.org, results above. Below are apps

c028298d-c6a6-4ae2-9423-787e849be222.csv

grischard commented 1 year ago

Should we officially deprecate the subdomains, and make them a redirect instead? That would avoid the issue @pnorman is mentioning.

pnorman commented 1 year ago

Should we officially deprecate the subdomains, and make them a redirect instead? That would avoid the issue @pnorman is mentioning.

Our first step should be to get most of the traffic to change, as doing a redirect adds latency.

fox91 commented 1 year ago

bc42abf5-864b-4b00-bb51-ea8a6a8b93b9.csv

I did a search of logs for top websites not using the tile.openstreetmap.org, results above.

@luckyalvy (@rainviewer) if you haven't seen it yet, check out this issue.

JaffaKetchup commented 1 year ago

This spreadsheet is being used to track the CSVs and their contact status above: https://docs.google.com/spreadsheets/d/1DxlviSEW3BWbuCWNV6ka3_IQWSDUN25oBPRxw3BAmNg/edit?usp=sharing.

luckyalvy commented 1 year ago

@luckyalvy (@rainviewer) if you have yet to see it, check out this issue.

We're sorry for being late with this change. The website and embedded map now use the new https://tile.openstreetmap.org endpoint for tiles.

mtmail commented 1 year ago

In reference to pnorman's list of high traffic users (https://github.com/openstreetmap/operations/issues/737#issuecomment-1698279868) I contacted Ruptela. Got positive response from the product owner to change the subdomain(s), reduce usage and eventually move away from tile.openstreetmap.org

otbutz commented 1 year ago

Instead of simply disabling the subdomains at point X, should we aim for a brownout? e.g. disabling the DNS entries for short time periods?

Firefishy commented 1 year ago

Instead of simply disabling the subdomains at point X, should we aim for a brownout? e.g. disabling the DNS entries for short time periods?

That is an option. Better option might to serve a static tile with a URL asking them to update their usage.

otbutz commented 1 year ago

@Firefishy that would be the best approach

JaffaKetchup commented 1 year ago

@Firefishy I like the idea of just putting some text overlayed on the tiles, so the map can still be seen, but there's some text as well. Maybe only apply the text to a percentage of tiles served?

Firefishy commented 1 year ago

@Firefishy I like the idea of just putting some text overlayed on the tiles, so the map can still be seen, but there's some text as well. Maybe only apply the text to a percentage of tiles served?

Not exactly easy, needs a backend which downloads the original tile and overlays the text and then serves the file. Needs to fast. I'd appreciate some help developing something for this. Maybe rust or go?

tomhughes commented 1 year ago

Seriously, forget it - trying to overlay text would be a huge effort.

otbutz commented 1 year ago

We might as well serve the static tile as part of the brownout.

tomhughes commented 1 year ago

You realise that we are serving this data from a CDN right, and not a general purpose computer where we can apply arbitrary transformations...

This is something that needs to scale to serving tens of thousands of tiles every second - just because it's theoretically possible to write something that overlays text doesn't mean it's reasonably possible at production scale.

otbutz commented 1 year ago

Do we have an easy-to-understand page that explains the necessary adjustments or do we want to link to this issue? We would need a short and concise URL if we wanted to provide the link as part of a static tile.

grischard commented 1 year ago

@otbutz we tend to link to this issue. Serving static tiles is still years away if it happens at all.

grischard commented 1 year ago

http2 connection coalescing will make desharding academical for many users, unless they're very unlucky with fastly dns. This makes this issue low priority.

Firefishy commented 11 months ago

♥ Keep up the good work all! Let us see how many projects we can get to fix the URL they use. 🗺

danieldegroot2 commented 11 months ago

This spreadsheet is being used to track the CSVs and their contact status above: https://docs.google.com/spreadsheets/d/1DxlviSEW3BWbuCWNV6ka3_IQWSDUN25oBPRxw3BAmNg/edit?usp=sharing.

@JaffaKetchup The last one zn4.sinopticoplus.com is an instance of the Frota ("Fleet") application by M2M Solutions, which in 2022 was acquired by Sonda. (They work with Brazilian transport agencies among others.) An accessible mirror of this application can be found at Mapa Extern Olinha.

otbutz commented 11 months ago

Notified Zabbix: https://www.zabbix.com/forum/zabbix-suggestions-and-feedback/470783-zabbix-is-using-an-outdated-openstreetmap-tile-url

I've someone has an active contract with them, you could also create a ticket: https://support.zabbix.com/secure/Dashboard.jspa