Publish metatile map with direct renderer tiles and status/dirty requests?

[nrenner]

As already mentioned on #osmf-operations, as a little side project I have been working on a map to show metatile boundaries, /status info and tiles of individual render servers and to send /dirty calls per metatile and renderer.

Before sharing, I wanted to ask if there are any concerns publishing this?

Potential concerns might be:

1. /status calls from map per metatile
   • but not active by default, seem to be fast, max 4 per pan/zoom on avg. screen size
2. might make direct render server access more publicly known
   • not a concern when blocked for outside traffic (#679), but then app needs exception from blocking
3. maintenance
   • Fastly config not public, no notification about changes
     • but so far I could still sort of follow by watching operations repo
   • configuration file (config.js) for easier adoption to smaller changes (e.g. new server)
   • could also be hosted on dev server or by operations
   • repo could be under osmlab or openstreetmap GitHub organization

Requirements:

1. permission for direct render server tile requests and status/dirty calls (#679)
2. Access-Control-Expose-Headers: * tile response header
   • for "tile info" overlay to get access to all response headers like x-tilerender (restricted to CORS-safelisted headers for cross-site requests by default)

[tomhughes]

Didn't I say on IRC that I didn't think this was likely to be acceptable?

[nrenner]

You also referred to pnorman, who suggested in a direct chat to discuss in an issue.

The questions then would be what parts of it are not acceptable, how could a solution look like or what is the alternative after disabling status/dirty for tile.openstreetmap.org in #681, which suggested:

Instead, if a user understands the internals of OSMF infrastructure, they need to issue it against the relevant backends.

[tomhughes]

Well I don't think we should be encouraging users to manually dirty tiles at all, though that is obviously just my position.

[pnorman]

We shouldn't need manual dirtying on a regular basis, and I think with weekly low-zoom re-rendering we might be there.

I see no issues with CORS for the x-tilerender header. This would need to be done on the CDN.

I'm not sure about allowing direct access to the backend servers.

I'm against the use of status, because it doesn't work with the CDN and there's no guarantee the status page is in sync with the tile. If there's information there to expose, let's do it with headers.

[nrenner]

dirty

As I understand, the dirty feature was implemented to address shortcomings of the tile expiry heuristics, that haven't changed:

• As deciding which tiles are effected, is a hard problem without actually rendering everything, this is only based on heuristics.

  • How often does the main (mapnik) map get updated

Examples are:

• relation-only changes
  • Slow rendering of some tiles of Openstreetmap Standard
• way crossing metatile without nodes
  • Carto: Knick in der Pupille? [de]
  • Wie augenblicklichen Stand von osm.org abrufen? [de] - Fehmarnbelttunnel showed different states along the way
• large area - only metatiles with perimeter nodes get expired
  • Problem with map 15 - OSM Help
  • continent-sized buildings just before carto release fill inner metatiles on access, but revert doesn't expire

Weekly low zoom update helps a lot, but

• vandalism should still be updated ASAP (e.g. New York label incident)
• mappers often take pride in being the first map to show newly opened bridges and motorways, manually updating low zooms still would be a nice to have for that

Mappers did use and do miss the dirty/status feature:

• Tile status info and rerender requests not working anymore. - reddit

  • I recently noticed that an edit I made was not rendered properly on the lowest zoom level, even several weeks after the edit. This has brought back an idea I had a while ago for a simple tool where you can easily select dirty tiles and submit them for rending

  • I need this kind of tool, and I don't mind getting involved with low-level stuff.

• Problem with map 15 - OSM Help

  • because of the new tile CDN there's no easy way to request an update

• tile status - OSM Help

  • so, there is no more way to know the date/hour of a tile ? And to know if a tile is in a queue to be recalculated ?

Don't know what problems dirty requests cause; are manual, single tile updates really that harmful, compared to carto release deployments invaldating all tiles at once? Maybe the tool could prevent using dirty when a server is busy, e.g. by querying server status from Prometheus API?

Mapper feedback is one of the main goals of the tile service, so allowing manual update where automatic doesn't happen would still be nice and contribute to that goal.

[tomhughes]

Dirty has never worked for low zoom tiles as far as I know - it certainly isn't supposed to work for them so if it does then that needs to be fixed.

[pnorman]

Dirty has never worked for low zoom tiles as far as I know - it certainly isn't supposed to work for them so if it does then that needs to be fixed.

Dirty requests have always worked on low-zoom tiles, by design. Of course, they don't work when not issued against the right server, so in practice users can't use them, but that's the same as high-zoom tiles.

[tomhughes]

I'm sorry but I'm the one that setup the current render platform and my intention was always to prevent any demand rendering of low zoom tiles because of how slow it is, and dirty works by triggering a demand render.