search

openstreetmap / operations

OSMF Operations Working Group issue tracking
https://operations.osmfoundation.org/
98 stars 13 forks source link

Prepare request to QGIS to better manage OSM tile demand #1019

Open Firefishy opened 6 months ago

Firefishy commented 6 months ago

QGIS's tile.openstreetmap.org tile usage continues to grow rapidly primarily due to a very small minority of users mass downloading tiles. The mass downloading of tiles is a strain on OpenStreetMap's limited resources.

At the moment QGIS's user-agent is requesting ~1300 tiles per second (24h average) with a very high portion (520 tiles per second) of these request are uncached. OpenStreetMap.org for reference is ~1630 tiles per second of which 485 tiles per second are uncached.

Many of the requests seem to be in violation of our usage policy

  1. Extremely heavy usage by a very small % of users who appear to be mass scraping tiles in violation of our usage policy.
  2. QGIS appears to insufficiently cache tiles. QGIS appears to only cache 100MB of tiles which is insufficient and likely in violation of our caching policy.
  3. While strictly not in violation of our policy, we are unable to identify if the tile request traffic is from QGIS proper or a plugin or a processing provider.

We should investigate the usage and formulate a request the QGIS to better manage the usage. QGIS are our friends, and we want to continue to allow OpenStreetMap tiles to be used in QGIS.

Firefishy commented 6 months ago

There are plugins which specifically facilitate mass tile downloading like https://github.com/nextgis/qgis_qtiles/issues/111

Firefishy commented 6 months ago

QGIS now also has the ability to mass scrape tiles via the processing tool which is a direct violation of our usage policy.

craigallan commented 6 months ago

I read this with concern. QGIS are our friends so we approach it gently. Is this not a matter where the Board should contact QGIS management and politely say that we have a problem and want to set up a channel of co-operation to fix it.

Firefishy commented 6 months ago

I read this with concern. QGIS are our friends so we approach it gently. Is this not a matter where the Board should contact QGIS management and politely say that we have a problem and want to set up a channel of co-operation to fix it.

Absolutely agree. QGIS are our friends. This ticket is to collect the details of the problem and clearly define what is the issue.

99% of the QGIS users are not the problem, it is the 1% of users who are mass downloading map tiles that are unlikely to ever be viewed.

craigallan commented 6 months ago

That was my thought. Someone is downloading crazy amounts of tiles they don't need. Nobody could ever view so many tiles. Is it an denial of service attack? But whatever it is the QGIS Board should not allow plug-ins that request crazy amounts of tiles. Anyway, once you have a plan, I'm happy to do formal emails - let me know.

Firefishy commented 6 months ago

I have clarified the opening paragraph of this ticket.

Firefishy commented 6 months ago

Is it an denial of service attack?

It is not a denial of service attack. It is a small minority of users mass downloading tiles likely for offline usage. There are much better ways to use OpenStreetMap offline (eg: raster maps like OrganicMaps).

An example might be a QGIS user attempting to download all tile.openstreetmap.org map tiles for Germany. Up to zoom level 15 it would 8GB across 1.3 million request, upto zoom 16 is 23GB 5.4 million request, upto zoom 17 is 61GB across 21 million requests, upto zoom 18 is 217GB across 86 million requests. Our tile.openstreetmap.org service also has zoom 19. We generate the higher zoom levels on-demand and which is taxing on server resources. Normally only a fraction of tiles are ever viewed.

craigallan commented 6 months ago

I'm thinking this through...

Can you clarify "very small % of users" even if it is a guesstimate. These few users are consuming 45% of our tile serving capacity and costing us a lot of money.

You're guessing that users are doing this to build a local store of all tiles at all zoom levels that reflects a QGIS project. Only if OSM data is a layer of the project will OSM tiles be added to the new local store of tiles.

And maybe they'll refresh that local store by doing the same again in a few days time. Quite possibly they are not aware of the problem that this causes to OSM.

We certainly cannot allow this to persist.

To control it from our end:

To control it from QGIS:

is that reasonable?

Firefishy commented 6 months ago

Great!

craigallan commented 6 months ago

Firefishy: Guillaume advises that you have it all under control and the Board does not need to assist. I stand down.

pnorman commented 5 months ago

assigning myself as I need to gather some data and examples

simgislab commented 5 months ago

we're ready to help on QTiles side custom user-agent is a great suggestion that will finally allow everyone to understand if QTiles is the correct suspect

matkoniecz commented 4 months ago

QGIS now also has the ability to mass scrape tiles via the processing tool which is a direct violation of our usage policy.

would it be helpful for me to file feature request that would add domain checking and fail on using this functionality with OSMF tile servers? As this by definition violates tile usage policy?