search

qgis / QGIS

QGIS is a free, open source, cross platform (lin/win/mac) geographical information system (GIS)
https://qgis.org
GNU General Public License v2.0
10.63k stars 3.01k forks source link

Export X Y Z Tiles slows non linearly with geographic size. High CPU, Extremely Low network activity. #30587

Open anakaine opened 5 years ago

anakaine commented 5 years ago

Describe the bug The larger the geographic area used when saving an xyz tile to a geotiff, the less network activity and the more processor activity occurs: Network requests become increasingly infrequent as the process progresses. Ie the longer the project is left to continue the greater the pause between network requests. Small increases in geographic extent can double the processing time. Small spikes of 80kb per minute, with the spike occurring over a 1-2 second interval. Processor usage remains very high for the duration. 50+%. Only noticable with very large areas - eg 100km x 100km. (Actual example - takes 24 hours to download a few hundred mb of tiles, then advises that 1,000,000 minutes remain. On a 4 core 16gb ram machine with 100mbit internet and no vendor throttling of tile delivery).

How to Reproduce

  1. Add XYZ Tile layer to layers list. For this example, use google. Same result with all.
  2. Right click XYZ Tile Layer --> Export --> Save As
  3. Unselect VRT, ensure you're saving a geotiff.
  4. Set Extent to Map Canvas Extent
  5. Resolution use "Horizontal and Vertical. Set to 0.75 (or 1). Force a high zoom level so we are fetching a lot of tiles.
  6. Change Profile to High Compression (DEFLATE)
  7. Click "OK"
  8. Observe frequency of network requests for tiles, and overall low processor usage,
  9. Wait 4-5 hours
  10. Observe network requests and processor usage. Network requests should be very low, processor usage should be very high.
  11. Observe that a process which can take less than an hour in other scripts can take days or more in QGIS.

QGIS and OS versions 3.6, 3.8

Additional context I have a python script that, using a bounding box can request tiles, embed geographic information, and then build the raster at the end. The script runs much, much faster than QGIS. The script first collects all tiles, and tags each one with geogrphic information using gdal. Only after all the tiles have been collected is the end geotiff built. Without knowing more it seems like QGIS is not saving all the tiles first, but rather is rebuilding the tiff after each tile, or row is downloaded. The problem seems exponential in nature. Add a bit more, takes unexpectedly long.

For comparisson at 1m res: 20km x 20km takes 0.5hr 84km x 71km takes 4.5hrs 150 x 150km takes 16,600 (sixteen thousand six hundred) hours

zenchemical commented 3 years ago

I don't know if you noticed this, but the one in 3.22 at least outputs tile sets that are out of bounds of the actual TMS tiles. For example, in google tiles it will output negative numbers, and for TMS it will output tiles like 1/1/12.png, which is out of scope for TMS since level 1 only contains 1/0/0 1/0/1 1/1/0 and 1/1/1.

It looks like it's spending a lot of time iterating and generating tiles that are out of the scope of the extent.

If anybody fixed this I would be interested.

gioman commented 3 years ago

If anybody fixed this I would be interested.

@zenchemical is not fixed, otherwise this would have been closed (but sometime something gets fixed indirectly, but your comment seems to suggest is not the case). If you are interested in fixing this you can hire a developer, see the commercial support page in QGIS site.