lutraconsulting / qgis-xyz-tiles

A processing algorithm to generate xyz tiles from map/layer extent.
GNU General Public License v2.0
20 stars 12 forks source link

race conditions affecting MBTilesWriter #30

Open mash-graz opened 4 years ago

mash-graz commented 4 years ago

your plugin included in QGIS 3.10 for linux seems to be plagued by race conditions when it becomes executed in multiple threads.

the issue doesn't appear in case of the DirectoryWriter, but the MBTilesWriter will hardly work at all and produce corrupt files with lots of missing tiles and other defects...

as a symptom of this particular issue, you should often see error messages like this:

ERROR 5: /tmp/test7.mbtiles: Access window out of range in RasterIO().  Requested (-569088,-368384) of size 256x256 on raster of 512x512.

the strange negative numbers are a side effect of unexpected changed values of the self._first_tile variable resp. colliding zoom factors which are running in parallel and overwrite the expected state one another...

i did try to fix the issue by replacing self._first_tile and self._zoom_ds with dictionaries indexed by the zoom factor to reduce the observable collisions. this had some positive effect, but it doesn't completely fix the issues. :(

unfortunately i do not know, how to fix this kind of thread lock requirements in case of QGIS plugins in a more adequate manner.

the versions of my affected debian linux installation:

QGIS-Version: 3.10.0-A Coruña
QGIS-Codeversion: 6c816b4204
Qt-Version: 5.11.3
GDAL-Version: 2.4.2
GEOS-Version: 3.8.0-CAPI-1.13.1 
PROJ-Version: Rel. 5.2.0, September 15th, 2018
Verarbeite Algorithmus…
Algorithmus XYZ-Kacheln erzeugen (MBTiles) startet…
Eingabeparameter:
{ 'BACKGROUND_COLOR' : QColor(0, 0, 0, 0), 'DPI' : 96, 'EXTENT' : '1716053.858612443,1721455.9642655044,5951485.136729757,5955838.399322829 [EPSG:3857]', 'METATILESIZE' : 4, 'OUTPUT_FILE' : '/tmp/test7.mbtiles', 'QUALITY' : 75, 'TILE_FORMAT' : 0, 'ZOOM_MAX' : 16, 'ZOOM_MIN' : 12 }

Using 8 CPU Threads:
Pushing all tiles at once: 11 tiles.
Ausführung nach 2.34 Sekunden abgeschlossen
Ergebnisse:
{'OUTPUT_FILE': '/tmp/test7.mbtiles'}