Export to Image and Convert map to raster to use C++/OpenCL Rendering pathway

[Saijin-Naib]

Feature description. It'd be great if the Export to Image and Convert map to raster algorithms made use of the C++/OpenCL rendering pipeline of other QGIS Raster processing algorithms in order to speed up the rendering of the output.

[elpaso]

Sounds interesting, what do you have in mind?

Today's OpenCL implementation in QGIS provides acceleration for:

• Raster calculator
• 9 cells algorithms (QGIS analysis library) slope, aspect and hillshade
• hillshade renderer

Actually there have been some problems/complains about OpenCL support on windows, and I'm currently thinking at replacing the implementation with OpenGL compute shaders, but it's not really a trivial change and I'd like to try another route before giving up with OpenCL (that is: https://github.com/robertwgh/libOpenCL-loader, I'm waiting to know the licence first).

[Saijin-Naib]

I'm fairly ignorant as to the raster pipeline, even with your awesome summary above, but my thoughts are roughly this:

Rendering the canvas out to an image using Export to Image and Convert map to raster both have a resample step wherein the source raster has to be (normally) downsampled to the desired export resolution in DPI (for Export to Image) or desired spatial resolution (Convert map to raster). I am under the assumption that this resample could be done in the OpenCL pathway, like some of the other raster analyses currently are.

This could lead to a huge decrease in render time for these output products, especially depending upon the target resolution.

I know I was one of the louder voices with OpenCL problems, but after having removed the OpenCL libs from my QGIS directory a while back, I've not experienced any further issues under Windows with any of my setups.

[elpaso]

Just curious, what are your current time measurements in map exports? What dpi and output sizes are we taking about?

[alexbruy]

Have you tried latest revisions where native code was replaced with GDAL calls? Is it still slow for you?

[Saijin-Naib]

QGIS version	3.10.0-A Coruña	QGIS code revision	6ffa89eb3e
Compiled against Qt	5.11.2	Running against Qt	5.11.2
Compiled against GDAL/OGR	3.0.2	Running against GDAL/OGR	3.0.2
Compiled against GEOS	3.8.0-CAPI-1.13.1	Running against GEOS	3.8.0-CAPI-1.13.1
Compiled against SQLite	3.29.0	Running against SQLite	3.29.0
PostgreSQL Client Version	11.5	SpatiaLite Version	4.3.0
QWT Version	6.1.3	QScintilla2 Version	2.10.8
Compiled against PROJ	6.2.1	Running against PROJ	Rel. 6.2.1, November 1st, 2019
OS Version	Windows 10 (10.0)
Active python plugins	CalculateGeometry; DigitizingTools; Discovery; geocode_sqlite; GeoCoding; go2mapillary; go2streetview; LAStools; loadthemall; MemoryLayerSaver; mmqgis; nominatim; nominatim_locator_filter; PeliasGeocoding; qdraw; QuickOSM; quick_map_services; report; searchlayers; StreetView; db_manager; processing

Just curious, what are your current time measurements in map exports? What dpi and output sizes are we taking about?

Depends heavily upon export document resolution and export spatial resolution, depending upon which export path I take.

Typically, I'm working with at least 300+ DPI for document resolution, or 1m/1ft per pixel for spatial resolution. Extents aren't typically huge... For an example of where/when I've used this method, see my answer to this GIS.StackExchange question: https://gis.stackexchange.com/questions/341739/extracting-processing-raster-information-from-lyr-or-kmz-file/341745#341745

Or see these data generated using the above methods to test against the Windows Image Components for direct geotiff editing within Paint.NET (which is live thanks to Rick Brewster!) https://github.com/Saijin-Naib/WIC-GeoTIFF-Testfiles

Have you tried latest revisions where native code was replaced with GDAL calls? Is it still slow for you?

I'm not certain whether or not the version I'm testing has those commits in, I've attached my QGIS version information above.

I'll try and profile the exports if I get a chance later. I'm headed to work here shortly again.

[elpaso]

btw, the export map to raster need the QgsCanvas as a drawing device and that one does not use OpenGL (not to mention OpenCL), so no acceleration is possible (for now).

[Saijin-Naib]

@elpaso , is that also true of the Export to Image function?

Thanks for the clarification.

[elpaso]

Sorry, I was meaning "Export map to image" from "Project" menu (which basically export the map to an image raster), AFAIK there is other one in processing that uses GDAL directly and a third one that uses QgsCanvas as well (and I wouldn't be surprised if there were a 4th and a 5th one, I'm discovering a few duplicate implementations lately).

[Saijin-Naib]

Apparently, one of my comments got lost:

I queued up a job friday before leaving work which was Convert Map to Raster of three Hillshade images.

Here's the code: processing.run("qgis:rasterize", {'EXTENT':'1387425.369328592,1434975.0786909275,1125916.4076546878,1158006.1357979402 [EPSG:3630]','TILE_SIZE':256,'MAP_UNITS_PER_PIXEL':0.9,'MAKE_BACKGROUND_TRANSPARENT':True,'MAP_THEME':'','LAYER':None,'OUTPUT':'TEMPORARY_OUTPUT'})

It took 4.67hr to render out on my workstation. I7-7800x, 16gb DDR4, writing to an SSD.

67.45 square kilometers, roughly.

[elpaso]

Well, doing some math it looks like 16 seconds to draw a single tile 256*256 tile (if I didn't make any mistake), it looks really too slow to me, there must be something else going on here, like opening the result file to add the tile and saving it again.

Btw, the algorithm is implemented in Python and OpenCL is not an option here (not because of Python, but because we need the QgsMapCanvas as a drawing device and that's not Open(C/G)L capable at the moment).

In any event, bumping up the tile size should give you some performance boost.

[Saijin-Naib]

Yeah, I'm not sure what happened, but that was what the result log showed me for the rendering time. I'm not sure if the fact that three large rasters were set with different blend methods and transparencies had anything to do with the Convert map to raster export being so slow, but yeah.

Mmm, good to know on the tile size, though I'm trying to standardize on 256x256 tiles to keep things consistent with COG and web-map practices.

[lieti]

Hi, did your find any solution for this problem? I have exactly same problem, blending 2 raster layers +/- 200 km2 with 1m/px resolution and it take hours (i9900K, 64 GB RAM, SSD). Different computers, different versions of QGIS 3.x I have no idea how I can increase performance.

[Saijin-Naib]

Hi, did your find any solution for this problem? I have exactly same problem, blending 2 raster layers +/- 200 km2 with 1m/px resolution and it take hours (i9900K, 64 GB RAM, SSD). Different computers, different versions of QGIS 3.x I have no idea how I can increase performance.

No, nothing new to report yet. I've been watching the PR/Commit history and I think some changes that went into QGIS in the interim might have helped, but I've yet to try to re-run this analysis to get a benchmark.

[Saijin-Naib]

Re-ran the same analysis with the same settings.

QGIS version	3.10.1-A Coruña	QGIS code revision	ef24c526da
Compiled against Qt	5.11.2	Running against Qt	5.11.2
Compiled against GDAL/OGR	3.0.2	Running against GDAL/OGR	3.0.2
Compiled against GEOS	3.8.0-CAPI-1.13.1	Running against GEOS	3.8.0-CAPI-1.13.1
Compiled against SQLite	3.29.0	Running against SQLite	3.29.0
PostgreSQL Client Version	11.5	SpatiaLite Version	4.3.0
QWT Version	6.1.3	QScintilla2 Version	2.10.8
Compiled against PROJ	6.2.1	Running against PROJ	Rel. 6.2.1, November 1st, 2019
OS Version	Windows 10 (10.0)
Active python plugins	CalculateGeometry; MemoryLayerSaver; networks; openlayers_plugin; PluginLoadTimes; realcentroid; refFunctions; snail; SpreadsheetLayers; db_manager; MetaSearch; processing

QGIS version: 3.10.1-A Coruña
QGIS code revision: ef24c526da
Qt version: 5.11.2
GDAL version: 3.0.2
GEOS version: 3.8.0-CAPI-1.13.1 
PROJ version: Rel. 6.2.1, November 1st, 2019
Processing algorithm…
Algorithm 'Convert map to raster' starting…
Input parameters:
{ 'EXTENT' : '1385977.153795466,1436423.2942240534,1125114.1644511065,1158808.3790015215 [EPSG:6541]', 'LAYER' : None, 'MAKE_BACKGROUND_TRANSPARENT' : True, 'MAP_THEME' : '', 'MAP_UNITS_PER_PIXEL' : 0.9, 'OUTPUT' : 'TEMPORARY_OUTPUT', 'TILE_SIZE' : 256 }

Execution completed in 20161.67 seconds
Results:
{'OUTPUT': 'C:/Users/bcarlock/AppData/Local/Temp/processing_2ba7a954c1484d85802fc9ba05cae002/3fc35a242cff4b08bbe1fdc954590d2f/OUTPUT.tif'}

Loading resulting layers
Algorithm 'Convert map to raster' finished

So, 5.61hr up from 4.67hr.

[elpaso]

Did you try a nightly build? I've merged the multi-threaded version a few weeks ago.

[Saijin-Naib]

Would QGIS-OSGeo4W-3.11.0-52-Setup-x86_64.exe cee03fd4869b48ac7dd930583042b7ec

QGIS version	3.11.0-Master	QGIS code revision	5b7decb37a
Compiled against Qt	5.11.2	Running against Qt	5.11.2
Compiled against GDAL/OGR	3.1.0dev	Running against GDAL/OGR	3.1.0dev
Compiled against GEOS	3.8.0-CAPI-1.13.1	Running against GEOS	3.8.0-CAPI-1.13.1
Compiled against SQLite	3.29.0	Running against SQLite	3.29.0
PostgreSQL Client Version	11.5	SpatiaLite Version	4.3.0
QWT Version	6.1.3	QScintilla2 Version	2.10.8
Compiled against PROJ	6.3.0	Running against PROJ	Rel. 6.3.0, January 1st, 2020
OS Version	Windows 10 (10.0)	This copy of QGIS writes debugging output.
Active python plugins	CalculateGeometry; MemoryLayerSaver; networks; openlayers_plugin; PluginLoadTimes; realcentroid; refFunctions; snail; SpreadsheetLayers; db_manager; MetaSearch; processing

have the appropriate commit(s) merged already?

[Saijin-Naib]

QGIS version	3.11.0-Master	QGIS code revision	5b7decb37a
Compiled against Qt	5.11.2	Running against Qt	5.11.2
Compiled against GDAL/OGR	3.1.0dev	Running against GDAL/OGR	3.1.0dev
Compiled against GEOS	3.8.0-CAPI-1.13.1	Running against GEOS	3.8.0-CAPI-1.13.1
Compiled against SQLite	3.29.0	Running against SQLite	3.29.0
PostgreSQL Client Version	11.5	SpatiaLite Version	4.3.0
QWT Version	6.1.3	QScintilla2 Version	2.10.8
Compiled against PROJ	6.3.0	Running against PROJ	Rel. 6.3.0, January 1st, 2020
OS Version	Windows 10 (10.0)	This copy of QGIS writes debugging output.
Active python plugins	CalculateGeometry; MemoryLayerSaver; networks; openlayers_plugin; PluginLoadTimes; realcentroid; refFunctions; snail; SpreadsheetLayers; db_manager; MetaSearch; processing

QGIS version: 3.11.0-Master
QGIS code revision: 5b7decb37a
Qt version: 5.11.2
GDAL version: 3.1.0dev
GEOS version: 3.8.0-CAPI-1.13.1 
PROJ version: Rel. 6.3.0, January 1st, 2020
Processing algorithm…
Algorithm 'Convert map to raster' starting…
Input parameters:
{ 'EXTENT' : '1385977.153795466,1436423.2942240534,1125114.1644511065,1158808.3790015215 [EPSG:6541]', 'EXTENT_BUFFER' : 0, 'LAYERS' : '', 'MAKE_BACKGROUND_TRANSPARENT' : True, 'MAP_THEME' : 0, 'MAP_UNITS_PER_PIXEL' : 0.9, 'OUTPUT' : 'TEMPORARY_OUTPUT', 'TILE_SIZE' : 256 }

Execution completed in 16432.31 seconds
Results:
{'OUTPUT': 'C:/Users/bcarlock/AppData/Local/Temp/processing_3ade38d53e304581a967cb754ab8f188/10cbb9b7433c42a08bd1d9e62af5a900/OUTPUT.tif'}

Loading resulting layers
Algorithm 'Convert map to raster' finished

So, 16,432.31s (4.56hr). So, faster than the initial report by .11hr (almost 7 minutes). I'm not entirely sure if that is within standard variation for a task of this length.

[elpaso]

I would try if increasing the tile size makes any difference. The process is now parallelized by multiple threads, but the raster writing is serialized, this means that the less writes the less threads are blocked.

[Saijin-Naib]

I would try if increasing the tile size makes any difference. The process is now parallelized by multiple threads, but the raster writing is serialized, this means that the less writes the less threads are blocked.

I'll give that a shot with the just-released QGIS-OSGeo4W-3.11.0-1-Setup-x86_64.exe 68ad3c445449dd63c7d5be71209e5b87

QGIS version	3.11.0-Master	QGIS code revision	cf6cd48b38
Compiled against Qt	5.11.2	Running against Qt	5.11.2
Compiled against GDAL/OGR	3.1.0dev	Running against GDAL/OGR	3.1.0dev
Compiled against GEOS	3.8.0-CAPI-1.13.1	Running against GEOS	3.8.0-CAPI-1.13.1
Compiled against SQLite	3.29.0	Running against SQLite	3.29.0
PostgreSQL Client Version	11.5	SpatiaLite Version	4.3.0
QWT Version	6.1.3	QScintilla2 Version	2.10.8
Compiled against PROJ	7.0.0	Running against PROJ	Rel. 7.0.0, March 1st, 2020
OS Version	Windows 10 (10.0)	This copy of QGIS writes debugging output.
Active python plugins	CalculateGeometry; MemoryLayerSaver; networks; openlayers_plugin; PluginLoadTimes; realcentroid; refFunctions; snail; SpreadsheetLayers; db_manager; MetaSearch; processing

Do you expect the response to be linear, such that testing with 512x512px tiles should yield roughly half the rendering time?

Or would you prefer I test a more extreme tile size?

[elpaso]

Hard to tell: it depends. The rendering process of each layer is also multithreaded (if multithreaded rendering is enabled, so there is a chance that the speed gain in parallelizing the tiles rendering is vanished by the CPU being busy while rendering the multiple layers.

Btw, I would be interested in running some metrics on my machine if you could share a sample project (possibly not too big, something that can complete in ~15 minutes).

[Saijin-Naib]

Hard to tell: it depends. The rendering process of each layer is also multithreaded (if multithreaded rendering is enabled, so there is a chance that the speed gain in parallelizing the tiles rendering is vanished by the CPU being busy while rendering the multiple layers.

Hmm, okay. I ran the same job as before but with 512x512px tiles after leaving work last night, but it bombed at some point after I left, so no result and no log. Not sure what happened, but I suppose I'll try re-running it again tonight after a fresh reboot just prior to leaving. If it bombs again, I'm just going to have to chalk it up to a regression in this latest weekly-nightly and wait until next tuesday.

Btw, I would be interested in running some metrics on my machine if you could share a sample project (possibly not too big, something that can complete in ~15 minutes).

https://1drv.ms/u/s!AvMZEGXuAwQzsvZxD2CjnahUAQ5eWQ?e=dGWHZW

This is a clip of the above data to about 4km sq, which by napkin-math, should work out to about a 15min job or less to Export Map to Raster.

[elpaso]

Thanks, unfortunately I cannot open the rasters:

ERROR 1: Cannot open TIFF file due to missing codec.
gdalinfo failed - unable to open 'HS1_Clip.tif'.

I have libgeotiff 4.0.9 and I'm afraid 4.0.10 is required for that kind of compression:

Compression (259) SHORT (3) 1<50000>

[Saijin-Naib]

Thanks, unfortunately I cannot open the rasters: ERROR 1: Cannot open TIFF file due to missing codec. gdalinfo failed - unable to open 'HS1_Clip.tif'.

I have libgeotiff 4.0.9 and I'm afraid 4.0.10 is required for that kind of compression: Compression (259) SHORT (3) 1<50000>

Please re-download the file above. I've replaced it with a new version using DEFLATE instead of ZSTD.

Thanks!

Hmm, okay. I ran the same job as before but with 512x512px tiles after leaving work last night, but it bombed at some point after I left, so no result and no log. Not sure what happened, but I suppose I'll try re-running it again tonight after a fresh reboot just prior to leaving. If it bombs again, I'm just going to have to chalk it up to a regression in this latest weekly-nightly and wait until next tuesday.

Bombed again. I'll have to wait to re-run the full test until the next weekly-nightly is posted this coming Tuesday :\

[elpaso]

Your example project completes in ~6 seconds on my machine (AMD Ryzen 7 1700 Eight-Core Processor).

Processing algorithm…
Algorithm 'Convert map to raster' starting…
Input parameters:
{ 'EXTENT' : '1416542.1914,1423104.1512,1139568.292,1146130.2364 [EPSG:3857]', 'EXTENT_BUFFER' : 0, 'LAYERS' : '', 'MAKE_BACKGROUND_TRANSPARENT' : True, 'MAP_THEME' : None, 'MAP_UNITS_PER_PIXEL' : 0.9, 'OUTPUT' : 'TEMPORARY_OUTPUT', 'TILE_SIZE' : 256 }

Execution completed in 5.87 seconds

[Saijin-Naib]

Your example project completes in ~6 seconds on my machine (AMD Ryzen 7 1700 Eight-Core Processor). Processing algorithm… Algorithm 'Convert map to raster' starting… Input parameters: { 'EXTENT' : '1416542.1914,1423104.1512,1139568.292,1146130.2364 [EPSG:3857]', 'EXTENT_BUFFER' : 0, 'LAYERS' : '', 'MAKE_BACKGROUND_TRANSPARENT' : True, 'MAP_THEME' : None, 'MAP_UNITS_PER_PIXEL' : 0.9, 'OUTPUT' : 'TEMPORARY_OUTPUT', 'TILE_SIZE' : 256 }

Execution completed in 5.87 seconds

Ah, crap. Non-linear (exponential? quadratic?) relationship between image area and processing time. My napkin-math was very, very wrong!

Let me make one better! I'll update/replace the project and then let you know when to re-download it again.

[Saijin-Naib]

Looks like this test area (4x as large as before) should yield about 2m-4m render times, dependent heavily upon tile size (as you predicted!).

QGIS version: 3.10.1-A Coruña
QGIS code revision: ef24c526da
Qt version: 5.11.2
GDAL version: 3.0.2
GEOS version: 3.8.0-CAPI-1.13.1 
PROJ version: Rel. 6.2.1, November 1st, 2019
Processing algorithm…
Algorithm 'Convert map to raster' starting…
Input parameters:
{ 'EXTENT' : '1412968.2132032951,1426443.1308406761,1132572.3512670745,1146047.2371109612 [EPSG:3857]', 'LAYER' : None, 'MAKE_BACKGROUND_TRANSPARENT' : False, 'MAP_THEME' : '', 'MAP_UNITS_PER_PIXEL' : 0.9, 'OUTPUT' : 'TEMPORARY_OUTPUT', 'TILE_SIZE' : 512 }

Execution completed in 100.29 seconds
Results:
{'OUTPUT': 'C:/Users/username/AppData/Local/Temp/processing_0aa03dca521e443d92fd646fd91b030c/071800339c164700b0162cc23a4478d1/OUTPUT.tif'}

Loading resulting layers
Algorithm 'Convert map to raster' finished
QGIS version: 3.10.1-A Coruña
QGIS code revision: ef24c526da
Qt version: 5.11.2
GDAL version: 3.0.2
GEOS version: 3.8.0-CAPI-1.13.1 
PROJ version: Rel. 6.2.1, November 1st, 2019
Processing algorithm…
Algorithm 'Convert map to raster' starting…
Input parameters:
{ 'EXTENT' : '1412968.2132032951,1426443.1308406761,1132572.3512670745,1146047.2371109612 [EPSG:3857]', 'LAYER' : None, 'MAKE_BACKGROUND_TRANSPARENT' : False, 'MAP_THEME' : '', 'MAP_UNITS_PER_PIXEL' : 0.9, 'OUTPUT' : 'TEMPORARY_OUTPUT', 'TILE_SIZE' : 256 }

Execution completed in 243.76 seconds
Results:
{'OUTPUT': 'C:/Users/username/AppData/Local/Temp/processing_0aa03dca521e443d92fd646fd91b030c/e8510f30beb54cf18017db9dd3e4281e/OUTPUT.tif'}

Loading resulting layers
Algorithm 'Convert map to raster' finished

Grab the latest revision from above, please.