CPU multicore optimization

[stefanocudini]

Hi I have a GIS workstation having 8 cpu cores and more RAM.

But I constrained to use the same plugin over several instances of QGis in order to obtain a parallel faster calculation of the tiles for same map...

It would be very nice to have a tile calculation for each zoom level distributed to each CPU.

[stefanocudini]

I just discovered that can be opened several instances of the plug-in the same instance of QGis, but this is still a little uncomfortable. Do you think you can create a fork of elaborations for different cpu core?

[luis-puhl]

Hello there, I'm rendering some orthoimages of roads and in my experiments rendering each layer extent (if you have many) proved to be the most effective way. I'm using the 'multi plug-in instance' approach as well.

In my opinion, one possible approach would be just to change the 'layer extent' select-box to check-boxes and fork the instance for each selected extent. The same could be done with the zoom.

This would cause a race condition on the source project and the target tiles, some of which may be rendered twice as the layers don't match perfectly on the tile grid.

Another way would be to run https://github.com/nextgis/QTiles/blob/master/tilingthread.py#L276 as a stand alone bash call so it could be remote controlled in many computers.

PS: Thanks to all the developers of this plugin, it's my first time working with GIS. PS[2]: Just leaving my impressions here, maybe someone will benefit from it.

[isghj5]

I got a hack version of multithreading working on my end by replacing lines 152:160(the for loop) in tilingthread.py with this:

        # the only issues here is that multiprocessing.dummy has no documentation... not a good sign
        from multiprocessing.dummy import Pool as ThreadPool
        threadPool = ThreadPool(4)
        threadPool.map(self.render, self.tiles)

Change "4" to however many threads you want, if you have an 8/16 core processor for instance.

You can't use the regular Pool module because it spawns N processes, and qgis complains that it cannot render images on a different process than the original? TheadPool works, but documentation is missing...

I call this a hack because it doesn't integrate with the UI fully. You can see when tiles get finished because that gets called from render(), but when the rendering job finishes it doesn't reset the interface bar, and if you hit cancel it doesn't stop the rendering job. I couldn't figure out how to fix that part and didn't care, since I could just kill the process and start over.

If only it was so easy to get it rendering on the GPU...

[antoinezambelli]

I just wanted to note that I was getting issues with mis-rendered tiles when trying to multithread this (#103 ). I've since fixed that. The cause is in render() when self.settings gets written to (most tiles were fine, it just happened to a couple dozen every few thousand tiles). Note for @isghj5 : ThreadPoolExecutor has some docs, but not the best.

All in tilingthread.py:

import threading
from concurrent.futures import ThreadPoolExecutor

In __init__() add the following at the end:

        self.max_threads = 6 # to taste.
        self.settings_dict = {'ThreadPoolExecutor-0_' + str(i): QgsMapSettings() for i in range(self.max_threads)}
        for k in self.settings_dict:
            self.settings_dict[k].setBackgroundColor(self.color)
            self.settings_dict[k].setOutputDpi(image.logicalDpiX())
            self.settings_dict[k].setOutputImageFormat(QImage.Format_ARGB32_Premultiplied)
            self.settings_dict[k].setDestinationCrs(QgsCoordinateReferenceSystem('EPSG:3857'))
            self.settings_dict[k].setOutputSize(image.size())
            self.settings_dict[k].setLayers(self.layers)
            if self.antialias:
                self.settings_dict[k].setFlag(QgsMapSettings.Antialiasing, True)
            else:
                self.settings_dict[k].setFlag(QgsMapSettings.DrawLabeling, True)

In run(), comment out the loop as per @isghj5 fix, and then call

with ThreadPoolExecutor(max_workers=self.max_threads) as threadPool:
        threadPool.map(self.render, self.tiles)

Now in render() change all occurrences of settings to settings_dict[threading.current_thread().name].

I couldn't figure out an elegant way of deep copying the original self.settings...

[sye55]

Hey first-timer, having similar issues with multicore CPU being under-utilised. Blender 3D used to use "placeholder" files for image frames for animation which was handy for crowd-contributed rendering into online storage. Each instance of the Qtiles could pre-check if a tile exists in a folder and skip to an un-rendered one? If someone wanted to update tiles later, they could remove the problem tiles from storage, and re-render them making sure it doesn't attempt to overwrite rendered tiles.