make processing of multiple thematic_objects for multiple relevant difference asynchronous?

[dieuska]

When executing aligner.process() for a series of relevant_distances, a for-loop is executed to loop over all thematic objects; and to loop over all relevant distances. These calculations are indipendent form eachother, so maybe it is an option to do these processin asynchronous, so it results in a faster processing, when doing the calculation for a lot of thematicobjects and relevant distances?

[dieuska]

@roefem , what do you think about this one? Could that be an option? pitfalls? It could result in a big performance gain of calculations for 60 distances are started at once instead of executing them one by one? https://github.com/OnroerendErfgoed/brdr/blob/0c588f46b21c400670390d53a143da0d93b263a4/brdr/aligner.py#L328-L343

[maarten-vermeyen]

My 2c:

I think the way to go here would be relying on shapely's ufunc interfaces to the geos library, which already allow for multithreading/processing outside of python. I'm not sure the current code could be easily refactored in this way though. In essence, this would require us to rewrite the algorithm to do all geometric operations on the arrays of thematic objects instead of using a for loop and processing each item individually.

# this will execute sequentially
for geom in geoms:
   shapely.buffer(geom, 10)

# this will execute in parallel
shapely.buffer(geoms, 10)

Alternatively, we could implementing your own form of multithreading or multiprocessing is, but whether this would be beneficial would depend on the actual CPU usage. Assuming most of our users would be using cPython (and thus be subject to the GIL), you would need to use multiprocessing to leverage more cpu cores. This would create quite a bit of complexity to the code, so the performance gain has to be quite big in order to offset the added complexity to the code base and the performance penalty multiprocessing would bring for single distance use cases. I also doubt the result would be faster than an approach based on the approach outlined above. But the proof of the pudding is in the eating...

[maarten-vermeyen]

We should also consider the possible impact on memory usage, as this will go up as well.

[dieuska]

With Threadpoolexecutor: duration: [28.681322, 14.334717, 12.512052, 13.467091, 30.641785, 32.707651, 12.23049, 13.854096, 14.236291] Min: 12.23049 Max: 32.707651 Mean: 19.185055 Median: 14.236291 Stdv: 8.705756538000847

without Threadpoolexecutor:

duration: [21.313991, 16.558168, 16.590126, 18.111118, 16.872433, 17.928071, 18.32295, 17.87116, 19.516652, 16.729241]

Min: 16.558168

Max: 21.313991

Mean: 17.981391

Median: 17.8996155

Stdv: 1.504459449440969