threaded plugin feature request: adjust maximum threads after initialization

[ajeb78]

I'd like to suggest adding a function to the threading plugin to allow the user to adjust the maximum number of threads to be used after plugin initialisation. Here's the use case:

I'm using lcms2 to add color management support to the GPL3 astrophotography processing software "Siril". Siril works in 2 ways: firstly, it can work on single images. In that case I can use all the available threads to process that one image, and I'd like to be able to let the lcms2 plugin use as many threads as there are virtual processors available. Secondly, it can work on sequences. In that case many images are processed in parallel: we may typically work with sequences of over 100 images. So the parallelisation happens at the image level, with each image using one thread. So in that case I would like to be able to throttle back the lcms2 plugin to only use 1 thread. And there are some in-between cases where the thread limit we use for the number of images processed is less than the total number of virtual CPUs (e.g. a sequence of only a few images, or where the images are very large and the available RAM becomes the limitation), so in that case the additional CPUs could be given over to lcms2 by adjusting the number of threads it can use. I guess I could do this by starting and stopping the plugin, but it would be simpler if there were a function available to adjust the number of threads it can use on the fly.

[ajeb78]

In fact I think perhaps I can, and maybe should, do most of what I want using contexts.

If I'm controlling threading at the image level, so that each image has one single thread and I'm processing multiple images in parallel each of which may wish to use lcms functions, should each of those threads have its own cmsContext?

And if that is the case, if a data structure is created by a function operating in one context can it be used by another context? (So for example if at program initialisation I do something like cmsHPROFILE profile = cmsCreateProfileTHR(context0, ...); can I later on do cmsCreateTransformTHR(context1, profile, ...);?

[mm2]

I think what you want is way more simple. You don't even need cmsContext. If all the images you want to convert are of same color space, you can share the color transform between threads.

In the main thread:

• Open profiles and create a transform
• For each image, launch a thread that calls cmsDoTransformLineStride(), use same transform handle
• Wait for threads completion with join threads or similar.
• Delete the transform

That's all. cmsDoTransformLineStride() is re-entrant. Caching happens at thread level. If color spaces are different, you could create one different transform per thread but that will take a lot more time. Re-use transforms and the profiles you open if possible.

The point of the threaded plug-in is to parallelize automatically the call to cmsDoTransforLineStride(). Is intended for yet-existing code that wants to be multi threaded with no programming. But in your case you have more control just managing the threads by yourself.

cmsContext is to completely isolate working environment. For example, one context using its own memory-management routines or another context using some plugins that the main context doesn't use.

[butcherg]

A cmsTransform can be parallelized simply by #pragma omp - ing the nested for loops that walk the image. With that, OMP can be passed the thread count you want used, so this can be adjusted in run-time. Here's how I do it in rawproc:

https://github.com/butcherg/rawproc/blob/master/src/gimage.cpp#L4818

[ajeb78]

Thanks for both your responses, I'm confident I can control the threading of lcms2 suitably now. Closing the issue.