Closed landmanbester closed 2 months ago
It's semi-official ... and it wasn't well received by everyone (see https://gitlab.com/aroffringa/wsclean/-/issues/175).
In the longer term I hope to get rid of environment variables entirely to avoid all this potential confusion. This implies that an explicit thread pool size needs to be set by ducc0.misc.resize_thread_pool()
whenever anything nonstandard is needed, but it might be a prize worth paying to avoid ambiguities.
DUCC0_NUM_THREADS
is what I was looking for. Thanks!
Ah sorry! I had thought you were aware of that one.
Still, for maximum future safety and minimum messing around with environment variables, I'd recommend doing
ducc0.misc.resize_thread_pool(<whatever number of threads you'd like>)
on program startup. You can also feed in the value of OMP_NUM_THREADS
there - or whatever other environment variable that controls your multithreading environment. This is a kind of flexibility I cannot provide purely from within ducc0
.
Just to clarify, is the expected behavior for ducc0
to always honour misc.thread_pool_size()
within a process? I'm asking because I'm having difficulty parallelizing multiple gridding instances spun up from separate python threads. Seems to work fine when I spin them up from separate processes though. (This could be a bug on my side, hence the need to clarify)
Just to clarify, is the expected behavior for ducc0 to always honour misc.thread_pool_size() within a process?
Within a process, yes. We need to be very careful with terminology here though ... to me a process is something that has a memory space distinct from every othe process on the machine, as opposed to threads, which always share a single memory space with other threads in that process.
I'm asking because I'm having difficulty parallelizing multiple gridding instances spun up from separate python threads.
I admit that I don't understand what this means. If you have several threads which all call ducc0.wgridder.<something>
, then all these calls will share the (single) thread pool of that Python process. The Python process only has one instance of the ducc0
library loaded, and therefore there can only be a single thread pool, no matter if you call import ducc0
in multiple threads. So if you are in such a situation, you need to be careful to specify nthreads
in all your ducc0
calls such that the sum of threads in all parallel calls is the size of the thread pool, so you don't overbook the hardware.
That said, this (calling simultaeously into ducc0
from multiple Python threads) is a scenario I haven't even tried yet. If it dosn't work as expected, we should look into it together.
To clarify: if you are on a node with n
cores, and you are running a Python process there with m
threads, you should set the ducc
thread pool size to n
and call your gridding tasks with nthreads=n//m
(if all threads are gridding in parallel).
If this doesn't work as expected (i.e. fill the node to 100%), I'd like to hear the exact symptoms.
This answers my question, thanks. I am in the scenario where I am calling simultaneously into ducc0
from multiple python threads (eg. gridding multiple imaging bands simultaneously) and I am seeing only thread_pool_size()
threads spinning up. This is easily dealt with, just needed to clarify
I noticed a while ago that
ducc
started honouring theOMP_NUM_THREADS
environment variable. Are there any others that users should be aware of? May be worth documenting this in the projectREADME
?