search

mrghg / py12box_invert

MIT License
3 stars 0 forks source link

Limit numpy number of threads #18

Open mrghg opened 3 years ago

mrghg commented 3 years ago

Currently, numpy uses all available threads, which will be inefficient when launching array jobs

lukewestern commented 3 years ago

We have this problem with the MCMC code too... There we limit the number of threads in the shell, e.g.: export OMP_NUM_THREADS=2 export OPENBLAS_NUM_THREADS=2

But it looks like a better workaround is: https://github.com/joblib/threadpoolctl

Not sure how this affects the pooling for the sensitivity calculations though, i.e. if you limit numpy to 2 threads, and have 8 calculations in parallel, will this force numpy to use only 2 threads for all jobs, or 2 for each job? It says: "Contrary to smp, threadpoolctl does not attempt to limit the size of Python multiprocessing pools (threads or processes) or set operating system-level CPU affinity constraints: threadpoolctl only interacts with native libraries via their public runtime APIs." which I think means it's OK?

mrghg commented 3 years ago

Not sure what that means! But it'd be great if you could investigate.

I think we also need to limit the MKL threads too?