MESMER-group / mesmer

spatially-resolved ESM-specific multi-scenario initial-condition ensemble emulator
https://mesmer-emulator.readthedocs.io/en/latest/
GNU General Public License v3.0
23 stars 18 forks source link

restrict number of used numpy threads #349

Open mathause opened 10 months ago

mathause commented 10 months ago

As already mentioned by @jschwaab in #168, estimating the localization radius is resource heavy. We should restrict the number of threads used for this.

https://stackoverflow.com/q/30791550/3010700

mathause commented 10 months ago
import os
import threadpoolctl

threadpoolctl.threadpool_limits(limits=12)

# or just for one function:

with threadpoolctl.threadpool_limits(limits=os.cpu_count() // 2):
    np.linalg.eig(arr)

threadpoolctl.threadpool_info()

Interestingly I got better performance by restricting the number of threads to the number of physical cpus.