search

MathOnco / valis

Virtual Alignment of pathoLogy Image Series
https://valis.readthedocs.io/en/latest/
MIT License
122 stars 29 forks source link

Enhancement: respecting cgroup cpu limits #114

Closed p-smirnov closed 4 months ago

p-smirnov commented 7 months ago

Currently, the code uses multiprocessing.cpu_count() to determine the number of CPUs to run parallel processes on. This doesn't play very well with "shared" environments such as K8s or HPC clusters, which use cgroups to limit resources assigned to processes.

Would it be possible to determine the number of available cores using os.sched_getaffinity(0) when this is available (on Linux platforms, presumably)? I will submit a pull request with a proposed change implementing this for review.

This still wouldn't control the number of CPUs used by Torch, but torch should obey the OMP_NUM_THREADS and MKL_NUM_THREADS environmental variables.

cdgatenbee commented 4 months ago

Hi @p-smirnov, Just wanted to let you know that I've merged your pull request into the latest version of valis (1.1.0). Thanks again!

Best, -Chandler