cpu_count_physical fails on some platforms / potential better default

[surfaceowl]

Issue: The loky context manager relies only on the lscpu util to get physical and logical cpu for the _count_physical_cores method, and raises a ValueError of found{cpu_count_physical} physical cores < 1` when lscpu is missing or fails. This makes loky less cross-platform friendly than it could be.

Impact: On systems where lscpu is missing or fails, a ValueError is raised as the string found 0 physical cores < 1 trigging a failure when creating multiprocessing Pools on these systems. This happens both when the env var LOKY_MAX_CPU_COUNT is set and when it is missing or unset.

Example containers where it is missing include Heroku and Alpine. Docker containers your team is building are easy to fix, but some platform providers make it more difficult (cough, cough Heroku), and would be nice to reduce Docker build noise for missing libraries.

Suggested solutions for consideration:

1. replace current ValueError with a try/except of other backup methods for lscpu that are more cross-platform friendly. This could be methods like os.cpu_count(), or cat proc/cpuinfo or some other *nix utility. This may have some downsides on accuracy on logical vs. physical cores, or cores not actually available to the process I'm not aware of... and I am not fully conversant on which utils are available on what platforms.
2. _count_physical_cores should return an INT value of 1 + a warning, not a string and ValueError. If any python is run then we have at least 1 CPU, and can emit a loud warning that _count_physical_cores did not work correctly. A users multiprocessing pool would surely be slow, but it would at least run rather than fail. Picking the wrong number of cpu would slow down multiprocessing, but that may be preferred to not running at all.
3. consider fancy checking availability for a small number of utils and using what is installed e.g. lscpu vs nproc or even psutil - though I understand the project does not want psutil as a hard dep from tests.

Additional Context: Other commands exist may give sufficiently accurate CPU info and be more broadly cross-platform:

• os.cpu_count() which comes with python;
• reading cat /proc/cpuinfo and splitting strings to get siblings for number of logical CPU andcpu cores` for number of physical CPU.
• other linux utils like nproc though I don't know how many of these are out there, and given the number of different platforms out there it may not be possible to find the silver bullet.
• Finally there is also the python lib psutil.

Separately - on Heroku I was able to install lscpu but due to some backend magic by Heroku maintainers, lscpu still failed from the cli returning the error lscpu: failed to determine number of CPUs: /sys/devices/system/cpu/possible: No such file or **directory.**

In discussion with Heroku maintainers, they suggested using 'nproc': """ ...just to follow up on my colleague's response: on Linux, please use nproc (available on Heroku's base images, from GNU coreutils) to determine the number if available processor cores.

Unlike lscpu, nproc will correctly represent the number of CPU cores that are available to the current process after any cgroup restrictions are applied, which might be the case e.g. in a local Docker environment, and could be employed by Heroku in the future on shared dynos.

In contrast, lscpu will always report the number of online cores, even if a process may not have access to all of them.

Observe the following differences in output for a container restricted to CPU cores 1 and 2:

~ % docker run --rm -ti --cgroupns host --cpuset-cpus 1-2 heroku/heroku:22 bash

root@5d5abb8e0108:/# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit Byte Order: Little Endian CPU(s): 6 On-line CPU(s) list: 0-5 Vendor ID: Apple Model: 0 Thread(s) per core: 1 Core(s) per socket: 6 Socket(s): 1 …

root@5d5abb8e0108:/# nproc 2

root@5d5abb8e0108:/# nproc --all 6 """

[zsedrovitstamas]

I had an issue on Windows 11 which prevented me running kmeans. ...Could not find the number of physical cores for the following reason: [WinError 2] The system cannot find the file specified... Using the absolute path of wmic in context.py helped: C:/Windows/System32/wbem/WMIC.exe instead of wmic

[ogrisel]

Thanks for the reports. I think both problems should be tackled:

• implement some of the fallback mechanisms proposed above in case lscpu fails to detect physical cores;

• try to see if the C:/Windows/System32/wbem/WMIC.exe file exists in case shutil.which("WMIC") is False.

Those two problems should be addressed in separate pull requests to make the review independent.

Any taker?

[ogrisel]

Note: I did not know that nproc was cgroup-aware. If this is always the case, we can probably use it to simplify _cpu_count_cgroup. Still we need to be able to find about hyper-threaded cores in case not cgroups quotas are undefined or larger than the number of physical cores detected by reading /proc/cpuinfo or using lscpu.

I tried the docker commands above with an old image of ubuntu (ubuntu:18.04) and nproc already behaved that way at the time.