WebAssembly / spec

WebAssembly specification, reference interpreter, and test suite.
https://webassembly.github.io/spec/
Other
3.09k stars 438 forks source link

[build] Use os.sched_getaffinity only if available #1727

Closed sideshowbarker closed 4 months ago

sideshowbarker commented 4 months ago

This change un-breaks the build in macOS and Windows environments, by only calling the Python os.sched_getaffinity() function where it’s available (which seems to be only on Linux) — and where it’s not available, using multiprocessing.cpu_count() instead.

Otherwise, the build fails in macOS and Windows environment, because the document/core/util/mathjax2katex.py program fails — and fails the build — with a _“module 'os' has no attribute 'schedgetaffinity'” message (because os.sched_getaffinity(0) is hardcoded into the program).

This patch will have no effect on CI (which runs under a Linux environment and is working fine) but should have the effect of enabling anybody in macOS and Windows environments to actually run the build successfully.

I’ve tested in my macOS 14.4 environment with Python 3.11.7, and the docs say that multiprocessing.cpu_count() is supported in all environments all the way back to Python 2.6 — so I can’t see any reason why it wouldn’t always work.

All that said, there’s a big difference between os.sched_getaffinity() and multiprocessing.cpu_count(); which is: os.sched_getaffinity() returns “the set of CPUs the calling thread of the current process is restricted to” while multiprocessing.cpu_count() simply returns “the number of CPUs in the system” — some of which may not actually be usable by the current process.

But I think for the purposes of this particular build, the difference between the two isn’t critical. I believe that what probably will happen if that part of the mathjax2katex.py program ends up being given a number of CPUs that exceeds the actually-usable number of CPUs is: that program will anyway end up spawning only the number of threads equal to the number of actually-usable CPUs — and if it tries to spawn any threads beyond that number, those threads just don’t spawn. I think.

Finally, for the record here: There is a os.cpu_count() that does apparently the same thing as multiprocessing.cpu_count() — but the docs indicate that os.cpu_count() is available only in Python 3.4+. So it seems safer to instead use multiprocessing.cpu_count() — which as noted above, is supported all the way back to Python 2.6.

rossberg commented 4 months ago

Hm, CI does not like the Python, though.

sideshowbarker commented 4 months ago

Hm, CI does not like the Python, though.

Yeah I just noticed https://github.com/WebAssembly/spec/actions/runs/7914013430/job/21602858413?pr=1727

I’ll attempt to fix it right now

sideshowbarker commented 4 months ago

OK, CI all green now