Detection of `#cores` for default parallelism settings doesn't work in cgroup

[danxmoran]

Describe the bug

I'm running Pants in k8s. The pod has 8 allocated CPUs. The host machine has 32 total cores.

I expected Pants to compute a default value of 8 for process_execution_local_parallelism and 4 for rule_threads_core. Instead it computes respective defaults of 32 and 16.

Based on python/cpython#80235 it seems this might be a fundamental problem with os.cpu_count(). There is a comment at the bottom of that issue showing a work-around that's been implemented in pylint - maybe Pants could do the same?

Pants version

abf8b681858510d18a22d04c4de5548bbcbf3cd5 (on the 2.12.x branch, set via export PANTS_SHA)

OS

Linux

[Eric-Arellano]

This is our current logic for CPU_COUNT:

https://github.com/pantsbuild/pants/blob/2de1dac4cd5aa24329789a5f9f705e3857a04d3c/src/python/pants/util/osutil.py#L16-L26

Making that more robust would definitely be appreciated in a PR!

[cburroughs]

I also tripped over this today. As the circles in the CPython ticket demonstrate there isn't an unambiguous correct answer that can be calculated from cgroups which leaves us in a bit of a pickle. (Given how old cgroups are now, this is certainly frustrating!)

• (A) Do nothing (status quo). This is consistent but almost always a bad choice the user will trip over in any sort of containerized CI environment.
• (B) Calculate based on cpu limits. This can be calculated unambiguously, but cpu limits are not always set.

• (C) Calculate based on cpu shares. While there are widely used conventions, this is not possible to calculate without ambiguity (jdk ripped theirs out). The user intend between cpu shares and limit is also ambiguous.

  I lean somewhat towards (B) as a marginal improvement on the status quo that doesn't preclude later changes.