search

llvm / llvm-project

The LLVM Project is a collection of modular and reusable compiler and toolchain technologies.
http://llvm.org
Other
29.15k stars 12.03k forks source link

analyze-build doesn’t honor cgroup CPU limitations if launched inside a container #57050

Open dsilakov opened 2 years ago

dsilakov commented 2 years ago

When launched without debug, analyze-build creates a pool of size None:

https://github.com/llvm/llvm-project/blob/main/clang/tools/scan-build-py/lib/libscanbuild/analyze.py#L238

In this case, python creates a pool of os.cpu_count() items. Unfortunately, cpu_count() doesn’t honor cgroup limitations – this is a known cpython issue (https://github.com/python/cpython/issues/80235) and it doesn’t look like it will be fixed in the near future.

This provides significant server load if you launch analyzer inside a container on some powerful machine.

At the same time, pylint has developed its own implementation of cpu counter which is aware of cgroups: _query_cpu(), https://github.com/PyCQA/pylint/blob/main/pylint/lint/run.py#L34

It would be nice to take _query_cpu() from pylint (or provide a similar solution) and call it instead of using None in https://github.com/llvm/llvm-project/blob/main/clang/tools/scan-build-py/lib/libscanbuild/analyze.py#L238

llvmbot commented 2 years ago

@llvm/issue-subscribers-clang-static-analyzer

steakhal commented 2 years ago

I rarely use scanbuild, but I think what you suggests makes sense. Put me on the review if you plan to propose a change.

dsilakov commented 2 years ago

Ok. Though after some experiments _cpu_count() implementation in pylint turned out to be not ideal and can work incorrectly on a real hardware - see https://github.com/PyCQA/pylint/issues/7338. So currently I use its truncated variant without a hack for AWS. Will monitor how it works and create a PR if everything is fine.