pantsbuild / pants

The Pants Build System
https://www.pantsbuild.org
Apache License 2.0
3.32k stars 636 forks source link

consider taking into accout core "type" for cpu core hueristics #21518

Open cburroughs opened 2 weeks ago

cburroughs commented 2 weeks ago

Is your feature request related to a problem? Please describe.

Circa 2024 a not uncommon workstation is an Apple M3 with a mix of "performance" and "efficiency" cores. (Some arm64 and x86_64 cpus also have a similar split.)

The various options that have cpu core based heuristics (ex: process_execution_local_parallelism) would may be better served by being based only on performance

Describe the solution you'd like Pants auto-magically makes great choices.

Describe alternatives you've considered Encouraging manual setting of the above settings in cases where the number of "total cores" seems high relative to DRAM.

Additional context See also #15840 for another reasonable heuristic adjustment.

sureshjoshi commented 2 weeks ago

Just a thought on this...

It's not a bad idea in general, but I'm always hesitant to try to outsmart the OS's scheduler. Any OS. In the past when I've done this myself, or used tooling that did, inevitably something in the OS changes and we're now stuck with a maintenance burden - or we've become a detriment someone's workflow.

There is probably some prior art here, in Dockerland maybe, or other projects which are multi-core and CPU intensive.

There might also be the future burden of over-customization requests (e.g. I want these cores for this goal only under these conditions). We have this to a small degree today, and if we start pinning cores 🤷🏽