Parallel test runner detects more cores than machine has

[sebastianbergmann]

Problem description

I am using PHP-CS-Fixer inside a virtual machine (VMware Workstation 17 Pro) that is running Fedora 40. This virtual machine has 6 cores, as can be seen by looking at /proc/cpuinfo:

❯ cat /proc/cpuinfo
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 85
model name  : Intel(R) Core(TM) i9-10900X CPU @ 3.70GHz
stepping    : 7
microcode   : 0x5003003
cpu MHz     : 3695.997
cache size  : 19712 KB
physical id : 0
siblings    : 6
core id     : 0
cpu cores   : 6
apicid      : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level : 22
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat avx512_vnni md_clear flush_l1d arch_capabilities
bugs        : spectre_v1 spectre_v2 spec_store_bypass swapgs itlb_multihit mmio_stale_data retbleed eibrs_pbrsb gds bhi
bogomips    : 7391.99
clflush size    : 64
cache_alignment : 64
address sizes   : 45 bits physical, 48 bits virtual
power management:

processor   : 1
vendor_id   : GenuineIntel
cpu family  : 6
model       : 85
model name  : Intel(R) Core(TM) i9-10900X CPU @ 3.70GHz
stepping    : 7
microcode   : 0x5003003
cpu MHz     : 3695.997
cache size  : 19712 KB
physical id : 0
siblings    : 6
core id     : 1
cpu cores   : 6
apicid      : 1
initial apicid  : 1
fpu     : yes
fpu_exception   : yes
cpuid level : 22
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat avx512_vnni md_clear flush_l1d arch_capabilities
bugs        : spectre_v1 spectre_v2 spec_store_bypass swapgs itlb_multihit mmio_stale_data retbleed eibrs_pbrsb gds bhi
bogomips    : 7391.99
clflush size    : 64
cache_alignment : 64
address sizes   : 45 bits physical, 48 bits virtual
power management:

processor   : 2
vendor_id   : GenuineIntel
cpu family  : 6
model       : 85
model name  : Intel(R) Core(TM) i9-10900X CPU @ 3.70GHz
stepping    : 7
microcode   : 0x5003003
cpu MHz     : 3695.997
cache size  : 19712 KB
physical id : 0
siblings    : 6
core id     : 2
cpu cores   : 6
apicid      : 2
initial apicid  : 2
fpu     : yes
fpu_exception   : yes
cpuid level : 22
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat avx512_vnni md_clear flush_l1d arch_capabilities
bugs        : spectre_v1 spectre_v2 spec_store_bypass swapgs itlb_multihit mmio_stale_data retbleed eibrs_pbrsb gds bhi
bogomips    : 7391.99
clflush size    : 64
cache_alignment : 64
address sizes   : 45 bits physical, 48 bits virtual
power management:

processor   : 3
vendor_id   : GenuineIntel
cpu family  : 6
model       : 85
model name  : Intel(R) Core(TM) i9-10900X CPU @ 3.70GHz
stepping    : 7
microcode   : 0x5003003
cpu MHz     : 3695.997
cache size  : 19712 KB
physical id : 0
siblings    : 6
core id     : 3
cpu cores   : 6
apicid      : 3
initial apicid  : 3
fpu     : yes
fpu_exception   : yes
cpuid level : 22
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat avx512_vnni md_clear flush_l1d arch_capabilities
bugs        : spectre_v1 spectre_v2 spec_store_bypass swapgs itlb_multihit mmio_stale_data retbleed eibrs_pbrsb gds bhi
bogomips    : 7391.99
clflush size    : 64
cache_alignment : 64
address sizes   : 45 bits physical, 48 bits virtual
power management:

processor   : 4
vendor_id   : GenuineIntel
cpu family  : 6
model       : 85
model name  : Intel(R) Core(TM) i9-10900X CPU @ 3.70GHz
stepping    : 7
microcode   : 0x5003003
cpu MHz     : 3695.997
cache size  : 19712 KB
physical id : 0
siblings    : 6
core id     : 4
cpu cores   : 6
apicid      : 4
initial apicid  : 4
fpu     : yes
fpu_exception   : yes
cpuid level : 22
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat avx512_vnni md_clear flush_l1d arch_capabilities
bugs        : spectre_v1 spectre_v2 spec_store_bypass swapgs itlb_multihit mmio_stale_data retbleed eibrs_pbrsb gds bhi
bogomips    : 7391.99
clflush size    : 64
cache_alignment : 64
address sizes   : 45 bits physical, 48 bits virtual
power management:

processor   : 5
vendor_id   : GenuineIntel
cpu family  : 6
model       : 85
model name  : Intel(R) Core(TM) i9-10900X CPU @ 3.70GHz
stepping    : 7
microcode   : 0x5003003
cpu MHz     : 3695.997
cache size  : 19712 KB
physical id : 0
siblings    : 6
core id     : 5
cpu cores   : 6
apicid      : 5
initial apicid  : 5
fpu     : yes
fpu_exception   : yes
cpuid level : 22
wp      : yes
flags       : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat avx512_vnni md_clear flush_l1d arch_capabilities
bugs        : spectre_v1 spectre_v2 spec_store_bypass swapgs itlb_multihit mmio_stale_data retbleed eibrs_pbrsb gds bhi
bogomips    : 7391.99
clflush size    : 64
cache_alignment : 64
address sizes   : 45 bits physical, 48 bits virtual
power management:

However, PHP-CS-Fixer detects 128 cores:

❯ ./tools/php-cs-fixer fix
PHP CS Fixer 3.59.3 (064efa1) 7th Gear by Fabien Potencier, Dariusz Ruminski and contributors.
PHP runtime: 8.3.9
Running analysis on 128 cores with 10 files per process.
Parallel runner is an experimental feature and may be unstable, use it at your own risk. Feedback highly appreciated!
Loaded config default from "/usr/local/src/phpunit/.php-cs-fixer.dist.php".
Using cache file "/usr/local/src/phpunit/.php-cs-fixer.cache/10.5-dev".
 1653/1653 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%

Fixed 0 of 1653 files in 3.849 seconds, 21.05 MB memory used

In this virtual machine, the parallel test runner is siginificantly slower than the sequential runner, especially when there is nothing to fix (3.849 seconds parallel, 0.034 seconds sequential):

❯ ./tools/php-cs-fixer fix
PHP CS Fixer 3.59.3 (064efa1) 7th Gear by Fabien Potencier, Dariusz Ruminski and contributors.
PHP runtime: 8.3.9
Running analysis on 1 core sequentially.
You can enable parallel runner and speed up the analysis! Please see usage docs for more information.
Loaded config default from "/usr/local/src/phpunit/.php-cs-fixer.dist.php".
Using cache file "/usr/local/src/phpunit/.php-cs-fixer.cache/10.5-dev".
 1653/1653 [▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓] 100%

Fixed 0 of 1653 files in 0.034 seconds, 19.05 MB memory used

Minimal reproducer

Not applicable.

Fixer version

3.59.3

PHP version

8.3.9

How do you run Fixer?

PHAR

Contribution Checks

• [X] I have verified if this problem was already reported
• [X] I am familiar with "Feature or bug?"

[Wirone]

Hi @sebastianbergmann, thanks for reporting the problem. Unfortunately, most probably this is not something we can do on our side, as we utilise fidry/cpu-core-counter which does the CPU auto-detection. We can only suggest creating an issue there, or configuring Fixer with the exact ParallelConfig you would like to use.

But maybe @theofidry can suggest if there's something we can fix on our side? We currently use FinderRegistry::getDefaultLogicalFinders() with a fallback to 1 core using DummyCpuCoreFinder.

[sebastianbergmann]

I ran cpu-core-finder's bin/diagnose.php script and saw that it looks at the output from nproc --all. This prints 128 for me:

❯ nproc --all
128

However, just nproc prints the correct number (or at least the number I would expect):

❯ nproc
6

[keradus]

We blindly rely on cpu-core-finder logic, I believe this should be raised/fixed there.

[Wirone]

It's already in progress in https://github.com/theofidry/cpu-core-counter/pull/127, when this is released, we need to change the usage to utilise new API.

[keradus]

what is the part there that would change to have 6 and not 128 ?

it would change that instead: ->getCount() - 1 OR 1 if crashed we would have ->getAvailableParallelism()

but getAvailableParallelism calls getCount under the hood, right? I dont see the fix for 128 vs 6

[Wirone]

@keradus

I will add more parameters in the future to address the points mentioned above, but they will be added as default parameters hence will not break the API.

I think we just need to use new API and it will at some point detect CPUs properly 😅. @theofidry correct me if I'm wrong 🙂.

[theofidry]

I think we just need to use new API and it will at some point detect CPUs properly

Yes the incorrect number of cores detected there is a separate issue and will be addressed separately.

[theofidry]

Checking again this issue, I could reproduce it temporarily: this is due to the VMWare configuration which overcommits the resources available.

This is a bit annoying as I expected nproc to be more reliable than the CPU info... I guess I'll have to revisit that.

[sebastianbergmann]

I can no longer reproduce this:

❯ ./tools/php-cs-fixer fix
PHP CS Fixer 3.64.0 (209f776) Space Sets by Fabien Potencier, Dariusz Ruminski and contributors.
PHP runtime: 8.3.11
Running analysis on 6 cores with 10 files per process.

[Wirone]

Thanks @sebastianbergmann for self-maintenance and the feedback 🍻!