Open kuznetsoveugeneskyboxlabs opened 11 months ago
Thanks for reporting this issue.
There is a discussion about what seems to be a related problem/the same core problem of Intel CPUs with hybrid core architectures having scheduling issues resulting in poor performance, particularly on Windows 10: https://github.com/fastbuild/fastbuild/pull/995
In that thread, the conclusions so far seem to be: a) Windows 10 in particular has scheduling issues, either due to Windows 10 not correctly supporting those CPUs and/or lack of chipset driver support b) Even on Windows 11, use of all cores may not be desirable
The involvement of process priority you mention is an interesting new piece of information (thanks again for reporting this)
My thoughts in general are:
Out of interest, do you have any Windows 11 machines of this configuration, and if so, do they exhibit different/improved scheduling behavior?
A couple of other thoughts/questions I had:
I agree, the issue you have posted seems to be related to my observations. Once that PR makes it to the codebase, I will gladly retest.
- it may be time to re-evaluate setting "below normal" priority.
I imagine for this to happen there needs to be done some testing to make sure that the PC is still usable while the build is going. I believe the Fastbuild team will have to drive that change.
Out of interest, do you have any Windows 11 machines of this configuration, and if so, do they exhibit different/improved scheduling behavior?
No, not at this moment.
- Have you checked that your Windows 10 machine has up-to-date chipset drivers?
I will have to get back to you on that.
- when you remove the priority setting and you see all cores being utilized, does it actually improve overall performance (i.e. reducing compile times)?
Yes, it does! For reference, I have a small network of computers that I am doing distributed builds.
I have encountered exactly the same issue on Windows Server 2022 with all Windows Updates installed, with 13th Gen Intel Core i5-13500. I have spent entire day finding out why building does not use all my CPU cores.
It is all about that single line of code: https://github.com/fastbuild/fastbuild/blob/178c7512ed4d3594bfad22bed967a5d557fc88b9/Code/Tools/FBuild/FBuild/Main.cpp#L106
I find it strange, it cannot be configured by for example environment variable, while it is entire process setting.
Default behavior: Release configuration: Time: 2m 20.765s Debug configuration: Time: 54.938s
Priority class enforced to Normal with Process Lasso
:
Release configuration: Time: 2m 5.557s
Debug configuration: Time: 29.604s
Time difference, especially for build without long linking is significant 54 seconds versus 29 seconds.
On modern Intel processors (Tested with i7-13700K) with Windows 10, I encounter the issue where FastBuild does not utilize all of the CPU cores (only 8 out of 24 available). This is an unfavorable interaction between how the Windows scheduler works with processes with Below-Normal priority which FBuild uses. Unfortunately, that is not good enough for our use case and we would like to max out the work initiator CPU to accelerate the build process.
This is how CPU utilization looks when running on Below-Normal process priority for FastBuild, not great!
I confirmed that setting a priority of FBuild to Normal resolves this issue and all cores are being utilized in this case. Thus I have a feature that I would like to propose (and I am already working on a PR) you as a user of FastBuild will be able to set the priority of the processes being spawned using a command line.
By default, you keep a Below-Normal with an option to bump it to Normal for this kind of edge case. (Windows only)
For reference, Incredibuild also has this type of an issue but it allows configurable process priority for its workers. In case you set it to Normal then it will mitigate this problem.
I wonder if this is a noticed issue and/or if anything has been done/proposed to address it? Any input is also appreciated!