Closed connebeest closed 2 years ago
Hello @connebeest
I can see some threads shutting down right after startup. Thus monitoring is frozen but you can still send commands from UI.
Do you have setup some isolcpus
or CPUSETS
in your system ?
Hi @cyring,
Yes, that is correct, we are using isolcpus
, here is our GRUB cmdline:
/boot/vmlinuz-5.4.0-94-generic root=UUID=6a72aa38-0163-4252-b2c3-b2e33d3ad986 ro recovery nomodeset dis_ucode_ldr amd_iommu=on iommu=pt modprobe.blacklist=nouveau isolcpus=4-31,36-63 nohz=on nohz_full=4-31,36-63 modprobe.blacklist=acpi_cpufreq idle=halt tsc=unstable libata.fua=1 swapaccount=1 vga=normal nofb nomodeset video=vesafb:off i915.modeset=0 systemd.unified_cgroup_hierarchy=0 systemd.legacy_systemd_cgroup_controller=1
We added modprobe.blacklist=acpi_cpufreq idle=halt tsc=unstable
for testing, because we found this in the CoreFreq Q&A. This does not fix the issue though.
Also want to note that we are using isolcpus
on a different system, where CoreFreq works perfectly, though it is an older version, 1.76.1
.
However that system is Intel-based, and running CentOS 7.
Hope this helps. Happy to provide any other useful info.
Thanks!
Hi @cyring,
Yes, that is correct, we are using
isolcpus
, here is our GRUB cmdline:
/boot/vmlinuz-5.4.0-94-generic root=UUID=6a72aa38-0163-4252-b2c3-b2e33d3ad986 ro recovery nomodeset dis_ucode_ldr amd_iommu=on iommu=pt modprobe.blacklist=nouveau isolcpus=4-31,36-63 nohz=on nohz_full=4-31,36-63 modprobe.blacklist=acpi_cpufreq idle=halt tsc=unstable libata.fua=1 swapaccount=1 vga=normal nofb nomodeset video=vesafb:off i915.modeset=0 systemd.unified_cgroup_hierarchy=0 systemd.legacy_systemd_cgroup_controller=1
We added
modprobe.blacklist=acpi_cpufreq idle=halt tsc=unstable
for testing, because we found this in the CoreFreq Q&A. This does not fix the issue though.Also want to note that we are using
isolcpus
on a different system, where CoreFreq works perfectly, though it is an older version,1.76.1
. However that system is Intel-based, and running CentOS 7.Hope this helps. Happy to provide any other useful info.
Thanks!
That's unfortunately the issue #326 I have not solved yet.
Workaround is to disable from the UI the unavailable CPUs ; those which are reserved by isolation.
Press shortcut #
or go to menu HotPlug CPU
Toggle OFF each CPU concerned by isolation
As soon as the CoreFreq map will be aligned with the kernel isolation map, the monitoring will engage.
This Wiki page also provides instructions to set this use-case.
Hi @cyring
Thanks for the feedback. The strange thing is, that CoreFreq is also not reporting anything for the CPU's that are not isolated, which are 0-3. Technically 2 & 3 are isolated using cgroups, but 0 & 1 are definitely not. Also, is this an issue specific to AMD CPUs and/or Ubuntu? Because as said, I do not have this is issue on a different system:
Also want to note that we are using isolcpus on a different system, where CoreFreq works perfectly, though it is an older version, 1.76.1. However that system is Intel-based, and running CentOS 7.
Thanks
Hi @cyring
Thanks for the feedback. The strange thing is, that CoreFreq is also not reporting anything for the CPU's that are not isolated, which are 0-3. Technically 2 & 3 are isolated using cgroups, but 0 & 1 are definitely not. Also, is this an issue specific to AMD CPUs and/or Ubuntu? Because as said, I do not have this is issue on a different system:
Also want to note that we are using isolcpus on a different system, where CoreFreq works perfectly, though it is an older version, 1.76.1. However that system is Intel-based, and running CentOS 7.
Thanks
Thanks for your various testings.
I'm still processing how the Kernel is handling isolation but I feel like it structurally change the CPU management. So far I have mastered the Hot-Plugging on a basic OS usage. Hot-Plugging is probably impacted by isolation in its source code.
Still some to study ...
About evaluating your hardware, I'm providing images [ 1 ] [ 2 ] with all prerequisites for CoreFreq. Those automated live CD should help you checking if your Processors are well supported.
Duplicate of #326
Hi @cyring,
Apologies for the delayed response, and thanks for the info.
We got it working in the end, it has to do with cgroups
that are being used on the system, isolcpu
has no effect on CoreFreq.
The way we get it to work with cgroups
is by running the CoreFreq daemon and CLI in the root cgroups cpugroup
, by using these commands:
cgexec -g cpuset:/ ./corefreqd -d
cgexec -g cpuset:/ ./cofrefreq-cli
Perhaps this also helps for #326.
One thing we noticed though, is that if you run perf top
, after having started CoreFreq with the above commands, it will crash the server. Although I'm not sure if that is related to our specific situation, or more generally.
If we just run VMs on the system (which is our use-case), it does not crash.
Hi @cyring,
Apologies for the delayed response, and thanks for the info.
We got it working in the end, it has to do with
cgroups
that are being used on the system,isolcpu
has no effect on CoreFreq. The way we get it to work withcgroups
is by running the CoreFreq daemon and CLI in theroot cgroups cpugroup
, by using these commands:
cgexec -g cpuset:/ ./corefreqd -d
cgexec -g cpuset:/ ./cofrefreq-cli
Perhaps this also helps for #326.
One thing we noticed though, is that if you run
perftop
, after having started CoreFreq with the above commands, it will crash the server. Although I'm not sure if that is related to our specific situation, or more generally. If we just run VMs on the system (which is our use-case), it does not crash.
Oh thank you for the cgroup tips.
About perf top
issue, it is linked with all symbols and my driver. See issue #214
I've not solved it yet but workaround is to start perf top
before loading corefreqk.ko
CoreFreq CLI is not displaying any data, even though kernel module is loaded and daemon running.
sudo ./corefreqd -d
`CoreFreq Daemon 1.89.4 Copyright (C) 2015-2022 CYRIL INGENIERIE
Processor [AMD Ryzen Threadripper 3970X 32-Core Processor] Architecture [Zen2/Castle Peak] 64/64 CPU Online. SleepInterval(1000), SysGate(2000), 2326 tasks
`
./corefreq-cli -k
Please advise on what could be wrong and how to fix it.
Thanks.