Closed jy-lefort closed 1 month ago
Something triggers throttling. Some sensor was calling for throttling as defined by configuration, othewise thermald doesn't do throttle. Please attach logs to check:
systemctl stop thermald sudo thermald --no-daemon --loglevel=info --adaptive Then trigger the condition and attach the generated log.
That's the result of running a parallel build. It seems to me that it throttles at 85°C:
[1676655320][INFO]Set : threshold:85000, temperature:94000, cdev:24(intel_pstate), curr_state:1, max_state:10
In that case it disables turbo (/sys/devices/system/cpu/intel_pstate/no_turbo
becomes 1) and decreases max_perf_pct
from 100% to as low as 70% (even 50% sometimes). Quite obviously, that has a catastrophic impact on performance.
I don't want thermald to do that, it didn't do that before the update. Without thermald, the CPU can get as hot as 100°C without any issues (100°C seems to be a kernel or CPU limit).
To summarize, thermald has to work out of the box (without having to tweak .xml files or downright uninstall it). It worked out of the box before.
This is because your system's thermal table has a bug, which didn't specify a target for balanced condition. Hence default limit kicked in where it calls to keep temp at 85C. So get throttled to reach that limit.
This need some workaround.
Please try to build from branch issue_388 and reproduce. Build procedure is in README.txt. https://github.com/intel/thermal_daemon/tree/issue_388
Thanks for the quick feedback. Sorry but as removing thermald works for me, I don't really have time nor inclination to start building branches and so on.
By "my system's thermal table", do you mean the default configuration of thermald as shipped by Fedora? If it's the case, I can open a bug on the Fedora bug tracker and link to this report.
If, on the other hand, you suspect it is an issue with thermald itself, maybe Intel Corporation can contract me to fix this issue? As you can see in the log, my CPU is recent (12th gen i7) and it's not very nice that to novice users, it appears twice as slow as competing AMD CPUs.
The thermal configuration is from the manufacturer of your laptop. This is nothing to do with thermald or Fedora. Intel doesn't define thermal tables.
BTW, what is the make and model of this latop?
Also please file a bug report with Fedora. Link this issue, so that they can build a version.
My laptop is a MSI Prestige 15.
Could it be a hardware issue such as a faulty temp sensor? It didn't do this initially, so I'm wondering.
Thanks for the details.
No. There is no HW issue. Probably there was some BIOS update which may have brought new tables. Thanks for reporting. I tested by simulating such table. So I will release it soon.
Thanks for the very quick feedback. I don't know much about modern BIOSes, all I know is whether I shall use a constref or a rvalue ref in C++. Out of curiousity I'll probably investigate furthermore.
I might be having the same issue (CC https://github.com/intel/thermal_daemon/issues/280). I have an LG Gram (i7-1260P, ADL), which ends up running at 400 MHz at random times, apparently unrelated to the system load or temperatures.
What I found in my investigation up to this point:
s-tui
doesn't always trigger itIA32_PACKAGE_THERM_STATUS
are offturbostat
output, bad on left, good on right:thermald
avoids the issuePKG Limit #1: ENabled (11.625 Watts, 28.000000 sec, clamp ENabled)
)thermald
output looks like this:The interesting part in there is:
[1679866345][INFO]Added zone 2 trip 2 clamp_valid 0 clamp 0 _min:0 _max:0
[1679866345][INFO]set cdev state index 18 state 11712000 wr:11712000
[1679866345][INFO]Set : threshold:90000, temperature:90050, cdev:18(rapl_controller), curr_state:11712000, max_state:125000
$ ls -1 /sys/class/thermal/
cooling_device0@
cooling_device1@
cooling_device10@
cooling_device11@
cooling_device12@
cooling_device13@
cooling_device14@
cooling_device15@
cooling_device16@
cooling_device17@
cooling_device2@
cooling_device3@
cooling_device4@
cooling_device5@
cooling_device6@
cooling_device7@
cooling_device8@
cooling_device9@
thermal_zone0@
thermal_zone1@
thermal_zone2@
thermal_zone3@
thermal_zone4@
thermal_zone5@
$ cat /sys/class/thermal/cooling_device*/cur_state
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
-1
10
thermald
keeps trying to write those increasingly negative values to cur_state
of that single device:[pid 9479] openat(AT_FDCWD, "/sys/class/powercap/intel-rapl/intel-rapl:0/energy_uj", O_RDONLY) = 8
[pid 9479] read(8, "15514942154\n", 8191) = 12
[pid 9479] close(8) = 0
[pid 9479] openat(AT_FDCWD, "/sys/class/thermal/thermal_zone2/temp", O_RDONLY) = 8
[pid 9479] read(8, "37050\n", 8191) = 6
[pid 9479] close(8) = 0
[pid 9479] newfstatat(AT_FDCWD, "/sys/class/thermal/cooling_device13/max_state", {st_mode=S_IFREG|0444, st_size=4096, ...}, 0) = 0
[pid 9479] openat(AT_FDCWD, "/sys/class/thermal/cooling_device13/max_state", O_RDONLY) = 8
[pid 9479] read(8, "0\n", 8191) = 2
[pid 9479] close(8) = 0
[pid 9479] write(1, "[1679900493][INFO]op->device:Pro"..., 44) = 44
[pid 9479] newfstatat(AT_FDCWD, "/sys/class/thermal/cooling_device13/cur_state", {st_mode=S_IFREG|0644, st_size=4096, ...}, 0) = 0
[pid 9479] openat(AT_FDCWD, "/sys/class/thermal/cooling_device13/cur_state", O_WRONLY) = 8
[pid 9479] write(8, "-459", 4) = -1 EINVAL (Invalid argument)
[pid 9479] write(1, "[1679900493][INFO]sysfs write fa"..., 83) = 83
[pid 9479] close(8) = 0
[pid 9479] write(1, "[1679900493][INFO]Set : threshol"..., 109) = 109
[pid 9479] newfstatat(AT_FDCWD, "/sys/class/thermal/cooling_device13/max_state", {st_mode=S_IFREG|0444, st_size=4096, ...}, 0) = 0
[pid 9479] openat(AT_FDCWD, "/sys/class/thermal/cooling_device13/max_state", O_RDONLY) = 8
[pid 9479] read(8, "0\n", 8191) = 2
[pid 9479] close(8) = 0
[pid 9479] poll([{fd=6, events=POLLIN}], 1, 4000) = 0 (Timeout)
acpi_mask_gpe=0x6e
to the kernel command line because it appeared to be causing an interrupt stormint3403_thermal
because it keeps complaining:[39288.463998] ACPI Error: No handler for Region [XIN1] (00000000a644a137) [UserDefinedRegion] (20221020/evregion-130)
[39288.464010] ACPI Error: Region UserDefinedRegion (ID=143) has no handler (20221020/exfldio-261)
[39288.464013] ACPI Error: Aborting method \_SB.PC00.LPCB.LGEC.SEN1._TMP due to previous error (AE_NOT_EXIST) (20221020/psparse-529)
[39288.469019] ACPI Error: No handler for Region [XIN1] (00000000a644a137) [UserDefinedRegion] (20221020/evregion-130)
[39288.469024] ACPI Error: Region UserDefinedRegion (ID=143) has no handler (20221020/exfldio-261)
[39288.469027] ACPI Error: Aborting method \_SB.PC00.LPCB.LGEC.SEN1._TMP due to previous error (AE_NOT_EXIST) (20221020/psparse-529)
[39288.469032] thermal thermal_zone6: failed to read out thermal zone (-5)
[1679902221][INFO]Set : threshold:90000, temperature:53050, cdev:13(Processor), curr_state:-9, max_state:0
[1679902225][INFO]op->device:Processor -10
[1679902225][INFO]sysfs write failed /sys/class/thermal/cooling_device13/cur_state
[1679902225][INFO]Set : threshold:90000, temperature:54050, cdev:13(Processor), curr_state:-10, max_state:0
[1679902229][INFO]op->device:Processor -11
[1679902229][INFO]sysfs write failed /sys/class/thermal/cooling_device13/cur_state
[1679902229][INFO]Set : threshold:90000, temperature:53050, cdev:13(Processor), curr_state:-11, max_state:0
[1679902233][INFO]cdev index:18 consecutive call, increment exponentially state 6041000 (min 28000000 max 125000) (8041000:1)
[1679902233][INFO]set cdev state index 18 state 6041000 wr:6041000
[1679902233][INFO]Set : threshold:90000, temperature:90050, cdev:18(rapl_controller), curr_state:6041000, max_state:125000
[1679902237][INFO]op->device:Processor -12
[snip]
[1679903687][INFO]sysfs write failed /sys/class/thermal/cooling_device13/cur_state
[1679903687][INFO]Set : threshold:90000, temperature:47050, cdev:13(Processor), curr_state:-374, max_state:0
[1679903691][INFO]op->device:Processor -375
[1679903691][INFO]sysfs write failed /sys/class/thermal/cooling_device13/cur_state
[1679903691][INFO]Set : threshold:90000, temperature:45050, cdev:13(Processor), curr_state:-375, max_state:0
[1679903695][INFO]cdev index:18 consecutive call, increment exponentially state 4041000 (min 28000000 max 125000) (8041000:2)
[1679903695][INFO]set cdev state index 18 state 4041000 wr:4041000
[1679903695][INFO]Set : threshold:90000, temperature:90050, cdev:18(rapl_controller), curr_state:4041000, max_state:125000
[1679903699][INFO]op->device:Processor -376
[1679903699][INFO]sysfs write failed /sys/class/thermal/cooling_device13/cur_state
[1679903699][INFO]Set : threshold:90000, temperature:46050, cdev:13(Processor), curr_state:-376, max_state:0
[1679903703][INFO]op->device:Processor -377
[1679903703][INFO]sysfs write failed /sys/class/thermal/cooling_device13/cur_state
There is a missing implementation in LG Gram, where it needs help from LG engineers to implement a kernel driver to talk to firmware via ACPI op region. Without that I suggest don't run thermald on this system. It will result in too many errors. That's you have to disable INT3403.
Regarding the cooling device state, I think there was a kernel regression. Please try the latest upstream kernel. max_state can't be 0 and also curr_state can't be -1 at start.
There is a missing implementation in LG Gram, where it needs help from LG engineers to implement a kernel driver to talk to firmware via ACPI op region. Without that I suggest don't run thermald on this system. It will result in too many errors. That's you have to disable INT3403.
Would it be possible to make thermald
not fail catastrophically when the ACPI info is missing?
Regarding the cooling device state, I think there was a kernel regression. Please try the latest upstream kernel. max_state can't be 0 and also curr_state can't be -1 at start.
I can't easily compile a kernel right now, but it appears that both of those are 0 at boot, on the latest Fedora kernel (6.2.9-300.fc38.x86_64
). thermald
tries to decrease cur_state
by 1 every couple of seconds.
I have been experiencing the same issue in past days/weeks. Temperatures are around 50-60 degrees Celsius, cpu frequency throttled down to 800 or 400 despite CPU utilization being 80-100%. Stopping/disabling/removing thermald (hack)-fixes the issue. Laptop: Lenovo T480. thermald --version: 2.5.6
I haven't had these thermal-throttling issues for years already and now they came back (with some update?). It started at some point in the past few weeks while being on Fedora 38. I have upgraded to Fedora 39 but the issue persisted. Now temporarily solved by removing thermald service. I am willing to provide more details for debugging if you want to look into it.
# systemctl status thermald.service
● thermald.service - Thermal Daemon Service
Loaded: loaded (/usr/lib/systemd/system/thermald.service; enabled; preset: enabled)
Drop-In: /usr/lib/systemd/system/service.d
└─10-timeout-abort.conf
Active: active (running) since Fri 2024-02-23 22:56:58 CET; 9min ago
Main PID: 9325 (thermald)
Tasks: 5 (limit: 38274)
Memory: 1.3M
CGroup: /system.slice/thermald.service
└─9325 /usr/sbin/thermald --systemd --dbus-enable --adaptive
Feb 23 22:56:57 lenovo-t480 systemd[1]: Starting thermald.service - Thermal Daemon Service...
Feb 23 22:56:57 lenovo-t480 thermald[9325]: 22 CPUID levels; family:model:stepping 0x6:8e:a (6:142:10)
Feb 23 22:56:58 lenovo-t480 thermald[9325]: 22 CPUID levels; family:model:stepping 0x6:8e:a (6:142:10)
Feb 23 22:56:58 lenovo-t480 thermald[9325]: sensor id 12 : No temp sysfs for reading raw temp
Feb 23 22:56:58 lenovo-t480 thermald[9325]: sensor id 12 : No temp sysfs for reading raw temp
Feb 23 22:56:58 lenovo-t480 thermald[9325]: sensor id 12 : No temp sysfs for reading raw temp
Feb 23 22:56:58 lenovo-t480 thermald[9325]: Polling mode is enabled: 4
Feb 23 22:56:58 lenovo-t480 systemd[1]: Started thermald.service - Thermal Daemon Service.
Hi folks, having similar issues to everyone on a Dell precision 5550 with Ubuntu 22.04, (kernel 6.9.3, thermald 2.5.6, Dell firmware 1.29.0).
Removing the int3403_thermal
module restores performance back to regular levels.
So whatever is going on, still a problem. It's a pity, because when the kernel module is removed, the performance of the laptop is excellent (45W for hours is IMHO reasonable and likely what you get on windows)
@spandruvada let me know if you would like to see a thermald log.
You can run https://github.com/intel/thermal_daemon/blob/master/test/thermal-debug-dump-ubuntu.sh and send the tar file it generates.
Please always open new issues for the last 2 issues. This issue is about an MSI/LG Gram issue. If you add to old issues, I will miss them.
I now use a Dell XPS 15 9530 (Core i7 13th gen) and thermald now seems to work as expected.
Thanks for your work.
Thanks @spandruvada ! I will do !
I'm running Fedora 37, where thermald is installed by default.
At some point in January 2023 (likely after a dnf update), my C++ projects started compiling much slower than before (about twice as slow) and the machine would be unusually cold during a build. The culprit was thermald, which was throttling the CPU for no reason.
Running "dnf remove thermald" solved the issue.
My CPU is a i7-1280P.