Closed jy-lefort closed 1 month ago
spandruvada
commented
1 year ago Something triggers throttling. Some sensor was calling for throttling as defined by configuration, othewise thermald doesn't do throttle. Please attach logs to check:
systemctl stop thermald sudo thermald --no-daemon --loglevel=info --adaptive Then trigger the condition and attach the generated log.
jy-lefort
commented
1 year ago That's the result of running a parallel build. It seems to me that it throttles at 85°C:
[1676655320][INFO]Set : threshold:85000, temperature:94000, cdev:24(intel_pstate), curr_state:1, max_state:10In that case it disables turbo (/sys/devices/system/cpu/intel_pstate/no_turbo becomes 1) and decreases max_perf_pct from 100% to as low as 70% (even 50% sometimes). Quite obviously, that has a catastrophic impact on performance.
I don't want thermald to do that, it didn't do that before the update. Without thermald, the CPU can get as hot as 100°C without any issues (100°C seems to be a kernel or CPU limit).
To summarize, thermald has to work out of the box (without having to tweak .xml files or downright uninstall it). It worked out of the box before.
spandruvada
commented
1 year ago This is because your system's thermal table has a bug, which didn't specify a target for balanced condition. Hence default limit kicked in where it calls to keep temp at 85C. So get throttled to reach that limit.
This need some workaround.
spandruvada
commented
1 year ago Please try to build from branch issue_388 and reproduce. Build procedure is in README.txt. https://github.com/intel/thermal_daemon/tree/issue_388
jy-lefort
commented
1 year ago Thanks for the quick feedback. Sorry but as removing thermald works for me, I don't really have time nor inclination to start building branches and so on.
By "my system's thermal table", do you mean the default configuration of thermald as shipped by Fedora? If it's the case, I can open a bug on the Fedora bug tracker and link to this report.
If, on the other hand, you suspect it is an issue with thermald itself, maybe Intel Corporation can contract me to fix this issue? As you can see in the log, my CPU is recent (12th gen i7) and it's not very nice that to novice users, it appears twice as slow as competing AMD CPUs.
spandruvada
commented
1 year ago The thermal configuration is from the manufacturer of your laptop. This is nothing to do with thermald or Fedora. Intel doesn't define thermal tables.
spandruvada
commented
1 year ago BTW, what is the make and model of this latop?
spandruvada
commented
1 year ago Also please file a bug report with Fedora. Link this issue, so that they can build a version.
jy-lefort
commented
1 year ago My laptop is a MSI Prestige 15.
Could it be a hardware issue such as a faulty temp sensor? It didn't do this initially, so I'm wondering.
Thanks for the details.
spandruvada
commented
1 year ago No. There is no HW issue. Probably there was some BIOS update which may have brought new tables. Thanks for reporting. I tested by simulating such table. So I will release it soon.
jy-lefort
commented
1 year ago Thanks for the very quick feedback. I don't know much about modern BIOSes, all I know is whether I shall use a constref or a rvalue ref in C++. Out of curiousity I'll probably investigate furthermore.
lnicola
commented
1 year ago I might be having the same issue (CC https://github.com/intel/thermal_daemon/issues/280). I have an LG Gram (i7-1260P, ADL), which ends up running at 400 MHz at random times, apparently unrelated to the system load or temperatures.
What I found in my investigation up to this point:
s-tui doesn't always trigger itIA32_PACKAGE_THERM_STATUS are offturbostat output, bad on left, good on right:
thermald avoids the issuePKG Limit #1: ENabled (11.625 Watts, 28.000000 sec, clamp ENabled))thermald output looks like this:The interesting part in there is:
[1679866345][INFO]Added zone 2 trip 2 clamp_valid 0 clamp 0 _min:0 _max:0
[1679866345][INFO]set cdev state index 18 state 11712000 wr:11712000
[1679866345][INFO]Set : threshold:90000, temperature:90050, cdev:18(rapl_controller), curr_state:11712000, max_state:125000$ ls -1 /sys/class/thermal/
cooling_device0@
cooling_device1@
cooling_device10@
cooling_device11@
cooling_device12@
cooling_device13@
cooling_device14@
cooling_device15@
cooling_device16@
cooling_device17@
cooling_device2@
cooling_device3@
cooling_device4@
cooling_device5@
cooling_device6@
cooling_device7@
cooling_device8@
cooling_device9@
thermal_zone0@
thermal_zone1@
thermal_zone2@
thermal_zone3@
thermal_zone4@
thermal_zone5@
$ cat /sys/class/thermal/cooling_device*/cur_state
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
-1
10thermald keeps trying to write those increasingly negative values to cur_state of that single device:[pid 9479] openat(AT_FDCWD, "/sys/class/powercap/intel-rapl/intel-rapl:0/energy_uj", O_RDONLY) = 8
[pid 9479] read(8, "15514942154\n", 8191) = 12
[pid 9479] close(8) = 0
[pid 9479] openat(AT_FDCWD, "/sys/class/thermal/thermal_zone2/temp", O_RDONLY) = 8
[pid 9479] read(8, "37050\n", 8191) = 6
[pid 9479] close(8) = 0
[pid 9479] newfstatat(AT_FDCWD, "/sys/class/thermal/cooling_device13/max_state", {st_mode=S_IFREG|0444, st_size=4096, ...}, 0) = 0
[pid 9479] openat(AT_FDCWD, "/sys/class/thermal/cooling_device13/max_state", O_RDONLY) = 8
[pid 9479] read(8, "0\n", 8191) = 2
[pid 9479] close(8) = 0
[pid 9479] write(1, "[1679900493][INFO]op->device:Pro"..., 44) = 44
[pid 9479] newfstatat(AT_FDCWD, "/sys/class/thermal/cooling_device13/cur_state", {st_mode=S_IFREG|0644, st_size=4096, ...}, 0) = 0
[pid 9479] openat(AT_FDCWD, "/sys/class/thermal/cooling_device13/cur_state", O_WRONLY) = 8
[pid 9479] write(8, "-459", 4) = -1 EINVAL (Invalid argument)
[pid 9479] write(1, "[1679900493][INFO]sysfs write fa"..., 83) = 83
[pid 9479] close(8) = 0
[pid 9479] write(1, "[1679900493][INFO]Set : threshol"..., 109) = 109
[pid 9479] newfstatat(AT_FDCWD, "/sys/class/thermal/cooling_device13/max_state", {st_mode=S_IFREG|0444, st_size=4096, ...}, 0) = 0
[pid 9479] openat(AT_FDCWD, "/sys/class/thermal/cooling_device13/max_state", O_RDONLY) = 8
[pid 9479] read(8, "0\n", 8191) = 2
[pid 9479] close(8) = 0
[pid 9479] poll([{fd=6, events=POLLIN}], 1, 4000) = 0 (Timeout)acpi_mask_gpe=0x6e to the kernel command line because it appeared to be causing an interrupt stormint3403_thermal because it keeps complaining:[39288.463998] ACPI Error: No handler for Region [XIN1] (00000000a644a137) [UserDefinedRegion] (20221020/evregion-130)
[39288.464010] ACPI Error: Region UserDefinedRegion (ID=143) has no handler (20221020/exfldio-261)
[39288.464013] ACPI Error: Aborting method \_SB.PC00.LPCB.LGEC.SEN1._TMP due to previous error (AE_NOT_EXIST) (20221020/psparse-529)
[39288.469019] ACPI Error: No handler for Region [XIN1] (00000000a644a137) [UserDefinedRegion] (20221020/evregion-130)
[39288.469024] ACPI Error: Region UserDefinedRegion (ID=143) has no handler (20221020/exfldio-261)
[39288.469027] ACPI Error: Aborting method \_SB.PC00.LPCB.LGEC.SEN1._TMP due to previous error (AE_NOT_EXIST) (20221020/psparse-529)
[39288.469032] thermal thermal_zone6: failed to read out thermal zone (-5)[1679902221][INFO]Set : threshold:90000, temperature:53050, cdev:13(Processor), curr_state:-9, max_state:0
[1679902225][INFO]op->device:Processor -10
[1679902225][INFO]sysfs write failed /sys/class/thermal/cooling_device13/cur_state
[1679902225][INFO]Set : threshold:90000, temperature:54050, cdev:13(Processor), curr_state:-10, max_state:0
[1679902229][INFO]op->device:Processor -11
[1679902229][INFO]sysfs write failed /sys/class/thermal/cooling_device13/cur_state
[1679902229][INFO]Set : threshold:90000, temperature:53050, cdev:13(Processor), curr_state:-11, max_state:0
[1679902233][INFO]cdev index:18 consecutive call, increment exponentially state 6041000 (min 28000000 max 125000) (8041000:1)
[1679902233][INFO]set cdev state index 18 state 6041000 wr:6041000
[1679902233][INFO]Set : threshold:90000, temperature:90050, cdev:18(rapl_controller), curr_state:6041000, max_state:125000
[1679902237][INFO]op->device:Processor -12
[snip]
[1679903687][INFO]sysfs write failed /sys/class/thermal/cooling_device13/cur_state
[1679903687][INFO]Set : threshold:90000, temperature:47050, cdev:13(Processor), curr_state:-374, max_state:0
[1679903691][INFO]op->device:Processor -375
[1679903691][INFO]sysfs write failed /sys/class/thermal/cooling_device13/cur_state
[1679903691][INFO]Set : threshold:90000, temperature:45050, cdev:13(Processor), curr_state:-375, max_state:0
[1679903695][INFO]cdev index:18 consecutive call, increment exponentially state 4041000 (min 28000000 max 125000) (8041000:2)
[1679903695][INFO]set cdev state index 18 state 4041000 wr:4041000
[1679903695][INFO]Set : threshold:90000, temperature:90050, cdev:18(rapl_controller), curr_state:4041000, max_state:125000
[1679903699][INFO]op->device:Processor -376
[1679903699][INFO]sysfs write failed /sys/class/thermal/cooling_device13/cur_state
[1679903699][INFO]Set : threshold:90000, temperature:46050, cdev:13(Processor), curr_state:-376, max_state:0
[1679903703][INFO]op->device:Processor -377
[1679903703][INFO]sysfs write failed /sys/class/thermal/cooling_device13/cur_stateThere is a missing implementation in LG Gram, where it needs help from LG engineers to implement a kernel driver to talk to firmware via ACPI op region. Without that I suggest don't run thermald on this system. It will result in too many errors. That's you have to disable INT3403.
Regarding the cooling device state, I think there was a kernel regression. Please try the latest upstream kernel. max_state can't be 0 and also curr_state can't be -1 at start.
lnicola
commented
1 year ago There is a missing implementation in LG Gram, where it needs help from LG engineers to implement a kernel driver to talk to firmware via ACPI op region. Without that I suggest don't run thermald on this system. It will result in too many errors. That's you have to disable INT3403.
Would it be possible to make thermald not fail catastrophically when the ACPI info is missing?
Regarding the cooling device state, I think there was a kernel regression. Please try the latest upstream kernel. max_state can't be 0 and also curr_state can't be -1 at start.
I can't easily compile a kernel right now, but it appears that both of those are 0 at boot, on the latest Fedora kernel (6.2.9-300.fc38.x86_64). thermald tries to decrease cur_state by 1 every couple of seconds.
scr4bble
commented
5 months ago I have been experiencing the same issue in past days/weeks. Temperatures are around 50-60 degrees Celsius, cpu frequency throttled down to 800 or 400 despite CPU utilization being 80-100%. Stopping/disabling/removing thermald (hack)-fixes the issue. Laptop: Lenovo T480. thermald --version: 2.5.6
I haven't had these thermal-throttling issues for years already and now they came back (with some update?). It started at some point in the past few weeks while being on Fedora 38. I have upgraded to Fedora 39 but the issue persisted. Now temporarily solved by removing thermald service. I am willing to provide more details for debugging if you want to look into it.
# systemctl status thermald.service
● thermald.service - Thermal Daemon Service
Loaded: loaded (/usr/lib/systemd/system/thermald.service; enabled; preset: enabled)
Drop-In: /usr/lib/systemd/system/service.d
└─10-timeout-abort.conf
Active: active (running) since Fri 2024-02-23 22:56:58 CET; 9min ago
Main PID: 9325 (thermald)
Tasks: 5 (limit: 38274)
Memory: 1.3M
CGroup: /system.slice/thermald.service
└─9325 /usr/sbin/thermald --systemd --dbus-enable --adaptive
Feb 23 22:56:57 lenovo-t480 systemd[1]: Starting thermald.service - Thermal Daemon Service...
Feb 23 22:56:57 lenovo-t480 thermald[9325]: 22 CPUID levels; family:model:stepping 0x6:8e:a (6:142:10)
Feb 23 22:56:58 lenovo-t480 thermald[9325]: 22 CPUID levels; family:model:stepping 0x6:8e:a (6:142:10)
Feb 23 22:56:58 lenovo-t480 thermald[9325]: sensor id 12 : No temp sysfs for reading raw temp
Feb 23 22:56:58 lenovo-t480 thermald[9325]: sensor id 12 : No temp sysfs for reading raw temp
Feb 23 22:56:58 lenovo-t480 thermald[9325]: sensor id 12 : No temp sysfs for reading raw temp
Feb 23 22:56:58 lenovo-t480 thermald[9325]: Polling mode is enabled: 4
Feb 23 22:56:58 lenovo-t480 systemd[1]: Started thermald.service - Thermal Daemon Service.
ejgallego
commented
2 months ago Hi folks, having similar issues to everyone on a Dell precision 5550 with Ubuntu 22.04, (kernel 6.9.3, thermald 2.5.6, Dell firmware 1.29.0).
Removing the int3403_thermal module restores performance back to regular levels.
So whatever is going on, still a problem. It's a pity, because when the kernel module is removed, the performance of the laptop is excellent (45W for hours is IMHO reasonable and likely what you get on windows)
@spandruvada let me know if you would like to see a thermald log.
spandruvada
commented
1 month ago You can run https://github.com/intel/thermal_daemon/blob/master/test/thermal-debug-dump-ubuntu.sh and send the tar file it generates.
spandruvada
commented
1 month ago Please always open new issues for the last 2 issues. This issue is about an MSI/LG Gram issue. If you add to old issues, I will miss them.
jy-lefort
commented
1 month ago I now use a Dell XPS 15 9530 (Core i7 13th gen) and thermald now seems to work as expected.
Thanks for your work.
ejgallego
commented
1 month ago Thanks @spandruvada ! I will do !
I'm running Fedora 37, where thermald is installed by default.
At some point in January 2023 (likely after a dnf update), my C++ projects started compiling much slower than before (about twice as slow) and the machine would be unusually cold during a build. The culprit was thermald, which was throttling the CPU for no reason.
Running "dnf remove thermald" solved the issue.
My CPU is a i7-1280P.