intel / S0ixSelftestTool

S0ix Selftest Tool is designed to do the initial debugging for the S2idle path CPU Package C-state and S0ix failures in Linux OS for Intel® Client platforms, it also supports the basic runtime PC10 status check.
GNU General Public License v2.0
135 stars 18 forks source link

turbostat update needed? #23

Closed mrkapqa closed 2 months ago

mrkapqa commented 4 months ago

Hello ,

i tested this on 2 lptop with option "-r on" and get error

Your system installed turbostat tool does not support this test platform. please try to get the latest version turbostat tool.

this on ubuntu 22.04 and 24.04.

could you tell me how to update properly', thanks.

qwang59 commented 3 months ago

Update turbostat tool binary to mainline version 2024.05.10

ioogithub commented 3 months ago

Update turbostat tool binary to mainline version 2024.05.10

Hi @qwang59, I can confirm this is not working as well. I have tried the script with the older version of turbostat and the new version 2024.05.10 uploaded last week.

I don't believe the issue is with the turbostat binary as it run separately fine, When I run: s0ix-selftest-tool.sh -r on

I get this error:

Turbostat log:
s0ix-selftest-tool.sh: line 274: //turbostat: No such file or directory

Your system installed turbostat tool does not support this test platform.
please try to get the latest version turbostat tool.

So it looks like the script can't find the turbostat executable.

If I run s0ix-selftest-tool.sh -s

I get this error:

The system failed to place S2idle entry command by turbostat, please check if the suspend if failed or turbostat tool version is old e.g. did you make turbostat tool executable or seperately run S2idle command: rtcwake -m freeze -s 15

I did verify that +x is indeed active for the turbostat file.

@mrkapqa were you every able to get this working?

ioogithub commented 3 months ago

I just noticed I also get two:

(standard_in) 2: syntax errors
(standard_in) 2: syntax errors

when running with the -r on arguments

Not sure if this is part of the problem.

Also I can confirm all of the dependencies mentioned here: https://github.com/intel/S0ixSelftestTool/blob/main/README.md are installed

-gawk -vim-commo -acpidump -secure boot is off

update:

when i run -s now I can get a bit further. Here is what I get:

LTR ignore for IP 0
PC10 residency is:
S0ix residency is:
(standard_in) 1: syntax error
s0ix-selftest-tool.sh line 1154: [: : integer expression expected
(standard_in) 1: syntax error
s0ix-selftest-tool.sh line 1159: [: : integer expression expected

I also see: s0ix-selftest-tool.sh: line 1135: echo: write error: Invalid argument

and I get this same integer expression expected error for 24 cycles. At the end it reports:

The PCIe bridge link power management state is:
0000:00:06.0 Link is in L0

The Link power management state of PCIe bridge 0000:00:06.0 is not expected.
which is expected to be L1.1 or L1.2, or user would run this script again.

If I run the script again I get the same result. Running with the -r on argument still shows the same error:


The CPU runtime PC10 residency when screen ON: %
The CPU runtime PC8 residency when screen ON: %

(standard_in) 2: syntax error
(standard_in) 2: syntax error

Your system installed turbostat tool does not support this test platform.

I am still guessing that the script is not parsing the data property. Perhaps there is another dependency required that is not mentioned?

ioogithub commented 3 months ago

More information:

When I look at the Turbostat output i see:

Pk%pc10
90.71

but then the script reports

Then CPU Package C-state 10 residency after S2idle is: 
S0ix residency after S2idle is:
(standard_in) 1: syntax error
(standard_in) 1: syntax error
(standard_in) 1: syntax error
(standard_in) 1: syntax error
(standard_in) 1: syntax error

Your system achieved PC8 residency: 90.71 but no PC10 residency:,noS0ix residency:

I am guessing this is the core of the error with this script, it parsed the output from turbostat wrong, it reports PC10 value from Turbostat "90.71" as PC8 and gets an error when trying to parse PC10.

Everything else it reports after this analysis is based on this wrong data.

I have tried turbostat version 2022.04.16 and 2024.05.10 and 2023.11.07 this script gives the same syntax errors for all versions.

rockowitz commented 2 months ago

Not much to add except that I too see the errors @mrkapqa reported.

mrkapqa commented 2 months ago

thank you, did copy "turbostat" from your zip to /usr/bin/ directory, now system has newer version

sudo ./s0ix-selftest-tool.sh -r on

---Check Runtime PC10 Residency during Screen ON---:

The system OS Kernel version is: Linux rich-ThinkPad-E590 6.8.0-41-generic #41-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug 2 20:41:06 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

modprobe cpufreq_stats failed Loaded 19 prior measurements Cannot load from file /var/cache/powertop/saved_parameters.powertop File will be loaded after taking minimum number of measurement(s) with battery only RAPL device for cpu 0 RAPL Using PowerCap Sysfs : Domain Mask f RAPL device for cpu 0 RAPL Using PowerCap Sysfs : Domain Mask f Devfreq not enabled glob returned GLOB_ABORTED Cannot load from file /var/cache/powertop/saved_parameters.powertop File will be loaded after taking minimum number of measurement(s) with battery only To show power estimates do 311 measurement(s) connected to battery only Leaving PowerTOP grep: /sys/kernel/debug/dri/0/i915_dmc_info: No such file or directory grep: /sys/kernel/debug/dri/0/i915_dmc_info: No such file or directory

The system will keep idle for 40 seconds then check runtime PC10 state:

grep: /sys/kernel/debug/dri/0/i915_dmc_info: No such file or directory grep: /sys/kernel/debug/dri/0/i915_dmc_info: No such file or directory cat: /sys/kernel/debug/dri/0/i915_dmc_info: No such file or directory (standard_in) 2: syntax error (standard_in) 2: syntax error

The CPU runtime PC10 residency when screen ON: % The CPU runtime PC8 residency when screen ON: 0.00%

Turbostat log: 30.004958 sec CPU%c1 CPU%c6 CPU%c7 Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc7 Pkg%pc8 Pkg%pc9 Pk%pc10 0.73 0.67 98.03 33.75 62.20 0.00 0.00 0.00 0.00 0.00 0.62 1.33 97.37 33.75 62.20 0.00 0.00 0.00 0.00 0.00 0.85 0.54 0.64 98.19 0.77 0.91 0.30 98.41 0.57 0.37 0.41 98.14 1.23

Your system installed turbostat tool does not support this test platform.
please try to get the latest version turbostat tool.

but there is still some error?

turbostat --version turbostat version 2024.05.10 - Len Brown lenb@kernel.org

qwang59 commented 2 months ago

Sorry guys, I'm aware of that the script has some issues on the recent platform with the latest version of the turbostat tool Because I'm not working on S0ix for the past half year, the s0ix script maintain is not timely, I kindly request some patience while I need to take my personal time to fix it, thanks!

driesbenoit commented 2 months ago

I had a look at the script yesterday and could track the issues. The problem is that recent versions of turbostat do not output the same variables as before. For example, GFX%Rc6 and Pkg%pc9 cannot be outputted anymore by recent turbostat versions. See: turbostat --list.

Turbostat does not complain about variables that are requested, but not available. It simply outputs the requested variables that are available for output, making it more difficult to pinpoint the issues.

The script currently uses hardcoded indices to extract the relevant values from the turbostat output. But since the output now has fewer columns, wrong values are extracted (e.g. the PC10 value is extracted as PC8) or when the hardcoded index is higher than the number of available columns a syntax error is printed in the log that the script generates. Just like @ioogithub mentioned above.

I have fixed these issues and now have it working again. I'll submit my changes in a PR later today.