Palmetto slower after firmware update

[skachm]

The Palmetto I built and upgrade to firmware 1.0 is running about 30% slower after the update. (The blackscholes benchmark that took 256s with factory firmware now takes 378s with firmware 1.0 for example.) I suspect the CPU is downclocking itself, but I'm not sure how to prevent that. Are there settings that I should tweak before building the firmware?

[skachm]

Update: The frequency was reduced by 1/3rd. Here's the old /proc/cpuinfo:

processor : 0 cpu : POWER8 (raw), altivec supported clock : 3000.000000MHz revision : 2.0 (pvr 004d 0200)

...

processor : 31 cpu : POWER8 (raw), altivec supported clock : 3000.000000MHz revision : 2.0 (pvr 004d 0200)

timebase : 512000000 platform : PowerNV model : palmetto machine : PowerNV palmetto firmware : OPAL v3

And here's the new /proc/cpuinfo:

processor : 0 cpu : POWER8 (raw), altivec supported clock : 2061.000000MHz revision : 2.0 (pvr 004d 0200)

...

processor : 31 cpu : POWER8 (raw), altivec supported clock : 2061.000000MHz revision : 2.0 (pvr 004d 0200)

timebase : 512000000 platform : PowerNV model : TN71-BP012
machine : PowerNV TN71-BP012
firmware : OPAL v3

[ozbenh]

On Wed, 2015-04-08 at 13:59 -0700, skachm wrote:

Update: The frequency was reduced by 1/3rd. Here's the old /proc/cpuinfo:

Two things:

• Check this isn't just Linux cpufreq, ie, change you linux governor to "performance" on all CPUs
• Check the resulting actual frequency and whether it matches what /proc/cpuinfo is displaying (ppc64_cpu --frequency)

Cheers, Ben.

processor : 0 cpu : POWER8 (raw), altivec supported clock : 3000.000000MHz revision : 2.0 (pvr 004d 0200)

...

processor : 31 cpu : POWER8 (raw), altivec supported clock : 3000.000000MHz revision : 2.0 (pvr 004d 0200)

timebase : 512000000 platform : PowerNV model : palmetto machine : PowerNV palmetto firmware : OPAL v3

And here's the new /proc/cpuinfo:

processor : 0 cpu : POWER8 (raw), altivec supported clock : 2061.000000MHz revision : 2.0 (pvr 004d 0200)

...

processor : 31 cpu : POWER8 (raw), altivec supported clock : 2061.000000MHz revision : 2.0 (pvr 004d 0200)

timebase : 512000000 platform : PowerNV model : TN71-BP012

machine : PowerNV TN71-BP012

firmware : OPAL v3

— Reply to this email directly or view it on GitHub.

[skachm]

The frequency measures at 2.067 GHz for all of the cores:

$ sudo ppc64_cpu --frequency min: 2.067 GHz (cpu 31) max: 2.067 GHz (cpu 1) avg: 2.067 GHz

I tried several options in linux power governor: performance, userspace set to max frequency, etc. without any change in the frequency.

$ sudo cpufreq-info -c 0 cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report errors and bugs to cpufreq@vger.kernel.org, please. analyzing CPU 0: driver: powernv-cpufreq CPUs which run at the same hardware frequency: 0 1 2 3 4 5 6 7 CPUs which need to have their frequency coordinated by software: 0 1 2 3 4 5 6 7 maximum transition latency: 4294.55 ms. hardware limits: 2.06 GHz - 4.32 GHz available frequency steps: 4.32 GHz, 4.29 GHz, 4.26 GHz, 4.22 GHz, 4.19 GHz, 4.16 GHz, 4.12 GHz, 4.09 GHz, 4.06 GHz, 4.02 GHz, 3.99 GHz, 3.96 GHz, 3.92 GHz, 3.89 GHz, 3.86 GHz, 3.82 GHz, 3.79 GHz, 3.76 GHz, 3.72 GHz, 3.69 GHz, 3.66 GHz, 3.62 GHz, 3.59 GHz, 3.56 GHz, 3.52 GHz, 3.49 GHz, 3.46 GHz, 3.42 GHz, 3.39 GHz, 3.36 GHz, 3.33 GHz, 3.29 GHz, 3.26 GHz, 3.23 GHz, 3.19 GHz, 3.16 GHz, 3.13 GHz, 3.09 GHz, 3.06 GHz, 3.03 GHz, 2.99 GHz, 2.96 GHz, 2.93 GHz, 2.89 GHz, 2.86 GHz, 2.83 GHz, 2.79 GHz, 2.76 GHz, 2.73 GHz, 2.69 GHz, 2.66 GHz, 2.63 GHz, 2.59 GHz, 2.56 GHz, 2.53 GHz, 2.49 GHz, 2.46 GHz, 2.43 GHz, 2.39 GHz, 2.36 GHz, 2.33 GHz, 2.29 GHz, 2.26 GHz, 2.23 GHz, 2.19 GHz, 2.16 GHz, 2.13 GHz, 2.09 GHz, 2.06 GHz available cpufreq governors: conservative, ondemand, userspace, powersave, performance current policy: frequency should be within 2.06 GHz and 4.32 GHz. The governor "userspace" may decide which speed to use within this range. current CPU frequency is 2.06 GHz (asserted by call to hardware). cpufreq stats: 4.32 GHz:39.38%, 4.29 GHz:0.00%, 4.26 GHz:0.00%, 4.22 GHz:0.00%, 4.19 GHz:0.00%, 4.16 GHz:0.00%, 4.12 GHz:0.02%, 4.09 GHz:0.01%, 4.06 GHz:0.00%, 4.02 GHz:0.01%, 3.99 GHz:0.01%, 3.96 GHz:0.01%, 3.92 GHz:0.00%, 3.89 GHz:0.01%, 3.86 GHz:0.01%, 3.82 GHz:0.01%, 3.79 GHz:0.01%, 3.76 GHz:0.00%, 3.72 GHz:0.01%, 3.69 GHz:0.02%, 3.66 GHz:0.05%, 3.62 GHz:0.00%, 3.59 GHz:0.04%, 3.56 GHz:0.01%, 3.52 GHz:0.00%, 3.49 GHz:0.00%, 3.46 GHz:0.00%, 3.42 GHz:0.00%, 3.39 GHz:0.00%, 3.36 GHz:0.01%, 3.33 GHz:0.00%, 3.29 GHz:0.01%, 3.26 GHz:0.01%, 3.23 GHz:0.01%, 3.19 GHz:0.00%, 3.16 GHz:0.01%, 3.13 GHz:0.01%, 3.09 GHz:0.01%, 3.06 GHz:0.01%, 3.03 GHz:0.00%, 2.99 GHz:0.01%, 2.96 GHz:0.00%, 2.93 GHz:0.00%, 2.89 GHz:0.00%, 2.86 GHz:0.01%, 2.83 GHz:0.00%, 2.79 GHz:0.00%, 2.76 GHz:0.00%, 2.73 GHz:0.00%, 2.69 GHz:0.00%, 2.66 GHz:0.00%, 2.63 GHz:0.00%, 2.59 GHz:0.00%, 2.56 GHz:0.00%, 2.53 GHz:0.01%, 2.49 GHz:0.00%, 2.46 GHz:0.00%, 2.43 GHz:0.00%, 2.39 GHz:0.01%, 2.36 GHz:0.00%, 2.33 GHz:0.00%, 2.29 GHz:0.00%, 2.26 GHz:0.00%, 2.23 GHz:0.00%, 2.19 GHz:0.00%, 2.16 GHz:0.00%, 2.13 GHz:0.00%, 2.09 GHz:0.00%, 2.06 GHz:60.21% (121630)

[skachm]

Also the output for performance:

$ sudo cpufreq-info -c 0 cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report errors and bugs to cpufreq@vger.kernel.org, please. analyzing CPU 0: driver: powernv-cpufreq CPUs which run at the same hardware frequency: 0 1 2 3 4 5 6 7 CPUs which need to have their frequency coordinated by software: 0 1 2 3 4 5 6 7 maximum transition latency: 4294.55 ms. hardware limits: 2.06 GHz - 4.32 GHz available frequency steps: 4.32 GHz, 4.29 GHz, 4.26 GHz, 4.22 GHz, 4.19 GHz, 4.16 GHz, 4.12 GHz, 4.09 GHz, 4.06 GHz, 4.02 GHz, 3.99 GHz, 3.96 GHz, 3.92 GHz, 3.89 GHz, 3.86 GHz, 3.82 GHz, 3.79 GHz, 3.76 GHz, 3.72 GHz, 3.69 GHz, 3.66 GHz, 3.62 GHz, 3.59 GHz, 3.56 GHz, 3.52 GHz, 3.49 GHz, 3.46 GHz, 3.42 GHz, 3.39 GHz, 3.36 GHz, 3.33 GHz, 3.29 GHz, 3.26 GHz, 3.23 GHz, 3.19 GHz, 3.16 GHz, 3.13 GHz, 3.09 GHz, 3.06 GHz, 3.03 GHz, 2.99 GHz, 2.96 GHz, 2.93 GHz, 2.89 GHz, 2.86 GHz, 2.83 GHz, 2.79 GHz, 2.76 GHz, 2.73 GHz, 2.69 GHz, 2.66 GHz, 2.63 GHz, 2.59 GHz, 2.56 GHz, 2.53 GHz, 2.49 GHz, 2.46 GHz, 2.43 GHz, 2.39 GHz, 2.36 GHz, 2.33 GHz, 2.29 GHz, 2.26 GHz, 2.23 GHz, 2.19 GHz, 2.16 GHz, 2.13 GHz, 2.09 GHz, 2.06 GHz available cpufreq governors: conservative, ondemand, userspace, powersave, performance current policy: frequency should be within 2.06 GHz and 4.32 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 2.06 GHz (asserted by call to hardware). cpufreq stats: 4.32 GHz:39.53%, 4.29 GHz:0.00%, 4.26 GHz:0.00%, 4.22 GHz:0.00%, 4.19 GHz:0.00%, 4.16 GHz:0.00%, 4.12 GHz:0.02%, 4.09 GHz:0.01%, 4.06 GHz:0.00%, 4.02 GHz:0.01%, 3.99 GHz:0.01%, 3.96 GHz:0.01%, 3.92 GHz:0.00%, 3.89 GHz:0.01%, 3.86 GHz:0.01%, 3.82 GHz:0.01%, 3.79 GHz:0.01%, 3.76 GHz:0.00%, 3.72 GHz:0.01%, 3.69 GHz:0.02%, 3.66 GHz:0.05%, 3.62 GHz:0.00%, 3.59 GHz:0.04%, 3.56 GHz:0.01%, 3.52 GHz:0.00%, 3.49 GHz:0.00%, 3.46 GHz:0.00%, 3.42 GHz:0.00%, 3.39 GHz:0.00%, 3.36 GHz:0.01%, 3.33 GHz:0.00%, 3.29 GHz:0.01%, 3.26 GHz:0.01%, 3.23 GHz:0.01%, 3.19 GHz:0.00%, 3.16 GHz:0.01%, 3.13 GHz:0.01%, 3.09 GHz:0.01%, 3.06 GHz:0.01%, 3.03 GHz:0.00%, 2.99 GHz:0.01%, 2.96 GHz:0.00%, 2.93 GHz:0.00%, 2.89 GHz:0.00%, 2.86 GHz:0.01%, 2.83 GHz:0.00%, 2.79 GHz:0.00%, 2.76 GHz:0.00%, 2.73 GHz:0.00%, 2.69 GHz:0.00%, 2.66 GHz:0.00%, 2.63 GHz:0.00%, 2.59 GHz:0.00%, 2.56 GHz:0.00%, 2.53 GHz:0.01%, 2.49 GHz:0.00%, 2.46 GHz:0.00%, 2.43 GHz:0.00%, 2.39 GHz:0.01%, 2.36 GHz:0.00%, 2.33 GHz:0.00%, 2.29 GHz:0.00%, 2.26 GHz:0.00%, 2.23 GHz:0.00%, 2.19 GHz:0.00%, 2.16 GHz:0.00%, 2.13 GHz:0.00%, 2.09 GHz:0.00%, 2.06 GHz:60.06% (121630)

(Ubuntu 14.10, Palmetto without any hardware additions or changes, op-build firmware v1.0 built natively on a Palmetto.)

[ozbenh]

On Thu, 2015-04-09 at 12:34 -0700, skachm wrote:

The frequency measures at 2.067 GHz for all of the cores:

What governor is active and how did you configure it ?

If it's running the "on demand" governor, it will go to the lowest frequency when the machine isn't under load.

You can try echo'ing "performance" in all the scaling_governor files of /sys/devices/system/cpu/cpu* and see what the result is.

Then make sure, using ppc64_cpu --frequency, that the frequency measured corresponds to what's in /proc/cpuinfo

clarity@clarity22:/sys/devices/system/cpu/cpu0/cpufreq$ sudo cpufreq-info -c 0 cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report errors and bugs to cpufreq@vger.kernel.org, please. analyzing CPU 0: driver: powernv-cpufreq CPUs which run at the same hardware frequency: 0 1 2 3 4 5 6 7 CPUs which need to have their frequency coordinated by software: 0 1 2 3 4 5 6 7 maximum transition latency: 4294.55 ms. hardware limits: 2.06 GHz - 4.32 GHz available frequency steps: 4.32 GHz, 4.29 GHz, 4.26 GHz, 4.22 GHz, 4.19 GHz, 4.16 GHz, 4.12 GHz, 4.09 GHz, 4.06 GHz, 4.02 GHz, 3.99 GHz, 3.96 GHz, 3.92 GHz, 3.89 GHz, 3.86 GHz, 3.82 GHz, 3.79 GHz, 3.76 GHz, 3.72 GHz, 3.69 GHz, 3.66 GHz, 3.62 GHz, 3.59 GHz, 3.56 GHz, 3.52 GHz, 3.49 GHz, 3.46 GHz, 3.42 GHz, 3.39 GHz, 3.36 GHz, 3.33 GHz, 3.29 GHz, 3.26 GHz, 3.23 GHz, 3.19 GHz, 3.16 GHz, 3.13 GHz, 3.09 GHz, 3.06 GHz, 3.03 GHz, 2.99 GHz, 2.96 GHz, 2.93 GHz, 2.89 GHz, 2.86 GHz, 2.83 GHz, 2.79 GHz, 2.76 GHz, 2.73 GHz, 2.69 GHz, 2.66 GHz, 2.63 GHz, 2.59 GHz, 2.56 GHz, 2.53 GHz, 2.49 GHz, 2.46 GHz, 2.43 GHz, 2.39 GHz, 2.36 GHz, 2.33 GHz, 2.29 GHz, 2.26 GHz, 2.23 GHz, 2.19 GHz, 2.16 GHz, 2.13 GHz, 2.09 GHz, 2.06 GHz available cpufreq governors: conservative, ondemand, userspace, powersave, performance current policy: frequency should be within 2.06 GHz and 4.32 GHz. The governor "userspace" may decide which speed to use within this range. current CPU frequency is 2.06 GHz (asserted by call to hardware). cpufreq stats: 4.32 GHz:39.38%, 4.29 GHz:0.00%, 4.26 GHz:0.00%, 4.22 GHz:0.00%, 4.19 GHz:0.00%, 4.16 GHz:0.00%, 4.12 GHz:0.02%, 4.09 GHz:0.01%, 4.06 GHz:0.00%, 4.02 GHz:0.01%, 3.99 GHz:0.01%, 3.96 GHz:0.01%, 3.92 GHz:0.00%, 3.89 GHz:0.01%, 3.86 GHz:0.01%, 3.82 GHz:0.01%, 3.79 GHz:0.01%, 3.76 GHz:0.00%, 3.72 GHz:0.01%, 3.69 GHz:0.02%, 3.66 GHz:0.05%, 3.62 GHz:0.00%, 3.59 GHz:0.04%, 3.56 GHz:0.01%, 3.52 GHz:0.00%, 3.49 GHz:0.00%, 3.46 GHz:0.00%, 3.42 GHz:0.00%, 3.39 GHz:0.00%, 3.36 GHz:0.01%, 3.33 GHz:0.00%, 3.29 GHz:0.01%, 3.26 GHz:0.01%, 3.23 GHz:0.01%, 3.19 GHz:0.00%, 3.16 GHz:0.01%, 3.13 GHz:0.01%, 3.09 GHz:0.01%, 3.06 GHz:0.01%, 3.03 GHz:0.00%, 2.99 GHz:0.01%, 2.96 GHz:0.00%, 2.93 GHz:0.00%, 2.89 GHz:0.00%, 2.86 GHz:0.01%, 2.83 GHz:0.00%, 2.79 GHz:0.00%, 2.76 GHz:0.00%, 2.73 GHz:0.00%, 2.69 GHz:0.00%, 2.66 GHz:0.00%, 2.63 GHz:0.00%, 2.59 GHz:0.00%, 2.56 GHz:0.00%, 2.53 GHz:0.01%, 2.49 GHz:0.00%, 2.46 GHz:0.00%, 2.43 GHz:0.00%, 2.39 GHz:0.01%, 2.36 GHz:0.00%, 2.33 GHz:0.00%, 2.29 GHz:0.00%, 2.26 GHz:0.00%, 2.23 GHz:0.00%, 2.19 GHz:0.00%, 2.16 GHz:0.00%, 2.13 GHz:0.00%, 2.09 GHz:0.00%, 2.06 GHz:60.21% (121630)

— Reply to this email directly or view it on GitHub.

[skachm]

I tried performance and userspace to 4.32 GHz, ppc64_cpu reported 2.067 GHz under both governors.

[ozbenh]

On Thu, 2015-04-09 at 16:18 -0700, skachm wrote:

I tried performance and userspace to 4.32 GHz, ppc64_cpu reported 2.067 GHz under both governors.

Ok. I observed that a while ago too but for me at least it seems fixed with the latest PNOR we built from github and the latest AMI code, are you fully up to date on your side ?

Cheers, Ben.

[williamspatrick]

On the original Palmetto code we did not have the OCC supported. The OCC ("On-Chip Controller") is a device inside the P8 module that varies the voltages and frequencies based on thermal readings, power cap settings, and Linux governor requests. The original code simply set the frequency to 3ghz, while the latest code has the OCC which can run at a large frequency range. (It looks like 4.32 to 2.067 is your processor module's range)

I believe the /proc/cpuinfo represents the nominal frequency of the module that Hostboot told skiboot/Linux via the devtree. There are three frequency points that each module is qualified at: safe, nominal, turbo. Nominal is the "label" frequency.

In this case the /proc/cpuinfo is showing 2ghz which is the safe frequency for all modules. Whenever the OCC encounters a problem it will force the processor to safe frequency to minimize potential of hardware damage. So, since I suspect your processor was running in safe from the boot time (based on this /proc/cpuinfo value), that would seem to indicate that the OCC failed to start during Hostboot. Do you see any indication of this in the early console output? You might see a message somewhere between "ISTEP 18.0" and "ISTEP 21.1" messages indicating we failed to start the OCC.

It is also possible that the OCC is failing, for some reason, during runtime.

@ozbenh mentioned seeing this before. I think when we have seen this before (OCC failing at boot) many of the cases were to outdated APSS code. The APSS is an FPGA related to the power supplies and voltage regulators (Analog Power something something). The code for this FPGA is also on github but I don't know how to update it: https://github.com/open-power/apss

@guillesilva / @sbroylesibm - Is this something you can help with? I recall there were some i2c commands you could run from the BMC to check the APSS status and code level to see if it is running properly and what the state of it was. If all of the APSS code loads that Tyan originally shipped are out of date, we'd probably recommend that they put an update and instructions on their website.

@Erich-Hauptli / @kyle-tyan - You might want to keep tabs on this issue. I suspect as we get more "adventurous" users, like @skachm, others are going to run into similar problems.

[ozbenh]

On Thu, 2015-04-09 at 20:27 -0700, Patrick Williams wrote:

On the original Palmetto code we did not have the OCC supported. The OCC ("On-Chip Controller") is a device inside the P8 module that varies the voltages and frequencies based on thermal readings, power cap settings, and Linux governor requests. The original code simply set the frequency to 3ghz, while the latest code has the OCC which can run at a large frequency range. (It looks like 4.32 to 2.067 is your processor module's range)

I believe the /proc/cpuinfo represents the nominal frequency of the module that Hostboot told skiboot/Linux via the devtree. There are three frequency points that each module is qualified at: safe, nominal, turbo. Nominal is the "label" frequency.

No, on any not-too-ancient kernel, it should display a frequency based on the current pstate (provided cpufreq is active in Linux).

In this case the /proc/cpuinfo is showing 2ghz which is the safe frequency for all modules. Whenever the OCC encounters a problem it will force the processor to safe frequency to minimize potential of hardware damage. So, since I suspect your processor was running in safe from the boot time (based on this /proc/cpuinfo value), that would seem to indicate that the OCC failed to start during Hostboot. Do you see any indication of this in the early console output? You might see a message somewhere between "ISTEP 18.0" and "ISTEP 21.1" messages indicating we failed to start the OCC.

It is also possible that the OCC is failing, for some reason, during runtime.

What we have observed in the past (but doesn't seem to be happening on my Palmetto with the latest op-build stuff) was that the OCC started up fine, gave us a pstate table, we picked the fastest state (4.3Ghz I believe), but ppc64_cpu --frequency would still measure 2Ghz.

Reading the PMSR would still return the pstate corresponding to 4.3Ghz though and the "safe mode" override bit in SCOM (whatever name it actually has) wasn't set, it was like the OCC didn't properly program the pstate frequencies under the hood.

There was no log of OCC failures from HBRT.

@ozbenh mentioned seeing this before. I think when we have seen this before (OCC failing at boot) many of the cases were to outdated APSS code. The APSS is an FPGA related to the power supplies and voltage regulators (Analog Power something something). The code for this FPGA is also on github but I don't know how to update it: https://github.com/open-power/apss

Is v5 still current ? One problem I've had is the BMC i2c bus getting stuck, in turn causing it to fail to talk to the APSS completely.

@guillesilva / @sbroylesibm - Is this something you can help with? I recall there were some i2c commands you could run from the BMC to check the APSS status and code level to see if it is running properly and what the state of it was. If all of the APSS code loads that Tyan originally shipped are out of date, we'd probably recommend that they put an update and instructions on their website.

@Erich-Hauptli / @kyle-tyan - You might want to keep tabs on this issue. I suspect as we get more "adventurous" users, like @skachm, others are going to run into similar problems.

First make sure you have BMC and PNOR completely up to date, at least from my perspective, it looks like something got fixed in the last 2 month or so.

Cheers, Ben.

[sbroylesibm]

FYI: The latest APSS level is 05, you can check it from the BMC with:

Get APSS level

/usr/local/bin/i2c-test -m 1 -b 2 -s 0x38 -rc 1 -d 0x01

Also, as a general sanity check on the APSS you can scan the IIC bus to the APSS to make sure the IIC slave address is responding with:

Scan IIC bus, APSS is on address 0x70

/usr/local/bin/i2c-test --scan -b 2

[skachm]

I first checked it out less than 2 months ago (around March 26th), but I checked out a fresh copy of the repo, built it, reflashed the firmware and the server seems to be working now.

$ sudo ppc64_cpu --frequency min: 4.334 GHz (cpu 31) max: 4.334 GHz (cpu 1) avg: 4.334 GHz

[williamspatrick]

Glad it is working for you. When you build a level for something other than testing, it is better to checkout a tag after the clone rather than HEAD. This way you get some content that has already been qualified by us. From: skachmSent: Monday, April 13, 2015 8:11 AMTo: open-power/op-buildReply To: open-power/op-buildCc: Patrick WilliamsSubject: Re: [op-build] Palmetto slower after firmware update (#132)I first checked it out less than 2 months ago (around March 26th), but I checked out a fresh copy of the repo, built it, reflashed the firmware and the server seems to be working now.

$ sudo ppc64_cpu --frequency min: 4.334 GHz (cpu 31) max: 4.334 GHz (cpu 1) avg: 4.334 GHz

—Reply to this email directly or view it on GitHub.