search

ROCm / ROC-smi

ROC System Management Interface
https://github.com/RadeonOpenCompute/ROC-smi/blob/master/README.md
179 stars 55 forks source link

Fan % seems incorrect for GPU2 #68

Closed Delaunay closed 4 years ago

Delaunay commented 5 years ago

On a system with 3 MI cards (i.e fanless/passively cooled). I get the following output. I expect the 6.67% fan to be 0.00%.

========================ROCm System Management Interface========================
================================================================================
GPU  Temp   AvgPwr  SCLK     MCLK    Fan    Perf  PwrCap  SCLK OD  MCLK OD  GPU%
1    38.0c  33.0W   1600Mhz  801Mhz  0.0%   auto  225.0W  0%       0%       18%
2    37.0c  35.0W   1600Mhz  801Mhz  6.67%  auto  225.0W  0%       0%       24%
3    38.0c  36.0W   1600Mhz  801Mhz  0.0%   auto  225.0W  0%       0%       28%
================================================================================
==============================End of ROCm SMI Log ==============================

The issue is also present with RadeonVII cards. When I manually set the the fans to 100%, rocm-smi says the fan is only running at 10.98%.

> rocm-smi --setfan 255
========================ROCm System Management Interface========================
GPU[1]          : Successfully set fan control to 'manual'
GPU[1]          : Successfully set fan speed to Level 255
GPU[2]          : Successfully set fan control to 'manual'
GPU[2]          : Successfully set fan speed to Level 255
==============================End of ROCm SMI Log ==============================
> rocm-smi 
========================ROCm System Management Interface========================
================================================================================
GPU  Temp   AvgPwr  SCLK    MCLK    Fan     Perf  PwrCap  SCLK OD  MCLK OD  GPU%  
1    34.0c  22.0W   809Mhz  351Mhz  100.0%  auto  250.0W  0%       0%       0%    
2    33.0c  20.0W   809Mhz  351Mhz  10.98%  auto  250.0W  0%       0%       0%    
================================================================================
==============================End of ROCm SMI Log ==============================

Additionally, you can find the raw values from sysfs below. MI machine:

> cat /sys/class/hwmon/hwmon1/pwm1
17
> cat /sys/class/hwmon/hwmon0/pwm1
0
> cat /sys/class/hwmon/hwmon2/pwm1
0

Radeon VII machine:

> cat /sys/class/hwmon/hwmon1/pwm1
255
> cat /sys/class/hwmon/hwmon2/pwm1
73
kentrussell commented 5 years ago

The issue actually lies in the VBIOS. For cards that have passive fans or use server coolers, some VBIOSes actually report that there is an air-cooled fan and reports those garbage numbers which are just whatever those registers have in them. The fan speed/control is only valid on cards that have an air-cooling fan on the card itself that is attached to the card, with the fan controller on the card as well. Otherwise, there is just junk data in there (and sysfs can't control coolers that are not connected to the card itself). I've found that our internal VBIOSes tend to say that there is air-cooling when there is actually none, so some of our partners may have fallen into that same situation.

To clarify, the cards are all passively cooled, right? If so, then the fan control and fan speed reporting won't be useful, unfortunately.

Delaunay commented 5 years ago

No, only the MI are passively cooled. The other values for the RadeonVII used the standard market cards with the fans.

kentrussell commented 5 years ago

Oh ok, sorry that I misunderstood there. So we've got 2 parts. At least #1 I explained reasonably well.

The other config both cards are Radeon VII's, and the 2nd one is just reporting the wrong fan. Is it actually cycling up to 100% (e.g. the fan gets super loud), or does it stay down? Is it limited to that card, or does it occur on the other card as well (and if so, is it intermittent, or only when in a certain PCIe bus).

What if you try to specify device #2 with "rocm-smi -d 2 --setfan 100%" , does it still end up doing the same thing where it accepts the value and does nothing? If the VBIOS has its own fan tuning (Sapphire does this), then it can sometimes override the kernel's fan settings, but I want to try to debug it a bit first to ensure that we cover all of the standard avenues before just assuming that it's another case like that.

Delaunay commented 5 years ago

Is it actually cycling up to 100% (e.g. the fan gets super loud), or does it stay down?

Yes, I checked both are running at 100%.

Is it limited to that card, or does it occur on the other card as well

We only have one machine with RadeonVII. So I cannot tell sorry.

What if you try to specify device #2 with "rocm-smi -d 2 --setfan 100%"

Still shows below 100% fan speed.

 > rocm-smi -d 2 --setfan 100%
=====================ROCm System Management Interface========================
GPU[2]          : Successfully set fan speed to Level 255
==============================End of ROCm SMI Log ==============================
> rocm-smi 
========================ROCm System Management Interface========================
================================================================================
GPU  Temp   AvgPwr  SCLK     MCLK     Fan     Perf  PwrCap  SCLK OD  MCLK OD  GPU%  
1    72.0c  219.0W  1802Mhz  1001Mhz  100.0%  auto  250.0W  0%       0%       100%  
2    32.0c  20.0W   701Mhz   351Mhz   18.82%  auto  250.0W  0%       0%       0%    
================================================================================
==============================End of ROCm SMI Log ==============================
kentrussell commented 5 years ago

Peculiar. The percentage reported there is just the result of 2 sysfs files: fan/fanmax (where fan is pwm1 and fanmax is pwm1_max). If you are setting the fan to 100% (255) and pwm1 is still reporting as 73 or some other low number, then that's a kernel bug with the hwmon reporting the wrong value.

Is it limited to just that card? If you remove the other RadeonVII card, does it show the same behaviour? If the card still runs at 100%, even though it's only reporting that it's running at 10%, then I don't think there's a hardware issue. After bumping it up to 100%, are there any powerplay-related messages in dmesg to note?

kentrussell commented 4 years ago

This should be addressed with an updated VBIOS. Please try that out and re-open this if the issue isn't resolved (had to get the VBIOS guys to update some invalid fields)