Run it in TrueNAS - Githubissues

slavikme commented 5 months ago

Any idea how to run it on TrueNAS? TrueNAS doesn't have pip installed and it seems that you can't easily (and not recommended to) install it.

luckylinux commented 4 months ago

@slavikme: No worries, I'm also extremely busy with 10 different Projects and more. I feel you.

By the way, the ipmitools raw does successfully executed without any errors, despite the fact that it doesn't change the fan speed at all. WHICH command did you execute though ? The one that is being currently run in the code ?

Try (immediately after your ipmitool Command):

echo $?

Just to make sure that the exit code is 0 (executed Normally and not just having the output suppressed).

In the latest Version I change Fan Mode to "Optimal" first, then set it to "Full Speed" later.

Nevertheless, another idea come to Mind. You only have 1 Fan installed, so it's possible that the Supermicro IPMI/BMC Firmware just thinks: "Oh, I only got 1 fan, then I go Full Speed".

That's IIRC what "Optimal" does. If one Fan fails, then it goes Full Speed.

I don't think it would be a stretch that "Full Speed" might do the same, especially where you do NOT have ANY Fan in one of the Control Zones. You only have ONE Fan in ONE Fan Zone.

Try to see if that changes when you add more Fans.

Furthermore, check the IPMI Event Log / Health Event Log for Clues.

On a separate Note, I was always experiencing Fan "Oscillations" (Full Speed - Zero - Full Speed - Zero - etc) and had to run a Script Similar to this to fix it:

#!/bin/bash

ipmitool sensor thresh FAN1 lower 150 225 300
ipmitool sensor thresh FAN2 lower 150 225 300
ipmitool sensor thresh FAN3 lower 150 225 300
ipmitool sensor thresh FAN4 lower 150 225 300
ipmitool sensor thresh FANA lower 150 225 300

I assume you already did so as your minimum Fan Speed is quite low.

But I can also confirm that when I tried my code (on a Supermicro X10SLM-F/X10SLL-F) with some low-speed Fans (Arctic F8 PWM, Arctic P8 Max PWM), then it just runs full speed anyways. Which it should NOT do ...

The Arctic P8 Max is by no means a low-speed Fan (much quieter than Stock though) with a relatively good Linearity over the entire speed Range:

I don't know what the Issue is to be honest. There might be a supplementary regulation done by Supermicro in their IPMI/BMC Firmware. Becase at 50% Fan Speed set, the Arctic P8 PWM was running at 5000 rpm which is Full Speed (100 %). Or maybe it's (also ?) monitoring the Power Consumption in Watts of the Fan, rather than its actual Speed ?

In my case, maybe I also need to lower then upper Value, because maybe the controller sees 50% and upper 12000 so it might do something with that as well. Maybe lowering the upper value via ipmitool to the Rated Speed of the FAN (+10%/+20%) might solve that Issue.

Because, on the same Server and Motherboard (exactly the same System, just swapping the Fans back to the Nidec Supermicro Fans) the Variable Fan Speed works as intended. So I doubt it's some funky feedback going on (like on e.g. HP Systems where you need to Ground one pin of the Fan Connector in order to bypass the Error etc).

In your case probably it's the fact that several Fans that are NOT installed that is the Issue.

Try installing several Fans, specifically MORE than 1 Fan for each Fan Zone.

luckylinux commented 4 months ago

@slavikme: One Thing caught my Eye in your ipmitool sensor output as well:

BPN-FAN2         | 13300.000  | RPM        | nc    | 500.000   | 400.000   | 300.000   | 13300.000 | 13400.000 | 13500.000

Not sure what "nc" means in this Case. As an Electrical Engineer it could be "Normally Closed". But in this case, maybe "Not Connected".

It's weird how it EXACTLY matches the 13300.000 in your 3rd column from the right (2nd/3rd before-last).

It could also be that your Sensor there is not connected or the Chip is broken.

Try disconnecting the Cable briefly but be careful with the Temperatures !!!!

Alternatively try to Change the upper Speeds via ipmitool. I guess it would be (untested):

ipmitool sensor thresh BPN-FAN2 upper 5000 5500 6000

Then see if the actual Value changes.

I only have either "ok" (Fan Speed above Lower Non-Critical and below Upper Non-Critical) or "na" (Fan NOT installed).

I don't have "nc" on my End. Do you know what it is otherwise ?

luckylinux commented 4 months ago

Of course it depends also on the Fan itself and its linearity Curve.

I'm currently trying this on a Workstation PC where some Fans are 3-pin (MEH :disappointed:).

And the other Fan is probably a NOCTUA P14/P12 on a NOCTUA NH D14/D15 CPU Cooler with some Low-Noise Adapters.

Minimum Fan Speed: 300 rpm, and apparently it will NOT go higher than 1000 rpm.

So Linearity of the FAN is also an important factor:

Some Fans will stay OFF until e.g. 20% Duty Cycle (below this level you will probably have Fan Oscillations and some Events in the IPMI Event Log due to Supermicro Fan-Failure Protection I think)
Some Fans will maybe be already at FULL Speed above 80% Duty Cycle (not a "Problem", but definitively this will "skew" what your Min/Max Settings should be)
Some (Industrial ?) Fans have a VERY non-linear Rotational Speed vs Duty Cycle

luckylinux commented 4 months ago

@slavikme: I can confirm there is "something". I just don't know what exactly.

To investigate Properly, one would need to connect an oscilloscope to one of the unused Fan Headers and monitor the duty Cycle of that (Reference Speed). Or even better with an Adapter monitor the Feedback of an Active Fan.

What I could observe, in "Full Speed" Mode, was that when I try to reduce (or even INCREASE) then Fan Speed manually using ipmitool, sometimes the Fan Speed just drops to Zero.

When "Non-Recoverable" Protection Kicks in (nr), then Fan will go to Full Speed (1000 rpm), which is higher than what I set using ipmitool raw 0x30 0x70 0x66 0x01 0x01 0xff.

I'm not sure why that is though :disappointed:.

root@HOST:/# date; ipmitool raw 0x30 0x70 0x66 0x01 0x01 0xff ; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:17:15 CEST

FANA             | 700,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x9f ; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:17:22 CEST

FANA             | 700,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:17:35 CEST

FANA             | 700,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:17:39 CEST

FANA             | 700,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x8f ; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:17:48 CEST

FANA             | 600,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:17:54 CEST

FANA             | 600,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x7f ; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:18:03 CEST

FANA             | 500,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:18:09 CEST

FANA             | 400,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x6f ; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:18:17 CEST

FANA             | 300,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:18:23 CEST

FANA             | 0,000      | RPM        | nr    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:18:28 CEST

FANA             | 1000,000   | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:18:33 CEST

FANA             | 1000,000   | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:18:38 CEST

FANA             | 1000,000   | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; ipmitool raw 0x30 0x70 0x66 0x01 0x01 0xaf ; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:18:56 CEST

FANA             | 900,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:19:02 CEST

FANA             | 900,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; ipmitool raw 0x30 0x70 0x66 0x01 0x01 0xbf ; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:19:11 CEST

FANA             | 900,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:19:15 CEST

FANA             | 900,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; ipmitool raw 0x30 0x70 0x66 0x01 0x01 0xcf ; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:19:22 CEST

FANA             | 300,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:19:26 CEST

FANA             | 0,000      | RPM        | nr    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:19:30 CEST

FANA             | 800,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:19:37 CEST

FANA             | 1000,000   | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; ipmitool raw 0x30 0x70 0x66 0x01 0x01 0xdf ; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:19:45 CEST

FANA             | 500,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:19:50 CEST

FANA             | 300,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:19:54 CEST

FANA             | 300,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:19:58 CEST

FANA             | 300,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:20:02 CEST

FANA             | 300,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:20:05 CEST

FANA             | 300,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:20:09 CEST

FANA             | 300,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; ipmitool raw 0x30 0x70 0x66 0x01 0x01 0xef ; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:20:16 CEST

FANA             | 400,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:20:20 CEST

FANA             | 500,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:20:24 CEST

FANA             | 500,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:20:27 CEST

FANA             | 500,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; ipmitool raw 0x30 0x70 0x66 0x01 0x01 0xff ; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:20:35 CEST

FANA             | 600,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:20:39 CEST

FANA             | 700,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:20:43 CEST

FANA             | 700,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000 

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:24:56 CEST

FANA             | 700,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; ipmitool raw 0x30 0x70 0x66 0x01 0x01 0xaf ; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:25:03 CEST

FANA             | 800,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:25:07 CEST

FANA             | 900,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; ipmitool raw 0x30 0x70 0x66 0x01 0x01 0xb5 ; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:25:16 CEST

FANA             | 900,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:25:20 CEST

FANA             | 900,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; ipmitool raw 0x30 0x70 0x66 0x01 0x01 0xbf ; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:25:28 CEST

FANA             | 900,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:25:32 CEST

FANA             | 900,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; ipmitool raw 0x30 0x70 0x66 0x01 0x01 0xcf ; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:25:41 CEST

FANA             | 500,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:25:46 CEST

FANA             | 0,000      | RPM        | nr    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:25:50 CEST

FANA             | 800,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:25:54 CEST

FANA             | 1000,000   | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; ipmitool raw 0x30 0x70 0x66 0x01 0x01 0xc5 ; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:26:02 CEST

FANA             | 900,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:26:07 CEST

FANA             | 900,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:26:11 CEST

FANA             | 900,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; ipmitool raw 0x30 0x70 0x66 0x01 0x01 0xc9 ; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:26:19 CEST

FANA             | 900,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; ipmitool raw 0x30 0x70 0x66 0x01 0x01 0xc7 ; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:26:27 CEST

FANA             | 900,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:26:32 CEST

FANA             | 1000,000   | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:26:36 CEST

FANA             | 1000,000   | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; ipmitool raw 0x30 0x70 0x66 0x01 0x01 0xc6 ; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:26:44 CEST

FANA             | 900,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:26:48 CEST

FANA             | 1000,000   | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:26:56 CEST

FANA             | 1000,000   | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000

As you can see, it seems that above 0xc6 the Fan just gives out and falls to Zero for whatever Reason. That's 198 out of the supposedly 255 for the full Range.

Above 0xc6 Supermicro IPMI/BMC controller says "Non-Recoverable Error" (nr) and makes it go 100% Speed (1000 rpm).

In your case nc probably means "Non-Critical". Try to raise those Speed Limits.

IPMI Log (you need to correct the Date/Time by 2h due to GMT+1 vs UTC Timestamps and Solar/Legal Hour):

1,2024/07/04 06:45:02,FANA            ,Fan,Lower Critical - Going Low - Assertion
2,2024/07/04 06:45:05,FANA            ,Fan,Lower Critical - Going Low - De-assertion
3,2024/07/04 06:45:40,FANA            ,Fan,Lower Critical - Going Low - Assertion
4,2024/07/04 06:45:44,FANA            ,Fan,Lower Critical - Going Low - De-assertion
5,2024/07/04 06:46:15,FANA            ,Fan,Lower Critical - Going Low - Assertion
6,2024/07/04 06:46:20,FANA            ,Fan,Lower Critical - Going Low - De-assertion
7,2024/07/04 06:50:44,FANA            ,Fan,Lower Critical - Going Low - Assertion
8,2024/07/04 06:50:47,FANA            ,Fan,Lower Critical - Going Low - De-assertion
9,2024/07/04 07:05:55,FANA            ,Fan,Lower Critical - Going Low - Assertion
10,2024/07/04 07:05:58,FANA            ,Fan,Lower Critical - Going Low - De-assertion
11,2024/07/04 07:09:11,FANA            ,Fan,Lower Critical - Going Low - Assertion
12,2024/07/04 07:09:11,FANA            ,Fan,Lower Non-Recoverable - Going Low - Assertion
13,2024/07/04 07:09:15,FANA            ,Fan,Lower Non-Recoverable - Going Low - De-assertion
14,2024/07/04 07:09:16,FANA            ,Fan,Lower Critical - Going Low - De-assertion
15,2024/07/04 07:09:29,FANA            ,Fan,Lower Critical - Going Low - Assertion
16,2024/07/04 07:09:29,FANA            ,Fan,Lower Non-Recoverable - Going Low - Assertion
17,2024/07/04 07:09:32,FANA            ,Fan,Lower Non-Recoverable - Going Low - De-assertion
18,2024/07/04 07:09:32,FANA            ,Fan,Lower Critical - Going Low - De-assertion
19,2024/07/04 07:09:37,FANA            ,Fan,Lower Critical - Going Low - Assertion
20,2024/07/04 07:09:37,FANA            ,Fan,Lower Non-Recoverable - Going Low - Assertion
21,2024/07/04 07:09:41,FANA            ,Fan,Lower Non-Recoverable - Going Low - De-assertion
22,2024/07/04 07:09:41,FANA            ,Fan,Lower Critical - Going Low - De-assertion
23,2024/07/04 07:09:49,FANA            ,Fan,Lower Critical - Going Low - Assertion
24,2024/07/04 07:09:49,FANA            ,Fan,Lower Non-Recoverable - Going Low - Assertion
25,2024/07/04 07:09:53,FANA            ,Fan,Lower Non-Recoverable - Going Low - De-assertion
26,2024/07/04 07:09:53,FANA            ,Fan,Lower Critical - Going Low - De-assertion
27,2024/07/04 07:09:58,FANA            ,Fan,Lower Critical - Going Low - Assertion
28,2024/07/04 07:09:58,FANA            ,Fan,Lower Non-Recoverable - Going Low - Assertion
29,2024/07/04 07:10:02,FANA            ,Fan,Lower Non-Recoverable - Going Low - De-assertion
30,2024/07/04 07:10:02,FANA            ,Fan,Lower Critical - Going Low - De-assertion
31,2024/07/04 07:15:59,FANA            ,Fan,Lower Critical - Going Low - Assertion
32,2024/07/04 07:16:00,FANA            ,Fan,Lower Non-Recoverable - Going Low - Assertion
33,2024/07/04 07:16:03,FANA            ,Fan,Lower Non-Recoverable - Going Low - De-assertion
34,2024/07/04 07:16:03,FANA            ,Fan,Lower Critical - Going Low - De-assertion
35,2024/07/04 07:16:14,FANA            ,Fan,Lower Critical - Going Low - Assertion
36,2024/07/04 07:16:14,FANA            ,Fan,Lower Non-Recoverable - Going Low - Assertion
37,2024/07/04 07:16:18,FANA            ,Fan,Lower Non-Recoverable - Going Low - De-assertion
38,2024/07/04 07:16:18,FANA            ,Fan,Lower Critical - Going Low - De-assertion
39,2024/07/04 07:18:22,FANA            ,Fan,Lower Critical - Going Low - Assertion
40,2024/07/04 07:18:22,FANA            ,Fan,Lower Non-Recoverable - Going Low - Assertion
41,2024/07/04 07:18:26,FANA            ,Fan,Lower Non-Recoverable - Going Low - De-assertion
42,2024/07/04 07:18:26,FANA            ,Fan,Lower Critical - Going Low - De-assertion
43,2024/07/04 07:19:28,FANA            ,Fan,Lower Critical - Going Low - Assertion
44,2024/07/04 07:19:28,FANA            ,Fan,Lower Non-Recoverable - Going Low - Assertion
45,2024/07/04 07:19:32,FANA            ,Fan,Lower Non-Recoverable - Going Low - De-assertion
46,2024/07/04 07:19:32,FANA            ,Fan,Lower Critical - Going Low - De-assertion
47,2024/07/04 07:25:47,FANA            ,Fan,Lower Critical - Going Low - Assertion
48,2024/07/04 07:25:47,FANA            ,Fan,Lower Non-Recoverable - Going Low - Assertion
49,2024/07/04 07:25:51,FANA            ,Fan,Lower Non-Recoverable - Going Low - De-assertion
50,2024/07/04 07:25:51,FANA            ,Fan,Lower Critical - Going Low - De-assertion
51,2024/07/04 07:26:22,FANA            ,Fan,Lower Critical - Going Low - Assertion
52,2024/07/04 07:26:22,FANA            ,Fan,Lower Non-Recoverable - Going Low - Assertion
53,2024/07/04 07:26:26,FANA            ,Fan,Lower Non-Recoverable - Going Low - De-assertion
54,2024/07/04 07:26:26,FANA            ,Fan,Lower Critical - Going Low - De-assertion

Important: in your (old-ish :smiley:) Version, once the Maximum/Minimum Fan Speed has been set by my Controller, no further ipmitool are Issued.

I changed this behavior in https://github.com/luckylinux/supermicro-fan-control/commit/8be9d7804ff2ca4d42394e28db3ef71edd3a1a04 in order to prevent e.g. Manual ipmitool Commands from permanently setting the Reference (if you want to do that, then stop my systemd Service :smiley:).

So in your Version, for sure, if you reach a point where the Supermicro Automatic Controller kicks in because it detected a non-recoverable Error (like in my Example above), then you'd be stuck at max Speed forever. To "fix" that you need to restart my Service:

systemctl restart supermicro-fan-control.service

Or build the latest Version.

luckylinux commented 4 months ago

@slavikme: According to This the Maximum Fan Speed is apparently NOT 0xff but 0x64.

Indeed at 0x64 I reach 1000 rpm which is the maximum I ever had, when Supermicro IPMI/BMC Controller Protection kicked in:

root@HOST:/# date; ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x64 ; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:44:47 CEST

FANA             | 1000,000   | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:44:52 CEST

FANA             | 1000,000   | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:44:57 CEST

FANA             | 1000,000   | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x32 ; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:45:04 CEST

FANA             | 800,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:45:08 CEST

FANA             | 700,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:45:12 CEST

FANA             | 700,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x16 ; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:45:21 CEST

FANA             | 400,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:45:27 CEST

FANA             | 300,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:46:11 CEST
Received a response with unexpected ID 0 vs. 124

FANA             | 300,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x64 ; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:46:19 CEST

FANA             | 600,000    | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:46:26 CEST

FANA             | 1000,000   | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000  

root@HOST:/# date; sleep 2 ; ipmitool sensor | grep -i FANA
2024-07-04T09:46:31 CEST

FANA             | 1000,000   | RPM        | ok    | 100,000   | 200,000   | 200,000   | 1500,000  | 1800,000  | 2000,000

@slavikme: I'll make those Values (0x64 vs 0xff) a Parameter in the Config File.

luckylinux commented 4 months ago

@slavikme: Please try this https://github.com/luckylinux/supermicro-fan-control/releases/tag/v0.2.0

It worked on my Workstation at least :smiley:.

luckylinux commented 4 months ago

I close this since the Feature has been implemented.

Builds for TrueNAS can be generated using build.sh run from a Debian Bookworm Machine.

For all Intents and Purposes, that has been shown to be working correctly.

@slavikme: For issues Related to e.g. the Specific Motherboard or other Feature Requests / BUG Reports please File a new Issue.

Note that as soon as I started playing with a Supermicro X11SSM-F with Intel Xeon E3-1230 v5, as opposed to X10SLL-F/X10SLM-F with Intel Xeon E3-12xx v3, I could see other Issues: ipmitool NOT returning e.g. CPU Temperature Correctly every single Time causing the Program to Crash/Restart and Fans ramping up/down. I'll File an Issue For that (the quick-fix in devel Branch was to just add another 2s Delay between ipmitool Commands).

luckylinux / supermicro-fan-control

Run it in TrueNAS #1