Closed ivi-lab closed 2 days ago
Same problem as #660 ?
Same problem as #660 ?
I had someone question if it was, so I opened this with a simple description of the issue.
Just for clarity, are you using the Active Cooler or the Case Fan ?
Just for clarity, are you using the Active Cooler or the Case Fan ?
Neither. A Noctua PWM 5V fan connected to the fan connector.
We're a bit stumped. Can you run some tests, first from a cold boot and then after a reboot, and post the output?
$ pinctrl | grep FAN
$ grep . /sys/class/hwmon/hwmon*/{temp,fan,pwm}*
Cold boot:
29: no pu | hi // FAN_TACH/GPIO29 = none 45: a0 pd | hi // FAN_PWM/GPIO45 = PWM1_CHAN3
/sys/class/hwmon/hwmon0/temp1_input:36400 /sys/class/hwmon/hwmon1/temp1_alarm:0 /sys/class/hwmon/hwmon1/temp1_crit:85850 /sys/class/hwmon/hwmon1/temp1_input:24850 /sys/class/hwmon/hwmon1/temp1_label:Composite /sys/class/hwmon/hwmon1/temp1_max:82850 /sys/class/hwmon/hwmon1/temp1_min:-273150 /sys/class/hwmon/hwmon1/temp2_input:24850 /sys/class/hwmon/hwmon1/temp2_label:Sensor 1 /sys/class/hwmon/hwmon1/temp2_max:65261850 /sys/class/hwmon/hwmon1/temp2_min:-273150 /sys/class/hwmon/hwmon1/temp3_input:27850 /sys/class/hwmon/hwmon1/temp3_label:Sensor 2 /sys/class/hwmon/hwmon1/temp3_max:65261850 /sys/class/hwmon/hwmon1/temp3_min:-273150 /sys/class/hwmon/hwmon4/temp1_input:35716 /sys/class/hwmon/hwmon4/temp1_raw:853 /sys/class/hwmon/hwmon2/fan1_input:0 /sys/class/hwmon/hwmon2/pwm1:0 /sys/class/hwmon/hwmon2/pwm1_enable:1
Soft reboot:
29: no pu | hi // FAN_TACH/GPIO29 = none 45: op dl pd | lo // FAN_PWM/GPIO45 = output
/sys/class/hwmon/hwmon0/temp1_input:38050 /sys/class/hwmon/hwmon1/temp1_alarm:0 /sys/class/hwmon/hwmon1/temp1_crit:85850 /sys/class/hwmon/hwmon1/temp1_input:35850 /sys/class/hwmon/hwmon1/temp1_label:Composite /sys/class/hwmon/hwmon1/temp1_max:82850 /sys/class/hwmon/hwmon1/temp1_min:-273150 /sys/class/hwmon/hwmon1/temp2_input:35850 /sys/class/hwmon/hwmon1/temp2_label:Sensor 1 /sys/class/hwmon/hwmon1/temp2_max:65261850 /sys/class/hwmon/hwmon1/temp2_min:-273150 /sys/class/hwmon/hwmon1/temp3_input:30850 /sys/class/hwmon/hwmon1/temp3_label:Sensor 2 /sys/class/hwmon/hwmon1/temp3_max:65261850 /sys/class/hwmon/hwmon1/temp3_min:-273150 /sys/class/hwmon/hwmon3/temp1_input:39783 /sys/class/hwmon/hwmon3/temp1_raw:844 grep: /sys/class/hwmon/hwmon*/fan*: No such file or directory grep: /sys/class/hwmon/hwmon*/pwm*: No such file or directory
Also just to note, at the start of a cold boot the fan speeds up/pulses twice before operating normally. However this doesn't happen after a soft reboot.
Well, it's clear what is wrong, just not why.
29: no pu | hi // FAN_TACH/GPIO29 = none 45: a0 pd | hi // FAN_PWM/GPIO45 = PWM1_CHAN3
29: no pu | hi // FAN_TACH/GPIO29 = none 45: op dl pd | lo // FAN_PWM/GPIO45 = output
In the reboot case, the fan control pin is an output driving low (i.e. fan on full), not a PWM output. Also, the kernel's PWM driver is not loaded after reboot, suggesting that the firmware did not enable it.
The firmware does briefly make GPIO 29 an output driving low during booting in order to spin the fan, so that it can use pulses on GPIO 45 to detect its presence. Once it has decided, it drives the 29 high to stop the fan. If it detects the fan then it triggers the loading of the pwm-fan overlay.
If something were to cause the firmware to abandon that sequence (and I have no theories yet as to what) then the pin could be left driving low. In fact, older firmwares did leave it low intentionally.
vcgencmd bootloader_version
report?Very interesting, thanks for the detailed explanation.
2025/03/10 17:10:37 version 2bb2ae640058a2f3aa8dcbdc2da33302e509668d (release) timestamp 1741626637 update-time 1741745629 capabilities 0x0000007f
It's a 60mm fan (NF-A6x25 5V PWM).
just checking mine: rpi5 8gb
pinctrl | grep FAN
29: no pu | hi // FAN_TACH/GPIO29 = none
45: op dl pd | lo // FAN_PWM/GPIO45 = output
grep . /sys/class/hwmon/hwmon*/{temp,fan,pwm}*
/sys/class/hwmon/hwmon0/temp1_input:23200
/sys/class/hwmon/hwmon1/temp1_alarm:0
/sys/class/hwmon/hwmon1/temp1_crit:84850
/sys/class/hwmon/hwmon1/temp1_input:17850
/sys/class/hwmon/hwmon1/temp1_label:Composite
/sys/class/hwmon/hwmon1/temp1_max:82850
/sys/class/hwmon/hwmon1/temp1_min:-273150
/sys/class/hwmon/hwmon1/temp2_input:17850
/sys/class/hwmon/hwmon1/temp2_label:Sensor 1
/sys/class/hwmon/hwmon1/temp2_max:65261850
/sys/class/hwmon/hwmon1/temp2_min:-273150
/sys/class/hwmon/hwmon2/temp1_input:28162
/sys/class/hwmon/hwmon2/temp1_raw:868
grep: /sys/class/hwmon/hwmon*/fan*: No such file or directory
grep: /sys/class/hwmon/hwmon*/pwm*: No such file or directory
BOOTLOADER: up to date
CURRENT: Mon Mar 10 05:10:37 PM UTC 2025 (1741626637)
LATEST: Mon Mar 10 05:10:37 PM UTC 2025 (1741626637)
RELEASE: latest (/usr/lib/firmware/raspberrypi/bootloader-2712/latest)
cat /etc/rpi-issue
Raspberry Pi reference 2023-12-05
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 70cd6f2a1e34d07f5cba7047aea5b92457372e05, stage4
vcgencmd bootloader_version
2025/03/10 17:10:37
version 2bb2ae640058a2f3aa8dcbdc2da33302e509668d (release)
timestamp 1741626637
update-time 1741693622
capabilities 0x0000007f
vcgencmd version
2025/03/10 17:10:37
Copyright (c) 2012 Broadcom
version 2bb2ae64 (release) (embedded)
uname -a
Linux raspberrypi 6.6.74+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.74-1+rpt1 (2025-01-27) aarch64 GNU/Linux
also using a 40mm noctua pwm 5v fan with a custom soldered JST SH 1.0 header connector
Interesting. That's the same state I get if I plug the fan in after the Pi has booted.
One thing you can try, to see if the FAN_TACH signal is working, is pinctrl poll FAN_TACH
. You should get a rapid stream of output like this:
29: lo // FAN_TACH/GPIO29
+1882us
29: hi // FAN_TACH/GPIO29
+2001us
29: lo // FAN_TACH/GPIO29
+1941us
29: hi // FAN_TACH/GPIO29
+1906us
...
Hit Ctrl-C to exit.
I tried holding the fan still while it rebooted, but that didn't seem to stop the fan detection, which is unexpected.
It might be worth checking that the pins in the JST connector are all straight - I've known a few people to have bent one, breaking the circuit, but it's not so likely that both of you are in the same boat,
I know the pins and wiring on mine is solid. If it's any help, this is what it shows:
Cold boot:
29: lo // FAN_TACH/GPIO29
+17091us
29: hi // FAN_TACH/GPIO29
+15593us
29: lo // FAN_TACH/GPIO29
+16564us
29: hi // FAN_TACH/GPIO29
+15636us
29: lo // FAN_TACH/GPIO29
+17089us
Soft reboot:
29: lo // FAN_TACH/GPIO29
+4797us
29: hi // FAN_TACH/GPIO29
+4443us
29: lo // FAN_TACH/GPIO29
+4660us
29: hi // FAN_TACH/GPIO29
+4442us
29: lo // FAN_TACH/GPIO29
+4837us
mine seems to be fine. not having this issue
hard poweroff coldboot & soft reboot sudo reboot now
29: hi // FAN_TACH/GPIO29
+2588us
29: lo // FAN_TACH/GPIO29
+3162us
29: hi // FAN_TACH/GPIO29
+2634us
29: lo // FAN_TACH/GPIO29
+3161us
also updated my kernel
Linux raspberrypi 6.12.18-v8+ #1862 SMP PREEMPT Wed Mar 12 12:31:19 GMT 2025 aarch64 GNU/Linux
no issues here. cant reproduce it
can the OP show which eeprom they are on?
vcgencmd bootloader_version
vcgencmd version
also Q, while we're here, is there a command to easily control PWM fan speed? i would like to test running full speed then set it back to auto.
can the OP show which eeprom they are on?
2025/03/10 17:10:37
version 2bb2ae640058a2f3aa8dcbdc2da33302e509668d (release)
timestamp 1741626637
update-time 1741745629
capabilities 0x0000007f
2025/03/10 17:10:37
Copyright (c) 2012 Broadcom
version 2bb2ae64 (release) (embedded)
i would like to test running full speed then set it back to auto.
The easy way is to hold the fan control signal low for full speed:
pinctrl FAN_PWM op DL
then return it to be a PWM signal for auto:
pinctrl FAN_PWM a0
Is any more info needed from me? I'm super keen for this to get fixed.
We're still not sure why this isn't working for you, so there's a test image which does four things:
sudo vclog -m | grep RPM
.You can download the update from here: https://drive.google.com/file/d/1znyhaqxqhPRmWPDYzX4OkM6QET48ATay/view?usp=sharing
Install it with:
sudo rpi-eeprom-update -d -f pieeprom_fandebug_250317.bin
sudo reboot
Here are the results after the test. During and after the soft reboot, the fan does not spin at all.
Cold Boot:
013875.629: RPM 2668, max RPM 2668
Soft reboot:
007471.741: RPM 0, max RPM 0
007973.170: RPM now 0
Just to double check this is the bootloader file you posted?
CURRENT: Fri 14 Mar 2025 13:30:17 UTC (1741959017)
Yes, that's the correct image - it's the only one with that diagnostic output.
During and after the soft reboot, the fan does not spin at all.
I'm both surprised and not surprised. Your symptoms were as if the fan was present at cold boot not present on a reboot, and that's what the debug output is saying, but at the moment I have no explanation for why this isn't working for you.
Fortunately, there is an easy workaround - just add dtparam=cooling_fan=on
to config.txt. It's harmless if there is no fan, with just a tiny increase in power consumption.
You should be fine with that trial EEPROM image until the next release is made. Hopefully we can work out the failure mechanism and come up with a fix by then, but it is feeling like a subtle hardware problem so I can't promise anything.
Thanks for the help so far. The fact that the fan consistently runs normally from a cold boot makes me think its a software issue.
I just feel like something is being skipped when it's rebooted. On a cold boot, the fan speeds up twice (probing?) before running normally. However on a reboot this never happens.
We'd like to get hold of some of those fans for testing. Can you (both) send us a link to where you bought them from, or at least an accurate product name?
Noctua NF-A6X25 5V PWM 60mm Fan https://noctua.at/en/products/fan/nf-a6x25-5v-pwm
Thanks. It will take a while to get some fans in to test, but in the meantime there is one more thing you can try. It won't solve your problem, but it might help our understanding.
Can you try enabling POWER_OFF_ON_HALT
?
$ sudo rpi-eeprom-config -e
POWER_OFF_ON_HALT=1
, or change POWER_OFF_ON_HALT=0
to POWER_OFF_ON_HALT=1
.dtparam=cooling_fan=on
.sudo poweroff
.sudo poweroff
dtparam=cooling_fan=on
.What makes this test interesting is that everything will be powered off except the fan.
Fortunately, there is an easy workaround - just add
dtparam=cooling_fan=on
to config.txt. It's harmless if there is no fan, with just a tiny increase in power consumption.
I just wanted to try this first. I reset everything back to the latest eeprom. Adding dtparam=cooling_fan=on
to config.txt fixes the issue. After 5 soft reboots, I can say the fan speed works properly. It even speeds up twice at the start, the same way it does for a cold boot.
This has me thinking, I assume that bit of code is read by the firmware and is being told "there's a pwm fan connected". But without it, normally the firmware would need some sort of sequence to detect if one is connected? So it seems on a cold boot it's going through that sequence but after a soft reboot it's being skipped.
So you didn't fancy trying the POWER_OFF_ON_HALT=1
test?
Sorry for the delay. Yes I have. Just to note there's a tiny error in step 1. with a 't' in 'eeprom'.
6. Cold boot the board, confirming that the fan is working.
Yes, it runs normally.
9. Is the fan detected?
Yes, it runs normally.
I also tried a soft reboot (with POWER_OFF_ON_HALT=1
and without dtparam=cooling_fan=on
).
The fan just runs full speed.
Just to note there's a tiny error in step 1. with a 't' in 'eeprom'.
Oops - fixed.
- Is the fan detected?
Yes, it runs normally.
Thanks - that's useful information.
it seems on a cold boot it's going through that sequence but after a soft reboot it's being skipped
There's one more debug image, which may shed some light on this: https://drive.google.com/file/d/1OKGoOixVK6XugRS3ARFaCEhM05RCNS_c/view?usp=drive_link Same procedure as before. This time the RPM diagnostic gains a "debug" value. The base value should be 1000000. Any increment above that is the count of how many pulses were counted by the firmware, which is how the RPM is derived.
With a stalled fan I get:
RPM 0, max RPM 298, debug 1000002
I predict you will see:
RPM 0, max RPM 0, debug 1000000
Rather than speculate any further, I'm going to wait until the fans arrive and run some proper tests. In the meantime, continue to use the dtparam.
One further question - how is the fan connected - have you spliced the fan's cable into the Raspberry Pi active cooler fan connector? If so, please attach a picture.
https://www.amazon.com/Noctua-NF-A4x10-5V-PWM-Premium/dp/B07DXS86G7
NF-A4x10-5V-PWM
my fan is non standard, the wire has been cut and a JST SH connector soldered on correctly pin matched. for two reasons: shorter cable, & cant find a JST SH to 4pin fan adapter.
i want to note that default 4 pin fan header connectors do not match the rpi JST SH pinout you must compare the fan pinout to the rpi pinout.
RPI5:
Pin1 red wire
Pin2 blue wire
Pin3 black wire
Pin4 yellow wire
Pin1 5V
Pin2 PWM
Pin3 GND
Pin4 Tach
NOCTUA
pin1 blue
pin2 green
pin3 yellow
pin4 black
pin1 PWN
pin2 TACH
pin3 5V
pin4 GND
I think the fact that the fans work from cold boot suggests the wires are correctly connected.
@P33M I've designed this adaptor cable and had it manufactured just for this purpose.
As @arrowgent has mentioned, the pin layouts are different.
Also as @pelwell mentioned, it works from cold boots. It was also my first thought that the wiring or fan was the issue. I've checked the layout, continuity, a second fan etc.
There's one more debug image, which may shed some light on this:
Here are the results:
Soft reboot:
No fan movement at all.
007521.309: RPM 0, max RPM 0, debug 1000000
Cold boot:
Fan starts and runs normally.
013948.264: RPM 2456, max RPM 2456, debug 1000061
In case you feel like some more investigation, and while we're waiting for the order to be delivered, there's another trial image: https://drive.google.com/file/d/1-pYUrgcJGXYC-eUSmzzskcBNVEFODLQF/view?usp=sharing
The debug field is now hexadecimal, with the following interpretation:
The value can be queried at runtime:
$ watch -n 0.1 sudo busybox devmem 0x1f0009c040
It appears that the FAN_TACH signal is gated by the FAN_PWM output - if FAN_PWM is held high then FAN_TACH is always high, and when FAN_PWM is driven low, FAN_TACH immediately goes low, then toggles as the fan revolves. You can observe the timing with a pinctrl poll FAN_TACH
in one window and things like pinctrl FAN_PWM op dl; sleep 0.15; pinctrl FAN_PWM op dh
in another.
Despite my terrible soldering I've been able to connect one of the new fans to my Pi 5 and reproduce the issue. The behaviour is exactly as described. Watching the FAN_PWM signal there is no obvious difference between boot and reboot, but there's a clear difference in FAN_TACH - it just does nothing in the reboot case. Logic dictated that something else must be different, and there aren't many pins left, so I tried driving the fan's 5V directly from one of the 5V header pins. With that configuration, it works after a reboot as well as a boot.
For safety reasons, the fan's 5V is derived from the same source as USB VBUS, so my theory is that VBUS is being driven differently on a reboot. However, I haven't been brave enough to connect my 3.3V logic analyser to the 5V VBUS signal to confirm that yet.
I was thinking of offering to express post a couple of my cables but it honestly costs a fortune (from Australia to UK) even in a letter. On top of that there's a customs form.
Anyway I'm really glad the issue has replicated and thanks for the info. What size fan did you order?
Do you still want me to try the previous image? I think you'll diagnose this way faster than me.
What size fan did you order?
60x25, 40x20 and 40x10 - just in case.
Do you still want me to try the previous image
No, there's no point now, except for personal curiosity.
However, I haven't been brave enough to connect my 3.3V logic analyser to the 5V VBUS signal to confirm that yet.
If you have a 1.8kΩ and 3.3kΩ resistor around (or a similar pair) you could throw in a simple voltage divider.
Thanks - I'm sure I could rustle up something like that in the morning. It was late, and I was hungry.
It was late, and I was hungry.
True story - I bumped into him in Tesco! 😂
So... these Noctua fans are too clever. It seems that they won't turn on the fan earlier than ~4.6s after the power is applied. For a boot this is fine because with the pause for the boot menu, the memory controller calibration etc., it's a good 6.6s after VBUS is driven high before we get round to trying the fan. Reboot is much quicker - the delay after VBUS is enabled drops to about 2.4s, which is 2.2s too short.
I've been able to verify this - adding an extra 2000ms of delay fixes the problem; 1800ms would probably be enough. We obviously don't want to delay all boots just to allow these fans time to get their act together, and we can't turn on VBUS any earlier than we do, so we're stuck. We could add an EEPROM config setting specifying how long to wait for the fan to start, but that's more complicated and essentially no better than the existing dtparam=cooling_fan
workaround.
Final(?) observation: the smaller fans, 40mm x 10mm and 40mm x 20mm, are much quicker to start - 2.3s and 2.7s respectively. This difference is enough to remove the need for an additional delay, but there may be variation between different instances of the same design.
I don't think it warrants adding an eeprom config. If the fan is smart enough to evade autodetection, then manually confirming that it's attached with a config.txt entry is sufficient.
I'm glad to hear you've figured it out. Unfortunately for me, my product was designed around that 60mm fan. Configuring the dtparam=cooling_fan
workaround into every device would be a nightmare. 2 seconds doesn't sound like a bad sacrifice to me for greater compatibility but I guess it's not my decision to make. I might have to test out the other (slimmer) Noctua NF-A6x15 5V fan and hope that it works instead.
2 seconds useless waiting will be a lot for embedded devices which have passive cooling.
Does your custom HW have a HAT eeprom (if not you really should consider it)? This will make it really simple to ship dt quirks without user interaction.
We can rule out adding large global delays or any other mechanism which makes the boot experience worse for everyone in order to apply a device specific quirk
There are at least four mechanisms available to add device specific quirks
If you are building and selling custom device (not clear from original post) then the rpi-image-gen and rpi-sb-provisioner tools are there simplify the creation and provisioning of OS customisation.
https://www.raspberrypi.com/news/introducing-rpi-image-gen-build-highly-customised-raspberry-pi-software-images/ https://github.com/raspberrypi/rpi-sb-provisioner
I think cold boot will also fail on CM5 or if network install is not enabled on pi5 because the delay won’t occur. If that’s not the case then we could consider using and aim bit to remember the probe across reboot. Assume the fan is still there if it was there at power on
I can confirm that runtime application of the dtparam works as expected. A custom DTB is yet another option.
My hardware is all off the shelf (apart from the fan cable). The OS (Home Assistant) is also untouched. I'm hoping to keep it as simple as possible.
My biggest concern is if any custom configuration like dtparam will remained saved. For example in the last HAOS update [15.0] there was a 'Update RPi firmware to fix boot with 2025-02-11 bootloader'.
A little off topic but I thought I'd try Ubuntu Desktop LTS (64 bit) and interestingly the fan works normally on soft reboots . Perhaps someone using Ubuntu ran into this problem before me.
It sounds like the EEPROM [config.txt] section is the best option. The tools always migrate the bootloader config when upgrading and I don’t believe HaOS would trample over user settings here without warning.
Describe the bug
After a soft reboot the fan speed continuously runs at 100%. The only way to fix it is with a soft or hard shut down.
Hardware: Pi 5 8GB with an official Pi 5 power supply, fan and a clean Pi OS install on a SD card.
I've tested both default and latest eeprom releases also on a second (brand new) Pi 5 8GB. I also tried a M.2 PoE HAT running HAOS with the same result.
Steps to reproduce the behaviour
Device (s)
Raspberry Pi 5
Bootloader configuration.
Default.
System
No response
Bootloader logs
No response
USB boot
No response
NVMe boot
No response
Network (TFTP boot)
No response