Open AdvancedFollower opened 2 years ago
It's working for me, running the current 64-bit Raspberry Pi OS release:
pi@raspberrypi:~ $ uname -a
Linux raspberrypi 5.10.92-v8+ #1514 SMP PREEMPT Mon Jan 17 17:39:38 GMT 2022 aarch64 GNU/Linux
pi@raspberrypi:~ $ vcgencmd version
Jan 20 2022 13:58:22
Copyright (c) 2012 Broadcom
version bd88f66f8952d34e4e0613a85c7a6d3da49e13e2 (clean) (release) (start)
pi@raspberrypi:~ $ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
pi@raspberrypi:~ $ ls /sys/devices/system/cpu/cpu0
cpu_capacity cpufreq of_node power regs subsystem topology uevent
pi@raspberrypi:~ $ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
ondemand
pi@raspberrypi:~ $ vcgencmd measure_clock arm
frequency(48)=600000000
pi@raspberrypi:~ $ echo performance | sudo tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
performance
pi@raspberrypi:~ $ vcgencmd measure_clock arm
frequency(48)=1400000000
@pelwell Did you tried on RPi3B+? Because for me it looks like this.
root@DietPi3:~# uname -a
Linux DietPi3 5.10.92-v8+ #1514 SMP PREEMPT Mon Jan 17 17:39:38 GMT 2022 aarch64 GNU/Linux
root@DietPi3:~# vcgencmd version
Jan 20 2022 13:58:39
Copyright (c) 2012 Broadcom
version bd88f66f8952d34e4e0613a85c7a6d3da49e13e2 (clean) (release) (start_cd)
root@DietPi3:~#
root@DietPi3:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@DietPi3:~# ls /sys/devices/system/cpu/cpu0
cpu_capacity of_node power regs subsystem topology uevent
root@DietPi3:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
cat: /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor: No such file or directory
root@DietPi3:~# vcgencmd measure_clock arm
frequency(48)=600000000
root@DietPi3:~# echo performance | sudo tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
tee: /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor: No such file or directory
performance
root@DietPi3:~# vcgencmd measure_clock arm
frequency(48)=600000000
scaling_governor
doesn't seems to exist.
Yes - that output is from a 3B+.
Please try the official 64-bit image, not DietPi.
interesting point, running rpi-update
on the very same system, seems to bring back scaling_governor
root@DietPi3:~# uname -a
Linux DietPi3 5.15.18-v8+ #1520 SMP PREEMPT Fri Feb 4 12:04:45 GMT 2022 aarch64 GNU/Linux
root@DietPi3:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
ondemand
@MichaIng maybe you like to have a look as well.
Also downgrading to the previous 1.20211118 kernel and bootloader solves it, so something is different with this particular version 1.20220120-1. Indeed a good idea to test with official RPi OS 64-bit. Kernel, bootloader and firmware are however the same, so unless it's something in config.txt or cmdline.txt I'm not sure what could affect something like CPUFreq kernel feature, especially something which affects the RPi 3B+ 🤔 exclusively.
EDIT: Commenting/removal of initial_turbo=20
could be tried (respectively adding it on RPi OS 64-bit). I remember that it broke CPU scheduling in the past with one of the first 5.4 kernel releases.
Can confirm, commenting out initial_turbo=20 from config.txt solves it on my DietPi 8.0.2. I'm guessing initial_turbo isn't enabled by default on stock Debian, then?
Great find. Nope is it not enabled on stock RPi OS. However, it shouldn't break CPUFreq, and it's strange that it does only on RPi 3B+.
@pelwell Any idea what may cause this what was introduced with last kernel/bootloader release and has been fixed (explicitly or implicitly by another change) in master/rpi-update?
I think we'll apply a live patch for RPi 3B+ to disable initial turbo and in case, when fixed until next DietPi release, offer to have it re-enabled during update. CPUFreq of course is much more important than the little boot time speedup with initial turbo.
This is more @popcornmix's domain.
I've run what I believe @Joulinar ran on RpiOS bullseye with the same firmware and kernel and it appeared to behave as expected.
pi@b3plus:~ $ uname -a
Linux b3plus 5.10.92-v8+ #1514 SMP PREEMPT Mon Jan 17 17:39:38 GMT 2022 aarch64 GNU/Linux
pi@b3plus:~ $ vcgencmd version
Jan 20 2022 13:58:22
Copyright (c) 2012 Broadcom
version bd88f66f8952d34e4e0613a85c7a6d3da49e13e2 (clean) (release) (start)
pi@b3plus:~ $ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 11 (bullseye)"
NAME="Raspbian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
pi@b3plus:~ $ ls /sys/devices/system/cpu/cpu0
cpu_capacity cpufreq of_node power regs subsystem topology uevent
pi@b3plus:~ $ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
ondemand
pi@b3plus:~ $ vcgencmd measure_clock arm
frequency(48)=600000000
pi@b3plus:~ $ echo performance | sudo tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
performance
pi@b3plus:~ $ vcgencmd measure_clock arm
frequency(48)=1400000000
pi@b3plus:~ $
@popcornmix
It seems the difference is, if you set initial_turbo=20
https://github.com/raspberrypi/linux/issues/4875#issuecomment-1031532751. As this is not a default value, it most probably working on plain RPi OS
Yes, I had added that.
pi@b3plus:~ $ vcgencmd get_config initial_turbo
initial_turbo=20
cpufreq governor is still working. Also tried with initial_turbo=60 and it works.
Hmm, so removing initial turbo from our config solves it but adding it to RPi OS does not cause it? An issue with a combination of initial turbo and something else? There is no overclocking set on RPi 3B+, so what may be closest related is temp_limit
and probably gpu_mem_*
. Otherwise testing our config.txt (+ arm_64bit=1
which is added later on 64-bit image generation) on RPi OS as a whole. For comparison, here the RPi OS 64-bit config: https://github.com/RPi-Distro/pi-gen/blob/arm64/stage1/00-boot-files/files/config.txt
Sadly I have no 3B+ here for testing. While the issue is solved with next LTS and we have a workaround until then, investigating the underlying reason may still be valuable, probably there is some specific timing involved which may reappear any time later when other things change.
Another question is whether only the 64-bit kernel is affected or whether the same issue appears when using the ARMv6 (Raspbian-based) or ARMv7 (Debian-based) image on RPi 3B+.
@MichaIng Yes, it works on DietPi ARMv7, just did a fresh install of https://dietpi.com/downloads/images/DietPi_RPi-ARMv7-Bullseye.7z on my 3B+
uname -a Linux DietPi 5.10.92-v7+ #1514 SMP Mon Jan 17 17:36:39 GMT 2022 armv7l GNU/Linux vcgencmd version Jan 20 2022 13:58:39 Copyright (c) 2012 Broadcom version bd88f66f8952d34e4e0613a85c7a6d3da49e13e2 (clean) (release) (start_cd) cat /etc/os-release PRETTY_NAME="Debian GNU/Linux 11 (bullseye)" NAME="Debian GNU/Linux" VERSION_ID="11" VERSION="11 (bullseye)" VERSION_CODENAME=bullseye ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" ls /sys/devices/system/cpu/cpu0 cpu_capacity cpufreq of_node power subsystem topology uevent cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor schedutil vcgencmd measure_clock arm frequency(48)=900096000 vcgencmd get_config initial_turbo initial_turbo=20 echo performance | sudo tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor performance vcgencmd measure_clock arm frequency(48)=1400000000
cpufreq governor is still working. Also tried with initial_turbo=60 and it works.
Same here - cpufreq is working fine running the most recent 64-bit kernel (5.15.21-v8+) on a 3B+ with initial_turbo=60.
So it seems to be only on DietPi, only with 5.10.92-v8+/ARM64, only on the Pi 3B+, and only when Initial Turbo is enabled. And it magically fixed itself with the 5.15.21-v8+ kernel.
As long as it doesn't magically break itself again with the next kernel after that I guess that's good enough...
Rewinding to rpi-5.10.92-v8+ (sudo rpi-update b4145cfaa
) on 3B+ with initial_turbo=60 and RPiOS still works.
Steps for further debugging could be: https://github.com/raspberrypi/linux/issues/4875#issuecomment-1033089026 Either commenting other config.txt settings on DietPi which are present only on DietPi by default, while leaving initial turbo enabled, or adding those settings on RPi OS, in addition to initial turbo.
Booting a 3B+ from a RPiOS image with 5.10.92 kernal and the DietPi config.txt (+arm_64bit=1
because the 32-bit kernel is still present) gives a working cpufreq.
It's not the config.txt settings.
I cannot imagine at all that something from cmdline.txt may cause this:
root=PARTUUID=$(findmnt -Ufnro PARTUUID -M /) rootfstype=ext4 rootwait fsck.repair=yes net.ifnames=0 logo.nologo console=serial0,115200 console=tty1
Otherwise what shall affect sysfs/whether CPUFreq is loaded or not, given we use the official raspberrypi-kernel, raspberrypi-bootloader, libraspberrypi0, libraspberrypi-bin and raspberrypi-sys-mods from archive.raspberrypi.org
without any other modification to the contained files or configs 🤔?
We use different (no) filesystem feature flags on image generation, while pi-gen
seems to at least disable huge_file
, not sure whether metadata_csum
and 64bit
are still disabled? Present on one script but not in the others, seems to be only set by pi-gen when USE_QCOW2
env var was set explicitly. However, I cannot imaging how this shall affect anything related. Partitioning and filesystems are otherwise is exactly the same, regarding type, offset and flags.
@AdvancedFollower
Could you paste another full dmesg
output from DietPi, with initial turbo set? A serial console really would be helpful 🤔.
I am having this exact issue, no cpu governor. However, changing turbo to 60 didn't fix it, commenting it out of config.txt solved the issue for me and brought back the governors. Below you will find the same commands from the others above, but with a big difference of being on a Pi4B:
Hardware : BCM2835
Revision : c03111
Serial : 100000007138d936
Model : Raspberry Pi 4 Model B Rev 1.1
initial_turbo=20
dietpi@Taskmaster:~$ uname -a
Linux Taskmaster 5.10.92-v8+ #1514 SMP PREEMPT Mon Jan 17 17:39:38 GMT 2022 aarch64 GNU/Linux
dietpi@Taskmaster:~$ vcgencmd version
Jan 20 2022 13:56:48
Copyright (c) 2012 Broadcom
version bd88f66f8952d34e4e0613a85c7a6d3da49e13e2 (clean) (release) (start)
dietpi@Taskmaster:~$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
dietpi@Taskmaster:~$ ls /sys/devices/system/cpu/cpu0
cpu_capacity of_node power regs subsystem topology uevent
dietpi@Taskmaster:~$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
cat: /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor: No such file or directory
dietpi@Taskmaster:~$ vcgencmd measure_clock arm
dietpi@Taskmaster:~$ echo performance | sudo tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
tee: /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor: No such file or directory
performance
dietpi@Taskmaster:~$ vcgencmd measure_clock arm
frequency(48)=300111328
No Turbo (working)
dietpi@Taskmaster:~$ uname -a
Linux Taskmaster 5.10.92-v8+ #1514 SMP PREEMPT Mon Jan 17 17:39:38 GMT 2022 aarch64 GNU/Linux
dietpi@Taskmaster:~$ vcgencmd version
Jan 20 2022 13:56:48
Copyright (c) 2012 Broadcom
version bd88f66f8952d34e4e0613a85c7a6d3da49e13e2 (clean) (release) (start)
dietpi@Taskmaster:~$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
dietpi@Taskmaster:~$ ls /sys/devices/system/cpu/cpu0
cpu_capacity cpufreq of_node power regs subsystem topology uevent
dietpi@Taskmaster:~$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
conservative
dietpi@Taskmaster:~$ vcgencmd measure_clock arm
frequency(48)=300058592
dietpi@Taskmaster:~$ echo performance | sudo tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
performance
dietpi@Taskmaster:~$ vcgencmd measure_clock arm
frequency(48)=1500345728
Very confusing, we have a bunch of RPi 4 systems where this issue does clearly not appear, probably the specific PCB revision 1.1?
However, to me this points more into the direction that there is a general issue with how firmware and CPUFreq play together, triggered by specific timings (or so) about what is when (not) loaded during boot, which are not met in most cases.
Does apt install rpi-update && rpi-update
solve it as well in your case?
https://forum.openwrt.org/t/rpi4-community-build/69998/1986?u=wulfy23 https://forum.openwrt.org/t/rpi4-community-build/69998/1991?u=wulfy23
165bd7bc5622ee1c721aa5da9af68935075abedd + kernel 5.10.92-ish
edit: hash above was wrong re-tested with 5.10.100
boot-DAT-ELF_20220115_175984a6dc4c321553e295cf9679a6020231caf4/ GOOD
boot-DAT-ELF_20220117_20c5829b0afe5b99d7e807ce90c382bd6997993f/ NOTGOOD<PROBLEM
boot-DAT-ELF_20220118_3f20b832b27cd730deb6419b570f31a98167eef6/ STILLNOTGOOD
boot-DAT-ELF_20220121_827fdd073638fa7b7292d1148fe0af7465111eae/ STILLNOTGOOD
boot-DAT-ELF_20220125_9c04ed2c1ad06a615d8e6479806ab252dbbeb95a/ GOOD>FIXED
boot-DAT-ELF_20220128_e5013c9d596e1a392e5193bc8cb822966b9c7d7c/ GOOD
boot-DAT-ELF_20220219_a6496ae5cd5e9b4cd5b38f7274fa94dce315fa7a/ GOODCURRENT
[ 0.028033] raspberrypi-firmware soc:firmware: Attached to firmware from 2022-02-04 12:05, variant start
[ 0.032038] raspberrypi-firmware soc:firmware: Firmware hash is a26faf97e3bf76bcc23949d7cdab2f96f399a0c3
reverting to some 6months older elf-dat resolved...
Thanks, it makes me feel better when it is not limited to DietPi 😅. Is initial_turbo
set on OpenWRT by default, so that this is one combining element for triggering the issue?
nope... this is all 5.10.92 and the newer firmware from around that date...
(approx 100 users of identical code with all sorts of config.txt stuff and all had the issue)
@wulfy23 posting output of vcgencmd version uname -a and contents of config.txt and cmdline.txt would be useful (when cpufreq governor is missing).
config.txt or cmdline.txt play no role as discussed...
dca632 /usbstick 53°# cat /boot/cmdline.txt
console=ttyAMA1,115200 root=PARTUUID=02070430-02 rootfstype=squashfs,ext4 rootwait fsckparts workqueue.power_efficient=0 dwc_otg.lpm_enable=0 usbcore.autosuspend=-1 snd_bcm2835.enable_headphones=1 snd_bcm2835.enable_hdmi=1 snd_bcm2835.enable_compat_alsa=0
dca632 /usbstick 54°# grep '^[a-z]' /boot/config.txt /boot/distroconfig.txt
/boot/config.txt:enable_uart=1
/boot/config.txt:uart_2ndstage=1
/boot/config.txt:include distroconfig.txt
/boot/config.txt:arm_64bit=1
/boot/config.txt:boot_delay=3
/boot/config.txt:dtoverlay=uart2
/boot/config.txt:dtoverlay=led70
/boot/config.txt:dtparam=i2c_arm=on
/boot/config.txt:dtparam=sd_poll_once
/boot/distroconfig.txt:dtoverlay=disable-bt
vcgencmd version I don't collect (but will do so next time such an event occurs if it helps)... I have posted the git commit hash above that you'd have to extrapolate from
or maybe I do?
[ 0.000000] Linux version 5.10.92 (builder@buildhost) (aarch64-openwrt-linux-musl-gcc (OpenWrt GCC 11.2.0 r18609-266b5c83c3) 11.2.0, GNU ld (GNU Binutils) 2.37) #0 SMP Mon Jan 17 11:38:43 2022
[ 0.000000] Machine model: Raspberry Pi 4 Model B Rev 1.1
dca632 /usbstick 55°# cat _zPREMOVED/pdmesg-202201181413-5.0.19-7.log | grep firmware
[ 0.049993] raspberrypi-firmware soc:firmware: Attached to firmware from 2022-01-17T19:20:34, variant start
[ 0.060003] raspberrypi-firmware soc:firmware: Firmware hash is bd34f55ef7b01b0a367f131060b561a2a58b80bb
[ 0.368364] cpu cpu0: Cannot get clock for CPU0
[ 0.368386] raspberrypi-cpufreq: probe of raspberrypi-cpufreq failed with error -2
and should you need the image in question; http://rpi4.wulfy23.info/builds/devel/rpi-4_snapshot_5.0.19-7_r18609_extra_popcornmix/rpi4.64-snapshot-27266-5.0.19-7-r18609-ext4-fac.img.gz (allow 3mins for the device to initialize on firstboot, web-ui/ssh is accessible at 192.168.1.1 from dhcp enabled client)
[root@dca632 / 39°]# vcgencmd bootloader_version
Apr 29 2021 17:11:25
Describe the bug
After upgrading from 20211118 (5.10.63) to 20220120 (5.10.92), CPU governors are not available and the Pi is locked to 600 MHz.
Downgrading to 1.20211118 kernel/bootloader restores the functionality.
Steps to reproduce the behaviour
Apt upgrade to raspberrypi-kernel/stable 1:1.20220120-1 and raspberrypi-bootloader/stable 1:1.20220120-1 and reboot
Device (s)
Raspberry Pi 3 Mod. B+
System
Debian 11.2
Jan 20 2022 13:58:39 Copyright (c) 2012 Broadcom version bd88f66f8952d34e4e0613a85c7a6d3da49e13e2 (clean) (release) (start_cd)
Linux DietPi 5.10.92-v8+ #1514 SMP PREEMPT Mon Jan 17 17:39:38 GMT 2022 aarch64 GNU/Linux
Logs
dmesg -l emerg,alert,crit,err produces no errors
Additional context
ls /sys/devices/system/cpu/cpu0: cpu_capacity of_node power regs subsystem topology uevent (no cpufreq)
G_HW_MODEL=3 G_HW_MODEL_NAME='RPi 3 Model B+ (aarch64)' G_HW_ARCH=3 G_HW_ARCH_NAME='aarch64' G_HW_CPUID=0 G_HW_CPU_CORES=4 G_DISTRO=6 G_DISTRO_NAME='bullseye' G_ROOTFS_DEV='/dev/mmcblk0p2' G_HW_UUID='f98c179a-a620-47de-a789-a46af87bf052' G_RASPBIAN=0 G_HW_ONBOARD_WIFI=1 G_HW_REVISION='a020d3' G_HW_PCB_REVISION=3 G_HW_MEMORY_SIZE=1024 G_HW_MANUFACTURER='Sony UK'