cpufreq: Implement cpufreq support for Raspberry Pi

[lategoodbye]

In a recent discuss someone requested for cpufreq support.

There are at least two options:

1. implement a bogus clock driver and use cpufreq-dt (OPP version)
2. implement a separate cpufreq-bcm2835 (non-OPP version)

Relevant discussions / patches:

• https://github.com/raspberrypi/linux/issues/1335
• https://github.com/raspberrypi/linux/issues/2061
• https://patchwork.kernel.org/patch/7369611/ (Upstream - non-OPP version)
• https://github.com/raspberrypi/linux/commit/e07e4786d6d032cd3d03a276010cf5cd738bafdc (Downstream)
• http://lists.infradead.org/pipermail/linux-rpi-kernel/2014-May/000913.html
• https://src.fedoraproject.org/rpms/kernel/raw/master/f/bcm2835-cpufreq-add-CPU-frequency-control-driver.patch (Upstream - OPP version)

Resulting TODOs:

• implement clock change notification handler (sdhost, i2c, aux, spi, dpi) https://github.com/torvalds/linux/commit/90ad2cbe88c22d0215225ab9594eeead0eb24fde.patch https://github.com/torvalds/linux/commit/c4b0510cc1571ff44e1d6024d92683d49a8bcfde.patch

[sergey-suloev]

The following patch works well , but I have never seen cpu at 900 yet (600 & 1200 only). https://github.com/orpaltech/armlinux/blob/master/patch/kernel/mainline/v4.19-rc2/rpi/0037-bcm2835-cpufreq-Add-CPU-frequency-control-driver.patch

[lategoodbye]

Unfortunately me too, even after implementing recalc_rate with RPI_FIRMWARE_GET_CLOCK_MEASURED.

[nullr0ute]

By just catting /sys/devices/system/cpu/cpufreq/policy0/scaling_cur_freq in brief testing I've seen it touch on 900 on it's way between 600/1200.

The patch looks to be just for the original RPi3, is there just more DT bits required for the 2/3+ (and presumably the various 0/1 devices)?

[nullr0ute]

[ 66.251546] cpu cpu0: dev_pm_opp_set_rate: failed to find current OPP for freq 1400000000 (-34)

[nullr0ute]

On the 3+ [ 66.251546] cpu cpu0: dev_pm_opp_set_rate: failed to find current OPP for freq 1400000000 (-34)

[lategoodbye]

This isn't a topic which isn't fully matured yet. So i'm not interested in pushing a quick shot for 4.19.

Currently i'm not sure which approach is better. The DT has the disadvantage that we need to model the OPPs on board level instead of SoC. That's why i started with RPI 3.

[nullr0ute]

I realise, I'm just playing to see if it's something that would be usable in the interim across the devices I need to support.

0001-add-1.4-ghz-OPP-for-the-3B.patch.txt

So this appears to work on the 3B+

cat /sys/devices/system/cpu/cpufreq/policy0/scaling_available

600000 900000 1200000 1400000

No idea if the 3B+ supports 1.2 as well as 1.4 though. I suppose similar would work for the other devices

[lategoodbye]

AFAIK firmware supports arbitrary values. So this should be okay.

But this was only the easy part, because all driver connected to the VPU clock must handle clock changes now.

[nullr0ute]

I have no doubts it was the easy part, is there details of which drivers are affected?

The best I've managed to find on the freqs supported seems to be this section in the wikipedia docs: https://en.wikipedia.org/wiki/Raspberry_Pi#Overclocking

[sergey-suloev]

If every driver needed to take CPU clock change into account that would be a hell. I am not sure how it is implemented in Rpi but normally it should be handled in only one driver, like ccu. Correct me if I am wrong.

[sergey-suloev]

@lategoodbye And what about voltages ? Usually OPP table contains freq + voltage, does Rpi need voltage change ?

[lategoodbye]

Please look at all driver which uses BCM2835_CLOCK_VPU for the notification changes: https://elixir.bootlin.com/linux/latest/source/arch/arm/boot/dts/bcm283x.dtsi

The VC4 firmware takes care of the voltages.

[sergey-suloev]

@lategoodbye Is there a way to change(or create) only one , let's cal it "hub driver", and what would it cost ? Maybe it is [compatible = "brcm,bcm2835-cprman"] ?

[lategoodbye]

Sorry, i don't understand question / intension?

[sergey-suloev]

@lategoodbye my point is that changing every driver in order to integrate OPP table doesn't make too much sense. In other words, it is stupid. My question : can you imagine any way that can allow make such changes in only one, "hub", or managing driver, i.e. ccu. I may be missing something, just want to understand. By "cost" I mean how much development effort it would take.

[lategoodbye]

I think there is a misunderstanding. You don't need to implement OPP table handling in every driver. But a lot of driver uses the VPU as clock input with its clock divider. So in case the VPU clock changes the clock divider should be adjusted, too. Otherwise the resulting frequency is wrong. Usually the divider settings are claimed by the relevant driver, so in case of a VPU clock change is makes sense to handle it by the relevant driver.

[nullr0ute]

@lategoodbye so that was my understanding around the drivers understanding that a clock would possibly change under them as opposed to being static. My query was is can we grep the DT or a string in drivers for which upstream drivers might be affected to work out what might be affected?

[lategoodbye]

As i wrote above grep for BCM2835_CLOCK_VPU within the DT.

[sergey-suloev]

@lategoodbye That's what I am talking about too. Can we have a special driver that will provide stable clocks for those drivers no matter if CPU clock changes or not?

[nullr0ute]

So having been running this for a little bit in Fedora one thing to note here when running something that I've noticed when running something that taxes the gpu too such as gnome on wayland we get shutdowns/lockups/poweroffs. I need to investigate more and get more debug.

The end driver might want to tie into the thermal framework and possibly the hwmon for undervoltage and use those for scaling decisions too.

[lategoodbye]

@nullr0ute Do you mean Sergey's patch? Which governor did you use?

[nullr0ute]

Yes, Sergey's patch, and the ondemand (Fedora default) govenor

[sergey-suloev]

@nullr0ute Can it be due to your power supply limits ? I personally haven't tried the patch with GUI tasks but I haven't had any issues with some heavy tasks in bash.

[nullr0ute]

@sergey-suloev I don't believe so, it's a good quality Anker PSU, I will be investigating more this week. I think it's more thermal than power TBH but I wasn't remotely connected in to see any logs at the time. The RPi3 doesn't have a heatsink on it.

[nullr0ute]

For reference with more testing, it's thermal, I get the following: kernel: thermal thermal_zone0: critical temperature reached (80 C), shutting down

[lategoodbye]

I want to point out to this commit 0fe4d2181cc4cb3eba303c0e03f878d2558d0f3a

In the lack of a cooling device this was left empty. But with a cpufreq driver we have such a cooling device.

[sergey-suloev]

@lategoodbye could you provide the code that should be added, please ?

[lategoodbye]

It's not code, it's device tree settings. Yes, later

[lategoodbye]

Based on Peter's cpufreq patch i made a patch to use CPU0 of RPI 3 as cooling device.

Disclaimer: Only compile tested

[nullr0ute]

I'll give it a test run given I have a RPi I can regularly thermally trip :)

[sergey-suloev]

@lategoodbye thanks a lot

[nullr0ute]

Initial feedback, the cooling device shows up tmon and P trip point is there. Will test the actual trips shortly

[nullr0ute]

BIG :+1: It seems to work pretty well. I've not managed to have my original RPi3 shut down due to thermal. Remotely running tmon and following the current freq in /sys/devices/system/cpu/cpufreq/policy0/ I've not managed to trip thermal in basic testing. It has got above the 70 P but managed to keep it well below Critical 80 and seems to work pretty well :+1:

[lategoodbye]

Thanks for testing. Maybe we should add a hysteresis of 2 degree.

[sergey-suloev]

@lategoodbye Stefan, is it possible to make Rpi3 work on 900MHz ?

[nullr0ute]

@sergey-suloev in my testing I see it running at 900mhz on occasion, it moves between 600/900/1200

[sergey-suloev]

@nullr0ute I can't make it work on 900 when I set manually with userspace gov. The real freq is always 1200.

[lategoodbye]

I think the 900 MHz which Peter is seeing is only internal in the cpufreq driver.

Edit: So i'm afraid that there is only minimum and maximum possible.

[nullr0ute]

I'm looking in /sys/devices/system/cpu/cpufreq/policy0/ is there other places that are more accurate?

[sergey-suloev]

@nullr0ute can you make it 900 manually ?

[lategoodbye]

@nullr0ute AFAIK there is no chance to fetch actual core frequency via mailbox.

The more accurate way should be: vcgencmd measure_clock arm

[sergey-suloev]

no matter how you fetch it, on my Rpi3 it is always 1200 after I set it manually to 900

[nullr0ute]

@lategoodbye unfortunately (last I looked at least) that tool isn't open source and doesn't work on aarch64 either

[sergey-suloev]

@nullr0ute it is open source and called userland https://github.com/raspberrypi/userland

but there are definitely some issues with aarch64

[lategoodbye]

@nullr0ute Recently i tested Fedora Workstation 29 Beta 1.5 on my RPi 3 B+. The whole system periodically freeze completely (no USB power on keyboard, no reaction on screen). After blacklisting cpufreq_dt the system becomes usable (including debug UART). Just a warning cherry-picking just this cpufreq patch is very dangerous and can leads to data loss e.g. on sdhost interface.

[ggardet]

What is the current status? Is there any upstream effort on going?

[lategoodbye]

Currently not. The first step would be to adapt the relevant driver (sdhost, i2c, aux, spi, dpi) for dynamic clock changes.

[lategoodbye]

Okay, here is a untested patch to dynamically adjust AUX UART divisor: https://gist.github.com/lategoodbye/5f4eea59e82c310519adf76dc095fe2e

Be aware this currently wont work with the implementation of clk-bcm2835, because it doesn't take VPU clock changes / Turbo mode into account.

[nullr0ute]

I wonder if the OPP framework is one way to achieve this, seems this series for qcom is similar to what needs to be achieved.

https://lists.freedesktop.org/archives/dri-devel/2019-March/211321.html

[lategoodbye]

First RFC series by Nicolas Saenz Julienne: http://lists.infradead.org/pipermail/linux-rpi-kernel/2019-April/008634.html