Latest kernel RT ARM64 .deb package

[JN19aban]

Describe the bug

Is there a possibility for a latest RT kernel for ARM64 on .deb installation package?

Regards.

Steps to reproduce the behaviour

``

Device (s)

Raspberry Pi CM4

System

``

Logs

No response

Additional context

No response

[kdoren]

@JN19aban

I just posted builds of kernel 6.1.54-rt15 and 6.1.54-llat under "releases".

Please note that the wiki has not yet been updated for installation under Debian bookworm. Bookwork requires that files be copied into /boot/firmware instead of simply /boot.

Please report back your results.

-Kevin

[JN19aban]

@kdoren

Thank you for your response.

I have installed it normally via your instructions.

But the CPU Governor it is still set on demand and not on performance.

The 6.1.54-rt15 is not on the "releases" section only the llat.

For now I use Debian Bullseye.

So far all good except the Cpu Governor.

Regards.

Jim

[kdoren]

@JN19aban

Please see this wiki entry about the CPU governor:

https://github.com/kdoren/linux/wiki/Performance-Governor-settings

Also, I do see 6.1.54-rt15 kernel under releases, maybe there was a delay in github making it visible, or maybe a browser refresh will help.

-Kevin

[kdoren]

FYI I updated the wiki instructions to work with Debian bookworm as well as bullseye. Please report any issues.

[MatthewReed303]

Hi @kdoren do you mind doing a release for stable 6.6.y as I keep get build errors trying to build myself following wiki. Is there a way we can automate a build and release on github say every stable rpi release?

[kdoren]

@MatthewReed303 I posted 64-bit builds for 6.6.35-rt34 for Pi4 and Pi5. Bookworm introduced some issues related to kernel deb package names (must end in -rpi-v8 or -rpi-2712, with no + sign). Also the deb package must have a README file in the /overlays directory, otherwise apt gets upset when generating initramfs. The kernel seems to work fine and did well under a 5-hour latency stress test. I updated the build and installation instructions, but they could use some testing.

[JN19aban]

@kdoren still have issues when go RT like it was on 6.1.y branch? I tried but seem to have an ultra high latency for my application on 6.6.31 and I avoid it again. The default kernel is much faster compare to RT and I do not know the reason yet.... I will try the RT from your build and I will let you know.

[kdoren]

Latency test on RPi5 had the following results, and was done while running a stress test. I don't have a test result for the stock kernel, but in the past testing, a low-latency kernel (which I believe is how the stock kernel is now configured) had slightly faster average latency than PREEMPT_RT. But real-time is not about average latency, it's about worst case, and PREEMPT_RT had better worst-case latency than low-latency kernel.

That being said, this is all workload dependent. You may want to review how your latency-sensitive tasks are configured. PREEMPT_RT does not magically improve performance, it's a tool that can be used to minimize worst-case latency for specific tasks. But that can take quite a bit of testing and experimentation.

[kdoren]

FYI I updated the files in the 6.6.34-rt34 release. The old files conformed to the Raspberry Pi kernel package conventions, which turns out to be bad because the installer will overwrite the stock kernel. This makes it painful to revert. The new files have + in the name, and no README in /overlays, so they will throw a couple of errors during apt install (unsupported kernel version, and initramfs failure. This allows installing in a separate directory under /boot/firmware, and easily reverting or changing kernels in /boot/firmware/config.txt.

[kdoren]

For comparison, here's a latency plot of what I believe is the current official kernel, 6.6.31+rpt-rpi-2712 on Pi5., which is a low-latency (PREEMPT) kernel, while under a stress test. It's much worse. Worst case latency was 1945 uSec vs 139 uSec for PREEMPT_RT. You can't see the falloff point on the graph because I was only binning latencies up to 400 uS.

[MatthewReed303]

Hi @kdoren thanks a lot for that!

Is there away to go back to using the normal overlay directory instead of /o and use the /overlays? I have to install additional drivers and compile them for the touchscreen and they use the overlay directory. Please see here the drivers and issues I have been having.

https://github.com/Chipsee/industrial-pi https://github.com/Chipsee/industrial-pi/tree/6.6.20 https://github.com/Chipsee/industrial-pi/issues/8

[kdoren]

@MatthewReed303 The methodology that I use and recommend installs RT kernel as an additional kernel, without replacing the stock kernel. This allows easy switching between kernels, and easily reverting to stock kernel.

Stock kernel stores its overlays in directory /boot/firmware/overlays; I don't want to touch that, or the kernel .img or initramfs files.

My RT kernel install procedure puts the files in /boot/firmware/$KERN (where i.e. $KERN=6.6.35-rt34-rpi-v8+). RT kernel overlays are in /boot/firmware/$KERN/o (because long overlay filespecs can cause overlay load failure). If you have any overlays to add to RT kernel, you can just copy them into that directory.

Use of the RT kernel is invoked in /boot/firmware.config.txt, by setting [all] above the section for that kernel. Set to [none] to disable that kernel. You can have multiple kernels installed this way. If no extra kernel is invoked with [all], the stock kernel will be used.

This also allows easy removal with apt remove (plus manually cleaning up /boot/firmware/config.txt and removing directory /boot/firmware/$KERN

The first files I posted triggered the install system to replace the stock kernel with RT kernel, by copying the RT files into /boot/overlay and other places. This made it difficult to revert to stock kernel. That's why I replaced the deb packages posted under releases with the '+' versions. These will show some errors during apt install, but that's because it's refusing to overwrite the stock kernel, which is what I want.

[MatthewReed303]

@kdoren okay thanks for the update and that make complete sense. I guess it's only an issue for adding drivers that are currently not in the kernel, such as the Chipsee touch screens I'm using. Would installing the Chipsee drivers first and then installing the RT kernel help or switching to the RT kernel ignore the drivers as they were compiled for the stock RPI kernel?

[kdoren]

@MatthewReed303 You'll need to experiment, but it might not be too bad.

The Chipsee link you reference says in the Q&A to use the goodix overlay. This is included in 6.6 kernel, so you shouldn't have to copy anything, just load it in /boot/firmware/config.txt:

dtoverlay=goodix,interrupt=20,reset=21

They also say to disable modprobe in the Chipsee init script:

sudo sed -i "s/^modprobe gt9xx/# modprobe gt9xx/g" /opt/chipsee/chipsee-init.sh

[MatthewReed303]

Hi @kdoren I managed to get it working by installing the Chipsee drivers first and once they completed I then installed RT kernel and seems to be up and running and touchscreen all working with the desktop.

I have ran into one issue setting up custom boot logo:

sudo plymouth-set-default-theme --rebuild-initrd pix
update-initramfs: Generating /boot/initrd.img-6.6.35-rt34-rpi-v8+
ERROR: Unsupported initramfs version (6.6.35-rt34-rpi-v8+)

How can I fix this error?

I also have added this line in /boot/firmware/cmdline.txt

logo.nologo vt.global_cursor_default=0 quiet splash plymouth.ignore-serial-consoles

But serial console is still showing etc do I need to add it else where using the RT kernel?

EDIT: I had to edit the cmdline.txt in your kernel directory instead and seems to work now

[kdoren]

@MatthewReed303

My interpretation of the error "ERROR: Unsupported initramfs version (6.6.35-rt34-rpi-v8+)" is that it is refusing to overwrite the stock kernel, which IMO is what we want. update-initramfs actually does generate a good initrd.img-$KERN file in /boot. This file is copied to /boot/firmware/$KERN during my installation process.

So I believe this error is desired and can be safely ignored. In fact, I was able to create kernel deb packages that did not generate the error, and I even briefly posted them in Releases. But then I discovered that they overwrote the stock kernel. When I used "apt remove" on the RT kernel, my system failed to boot, and recovery was a bit of a pain. So I replaced the Release files with ones that generate the error.

If you ever do need to re-create the initramfs, you'll need to copy the updated one from /boot/initrd.img-$KERN to /boot/firmware/$KERN/