Linux 5.5 and 5.6 changes to backport to Ubuntu 20.04 based on 5.4 for best experience

[bjf]

The Ubuntu 20.04 release will be based on the upstream Linux 5.4 kernel. We want to make the experience for Raspberry Pi users to be as positive as possible. Therefore we are interested in any Raspberry Pi commits going into the 5.5 and 5.6 Linux kernels which we will consider backporting to our 20.04 Ubuntu kernel.

[pelwell]

@vianpl, the new BCM2835 maintainer, is currently building on the work started by @lategoodbye, upstreaming more of the Pi4 downstream patches. In addition to things already in 5.5 this includes:

1. The brcmstb PCIe driver.
2. VCHIQ support for >1GB RAM.
3. 40-bit DMA support.
4. The thermal and HWRNG drivers.
5. A quirk to enable power saving in the VL805.
6. Probably a few others I've forgotten.

[satmandu]

It would also be nice to have the V4L2 patches being worked on for upstreaming in the ubuntu kernel, but I'm biased since my RPI/ubuntu based babycam monitor streams h264...

[juergh]

It would also be nice to have the V4L2 patches being worked on for upstreaming in the ubuntu kernel, but I'm biased since my RPI/ubuntu based babycam monitor streams h264...

We have these commits in the Focal linux-raspi2-5.4 kernel.

[6by9]

https://github.com/6by9/linux/tree/staging_next_upstreaming is the cleaner branch for those on top of staging-next, as vc04_services have moved on slightly since 5.4. We (possibly via others turning the handle) are looking at upstreaming those to mainline soon.

[satmandu]

Also another request for getting zfs modules into the official ubuntu rpi kernel modules package please as per https://bugs.launchpad.net/ubuntu/+source/linux-raspi2/+bug/1852119. Getting the official zfs-dkms precompiled saves a lot of time for installs on rpi arm64 builds.

[JamesH65]

Isn't that entirely down to Ubuntu/Canonical, or am I misunderstanding what you mean? There is no official Ubuntu rpi kernel, just our own kernel.

[pelwell]

This issue was opened by @bjf (a Canonical employee) with my encouragement as a way to get some input from the Pi maintainers and users. As such I think @satmandu's request is reasonable.

[JamesH65]

Ah, ok, cool. As you were.

[juergh]

@satmandu Have you actively used ZFS on the Rpi 4? We (Ubuntu) don't really recommend it on 'slow' devices and 4GB is the absolute minimum from our experience.

[JamesH65]

Worth a read https://en.wikipedia.org/wiki/ZFS#Inappropriately_specified_systems

[satmandu]

@juergh Yes. I currently have two rpi4 devices booted using zfs rpools on ubuntu/arm64. They work great as a zfs receive capable backup NAS on the LAN and as a dump1090 machine connected via wifi. The former let me decommission my old Synology backup NAS. (They are both also running as kubernetes slaves for use with distcc, and I have swap off on both.)

Machine 1: (rpool on external usb ssd) System information as of Wed Mar 4 16:14:04 EST 2020

System load: 0.89 Usage of /home: unknown Memory usage: 30% Swap usage: 0% Temperature: 53.1 C Processes: 329 Users logged in: 1

(uname -a ; cat /etc/issue ; cat /proc/cpuinfo ; zpool list ; free -m ) > info.txt ; cat info.txt| grep -v Serial | pastebinit https://paste.ubuntu.com/p/zMs2W4w3wS/

Machine 2:(rpool formerly a zpool mirror on sdcard & external usb flash drive, now just sdcard) System information as of Wed Mar 4 16:13:11 EST 2020

System load: 0.49 Usage of /home: unknown Memory usage: 33% Swap usage: 0% Temperature: 40.4 C Processes: 474 Users logged in: 0 (uname -a ; cat /etc/issue ; cat /proc/cpuinfo ; zpool list ; free -m ) > info.txt ; cat info.txt| grep -v Serial | pastebinit https://paste.ubuntu.com/p/SdysxnD9JQ/

[shumvgolove]

@pelwell Hello. I am using another distro that uses 5.4 LTS kernel - Alpine. Linux 5.4.12 basically just worked™, but with recent versions (5.4.24) USB ports stopped working. From my searching, I've found that this commit was reverted. Can you explain your reasons and possibly return the commit back? Thanks in advance.

[pelwell]

As of this afternoon, rpi-5.4.y is using the upstream PCIe driver back-ported from 5.6. All previous pcie-brcmstb commits have been reverted. If you are saying it's not working on ARMv8 then I'll investigate.

[shumvgolove]

I am using aarch64 and latest kernel that Alpine have built (9th March). I'll wait for new kernel update and report back if I have any trouble. Thank you!

[pelwell]

On 64-bit rpi-5.4 only (5.5 and 5.6 are OK) the CMA region is being created too high, causing havoc. It shouldn't be too difficult to solve.

[lategoodbye]

Just guessing: ARM: dts: bcm2711: force CMA into first GB of memory ARM: dts: bcm283x: Unify CMA configuration dma-contiguous: CMA: give precedence to cmdline

[vianpl]

@shumvgolove You can you add cma=64M@0x0-0x40000000 to your kernel command line. It'll probably fix your issue.

@pelwell I don't recall when all the DMA fixes where upstreamed, but we most probably need to back port device-tree fixes (as @lategoodbye pointed out). Also, probably in arch/arm64 and dma-direct. I can have a look at it if you want.

[pelwell]

Thanks, but I'm on it - if we revert the upstream DT patches then we get to fix the breakages. My post was meant to be an updated rather than a cry for help.

[pelwell]

It's a merge/commit-order issue. For a while now we've been using 47fec0dcefa53f44985e95785021447d646daa2b to get us a <1GB DMA zone. 1a8e1cef7603e218339ac63cb3178b25554524e5 made that unnecessary, but then bff3b04460a80f425442fe8e5c6ee8c3ebef611f once more required it (or something like it) for the case where we're not using DT to configure CMA (that's a change I'm not ready to make yet). The 5.5 and 5.6 branches still have the original patch on top of @vianpl's work, which is why they've been working for us. The commit order on 5.4 undid its change, hence the breakage.

A fix is simple, but I first want to reapply those patches as cleanly as possible.

[ghost]

@satmandu Have you actively used ZFS on the Rpi 4? We (Ubuntu) don't really recommend it on 'slow' devices and 4GB is the absolute minimum from our experience.

I'm using ZFS successfully on a 4GB Pi, but only for a non-boot device. It works well, and I've found it reliable, although I have not stress tested it. The obvious advice about leaving dedup switched off applies.

Also note that ZFS cannot be used for swap due to the risk of deadlocks - it appears this wasn't fixed in upstream (i.e. Oracle) ZFS before it went closed source, since it also happens on Illumos.

Finally, including any CDDL-licensed module binaries which come from Sun/Oracle code in this, the official Raspberry Pi kernel repo, opens up Raspberry Pi to the possibility of being sued by Oracle. While this chance may be slim, it would need to be considered by Raspberry Pi management, I would suggest.

[shumvgolove]

@vianpl Thanks for suggestion, I'll try this when I get back home. Can you explain what does it do exactly?

btw for clarification I have rpi4 4Gb variant, aarch64 arch, 5.4.24 Linux.

[satmandu]

@andrum99 This discussion is about backporting changes to the ubuntu kernel (and/or to the RPI supporting kernel packages on ubuntu). Nobody is suggesting adding zfs or CDDL modules to the RPI kernel repository.

As it is the RPI kernels have configuration options set which don't allow arm64/zfs to be installed as a module due to conflicts with the GPL-only symbol 'preempt_schedule_notrace' due to the following enabled options CONFIG_PREEMPT and PREEMPT_RT_FULL. Unsetting those and enabling CONFIG_PREEMPT_VOLUNTARY allows zfs to be used just fine though.

[ghost]

@andrum99 This discussion is about backporting changes to the ubuntu kernel (and/or to the RPI supporting kernel packages on ubuntu). Nobody is suggesting adding zfs or CDDL modules to the RPI kernel repository.

This is the official Raspberry Pi kernel repository. You have suggested adding ZFS kernel modules to the Ubuntu kernel build. I am aware this is not the same as suggesting this be added to the official Raspberry Pi kernel builds, however I thought it was important to mention this since, like I said, this is the official Raspberry Pi kernel repo, which also contains built kernel binaries.

As it is the RPI kernels have configuration options set which don't allow arm64/zfs to be installed as a module due to conflicts with the GPL-only symbol 'preempt_schedule_notrace' due to the following enabled options CONFIG_PREEMPT and PREEMPT_RT_FULL. Unsetting those and enabling CONFIG_PREEMPT_VOLUNTARY allows zfs to be used just fine though.

That's unfortunate. I'm running ZFS on the official Raspberry Pi kernel v 4.19.y (currently 4.19.97-v8+), so if 5.4 is going to break ZFS I would need to build my own kernel, which I would rather not do. The whole point of my running the official Raspberry Pi distro and kernel is that it has had the most testing. I guess I need to have a look at Raspberry Pi's 'next' kernel (5.4.y), when it arrives.

[pelwell]

The final change (after some tidying) was removing https://github.com/raspberrypi/linux/commit/bff3b04460a80f425442fe8e5c6ee8c3ebef611f from rpi-5.4.y for now. rpi-5.5.y and rpi-5.6.y will be updated similarly.

[pelwell]

It's all done now - let me know if you see any issues.

[shumvgolove]

Thank you!

[shumvgolove]

Okay, so Alpine had built the linux-5.4.25 kernel today at 11:26:16 and that doesn't fix my issue nor @vianpl suggestion.

Once again my problem is - I have rpi4 4 gb version and my USB ports doesn't work - #3093 is related issue. For workaround this I can set total_mem=3072 in config.txt, but that's shrinks my available memory to 3gb. There was pull request #3144 that basically fix everything but it was reverted. Am I missing something?

[pelwell]

You're missing the fact that #3144 is a patch to a version of the driver that is no longer used. Any fix to your problem is not going to be through us reverting to the downstream driver - we need to move forward.

[macmpi]

@shumvgolove rather than relying on build date/time, you may need to check rpi-5.4.25-alpine.patch if it contains @pelwell 's recent cleanup

[shumvgolove]

@macmpi I checked the Alpine patch, but it does not contain today's changes unfortunately.

@pelwell Thanks for your explanation and sorry for stupid questions.

[macmpi]

You'll have to wait for a next Alpine linux-rpi release then, unless @ncopa considers this USB regression for rpi4 4 gb justifies a 5.4.25 re-build with recent changes in.

[shumvgolove]

@pelwell So Alpine had rebuilt the kernel with recent changes, but unfortunately it doesn't fix USB issues. Should I fill out new issue?

[pelwell]

Open an issue with the Alpine devs initially - I'd be very surprised if none of them have been using USB on a Pi 4.

[shumvgolove]

I've opened the Alpine issue week ago - here. @ncopa also reported that his apple magic keyboard with USB cable doesn't work.

[ncopa]

The dell keyboard works so USB host controller is not broken.

[shumvgolove]

Yes, I should have mentioned that.

I have this configuration: config.txt and cmdline.txt. Upon plugging keyboard dmesg doesn't print any new lines - dmesg.log. Can I provide anything else?

[pelwell]

Do you have any other USB devices, and have you tested them?

[shumvgolove]

TLDR: I used the old firmware that came with the Alpine 3.11.3 release.

After searching, I found that Alpine updates the firmware in /boot/dtbs-rpi4/ when the firmware should be in /boot/. After copying the content to boot, rpi4 works as expected. Thanks for your work @pelwell and sorry for wasting your time.

[pwuertz]

@bjf, @juergh. You were saying that "We have these commits in the Focal", but as reported in Ubuntu bug 1873757 and Ubuntu bug 1876259, the Ubuntu Kernel is still affected by the notorious no-USB-on-64bit-4GB bug and is missing the bcm codec module. Would you mind giving us some info on this?

[juergh]

@pwuertz The no-USB issue is because the reporter is using an old image which contained the wrong DTBs. That has been fixed a while ago. Adding the bcm codec module is on my list of things to do.

[satmandu]

FYI zfs has now officially been added to the default raspberry pi Ubuntu 20.04/Focal kernel: https://bugs.launchpad.net/ubuntu/+source/linux-raspi/+bug/1884110/comments/1