Boot Issue with fbtft enabled arm64 rpi-5.6.y kernels for post feb 12 firmware on ubuntu 3b+

[satmandu]

I have an ubuntu/arm64 setup based off of focal 20.04 running on a 3b+. I am loading the kernel vmlinuz straight from config.gz without using uboot. The kernel I compiled off of rpi-5.6.y, specifically g5e352b6a4. I am using the stock fbtft module to enable a waveshare35a clone display.

dtb for the display is here: https://github.com/swkim01/waveshare-dtoverlays/blob/master/waveshare35a.dtbo

Kernel config for this kernel: https://paste.ubuntu.com/p/g3MGfDGmyW/

(I have this kernel booting on a 3b,

Describe the bug When I upate to Feb 20 firmware, I am no longer able to boot, with boot failing around lan initialization. Reverting to Feb 12 firmware allows boot to continue.

To reproduce

I'm using a simple script to update firmware. download_firmware.sh:

#!/bin/bash
hash=${1}
urlbase="https://github.com/raspberrypi/firmware/raw"
for f in bootcode.bin fixup.dat fixup4.dat fixup4cd.dat fixup4db.dat fixup_cd.dat fixup_db.dat fixup_x.dat start.elf start4.elf start4cd.elf start4db.elf start_cd.elf start_db.elf start_x.elf
do
    echo "downloading $f"
    sudo rm -f $f
        sudo curl -OLf ${urlbase}/${hash}/boot/${f}
done
echo $hash | sudo tee rpi_git_hash

config.txt:

[pi3]
#kernel=uboot_rpi_3.bin
kernel=vmlinuz

[all]
arm_64bit=1
device_tree_address=0x03000000
max_framebuffers=2
framebuffer_priority=2

# The following settings are "defaults" expected to be overridden by the
# included configuration. The only reason they are included is, again, to
# support old firmwares which don't understand the "include" command.

enable_uart=1
cmdline=nobtcmd.txt

dtparam=i2c_arm=on
dtparam=spi=on
initramfs initrd.img followkernel
dtparam=spi=on
dtoverlay=pi3-disable-bt
dtoverlay=vc4-kms-v3d,cma-256
gpu_mem=128
start_x=1
enable_uart=1

nobtcmd.txt:

net.ifnames=0 dwc_otg.lpm_enable=0 console=tty1 root=LABEL=writable rootfstype=ext4 rootwait fsck.mode=force fsck.repair=yes fbcon=map:10 fbcon=font:ProFont6x11 logo.nologo nosplash overlayroot=disabled

To reproduce I go to /boot/firmware (where boot is on ubuntu). working firmware: (Feb 12) cd /boot/firmware ; download_firmware.sh f4b58692fef0b9c16bd4564edb980fff73a758b3 ; sudo reboot not working firmware: (Feb 20) cd /boot/firmware ; download_firmware.sh 02bff4e75712094ffb1ab2ec58a8115ca3e52290 ; sudo reboot

System Copy and paste the results of the raspinfo command in to this section. Alternatively, copy and paste a pastebin link, or add answers to the following questions:

• Pi3B+
• Ubuntu focal (but modified to load kernel directly via the rpi bootloader)
• Which firmware version (vcgencmd version)?

  sudo vcgencmd version
  Feb 12 2020 12:38:08 
  Copyright (c) 2012 Broadcom
  version 53a54c770c493957d99bf49762dfabc4eee00e45 (clean) (release) (start)

• Which kernel version (uname -a)?

  Linux bridge 5.6.0-rc7-g5e352b6a4-raspi2 #1 SMP Fri Mar 27 10:29:57 EDT 2020 aarch64 aarch64 aarch64 GNU/Linux

  Logs dmesg: https://paste.ubuntu.com/p/sZ2NRvBw2Y/

(I noticed this when I was testing the common firmware branch/next firmware and realized I wasn't able to boot.)

[pelwell]

Does it boot without the waveshare fbtft overlay? What happens if you run your kernel on Raspbian image? Try to reduce the requirements to the simplest possible.

Do you really need to load the DTB to an explicit address?

[satmandu]

Does not boot without the waveshare fbtft overlay. No change with dtb loading to explicit address disabled. (This was an artifact of previously booting using u-boot, I think.)

Next step will be to try with a raspbian image...

[satmandu]

Tried a raspbian image, and no problem there....

I noticed the raspbian image isn't using a initrd image.

And that's the crux of the matter. When I get my ubuntu image to stop having config.txt use initramfs initrd.img followkernel with current firmware (start. and fixup.) it loads just fine.

Even using the current 5.4.29-v8+ kernel8.img I can't get it to boot without disabling the initrd.

(Also I can't get wifi to work with the 5.4.29-v8+ kernel on my ubuntu system, but that's another issue.) Now showing this:

sudo vcgencmd version
Apr  7 2020 18:50:57 
Copyright (c) 2012 Broadcom
version 6ab498ee82102a9669a1909ae90341e3ecc2128f (clean) (release) (start)

[satmandu]

(This is going to be a problem since several of my rpi4b devices have zfs rpools and need the initramfs to load the zfs modules.)

[pelwell]

The only difference in initramfs handling between those two firmwares is that the length of the initramfs file gets rounded up to a multiple of 4 for alignment reasons - it's a very small change.

In your original report you link to a dmesg log - is that for the failure case? If so, can I get a log from it booting succesfully with the older firmware?

What would speed up the investigation is an image of an SD card I can test with. Perhaps you can squeeze one onto a 4GB or 8GB card to keep the upload manageable.

[satmandu]

This may be able to replicate the issue on a fresh raspbian image.

TL;DR: initramfs isn't loaded if cmdline.txt has root=LABEL=rootfs in it.

sudo BRANCH=next rpi-update
sudo reboot

I'm not sure if the PARTUUID is the same for all raspbian sd card images, or if it is generated, but this worked on my card's cmdline.txt fie.

sudo sed -i 's/root=PARTUUID=738a4d67-02/root=LABEL=rootfs/g' /boot/cmdline.txt
sudo update-initramfs -c -k `uname -r`
sudo cp /boot/initrd.img-`uname -r` /boot/initrd.img
echo -e "arm_64bit=1\ninitrd.img followkernel" | sudo tee -a /boot/config.txt
sudo reboot

[pelwell]

It worked on my not-clean Raspbian image, except:

• I renamed the old root= to xroot= rather than deleting it,
• I manually added the arm_64bit=1 before the first reboot, otherwise you end up building a 32-bit initrd for a 64-bit kernel, and
• I included "initramfs" before "initrd.img followkernel", otherwise it is ignored.

[satmandu]

Oops on the initramfs portion of config.txt!

I'm not sure where that change happened in my config. Weirdly everything else is suddenly working. (and even the raspbian image when I added initrd=initrd.img to the cmdline.txt )

I think it is fair to say that this was entirely a screwup in my own configuration somewhere, as the current next firmware is loading just fine on multiple 3b+ devices. (testing 4b now...)

Sorry for the bother.

[pelwell]

Well, if you do convince yourself that there is a real problem there, you know where to find us...