BPi-R4: NVMe boot does not work when booting from NAND/EMMC

[BtbN]

I've got the system booting fine when u-boot is installed on an SD card. But the moment I try installing it into the NAND or EMMC, the initial "pci enum" does show the NVMe disk. But after that, no nvme device is detected, and another call to "pci enum" does not see it anymore. This all works fine when u-boot is installed to an SD Card.

I noticed that NVMe support is disabled in the nand defconfig. So I guess this is a known issue? Should it work when flashed to emmc?

[frank-w]

Have it only basicly tested with sd...problem on nand/nor is no easy way to store which file should be loaded...we have no uEnv.txt there.

But i can enable the driver of course,but we need a way to easily store filename for booting from nvme...maybe i use default bpi-rX.itb

On emmc i can use same as sd...via bpi-boot partition

have added it...

[BtbN]

Hm, the custom bootcmd I wrote into my uEnv_r4.txt is being executed just fine from nand, so I don't think that's the problem.

Nvme/pcie seems to be failing at a more fundamental level there. And the same happens when booting from emmc as well. nvme scan does not find the device, and after that it seems to be entirely gone from the bus. If I then manually boot my os by putting the Kernel on a USB stick, it uses the nvme SSD just fine, so it's not a hardware level issue.

On June 29, 2024 9:09:13 AM GMT+02:00, Frank Wunderlich @.***> wrote:

Have it only basicly tested with sd...problem on nand is where to store which file should be loaded...we have no uEnv.txt there

-- Reply to this email directly or view it on GitHub: https://github.com/frank-w/u-boot/issues/17#issuecomment-2198021880 You are receiving this because you authored the thread.

Message ID: @.***>

[frank-w]

Have you tested with the uboot build after you open this issue? Have added the missing options...

I do not mean the builtin environment from uEnv_r4.txt,i mean the override placed on BPI-BOOT partition only works on mmc as nand/nor does not have such partitions...i know there is envoffset and device,but this is not that easy to handle outside of uboot

[BtbN]

Ah, I'll give it a try and report back! Thanks

[BtbN]

That's pretty much the exact changes I had done myself already, since I copied those options from the sd config to add pci/nvme support, and I copied the pstore ones while at it.

On an unrelated note, the r4 emmc defconfig now produces these warnings:

generated_defconfig:63:warning: override: reassigning to symbol SPI_FLASH_WINBOND
generated_defconfig:66:warning: override: reassigning to symbol SPI_FLASH_MTD

Since the options are already defined previously.

I'll still give it a try again, but the config is virtually identical, just with maybe the order of options slightly shuffled.

[frank-w]

Ah ok,did not know you already tried that.

But when options are same it is strange that it does not work... currently have no idea

I only noticed that r4 nvme does not work when pcie2 is enabled...but dts should be same so far except mmc

[BtbN]

Could the difference be pcie2 being disabled in the dts for the -sd variant, but not the non-sd one? I vaguely recall reading something about pcie2 interacting weirdly with NVMe:

https://github.com/frank-w/u-boot/blob/2024-04-bpi/arch/arm/dts/mt7988-sd-rfb.dts#L145 vs. https://github.com/frank-w/u-boot/blob/2024-04-bpi/arch/arm/dts/mt7988-rfb.dts#L194

?

[frank-w]

Oh,yes...this is probably the cause....if pcie is not working with pcie2 enabled...thought emmc dts is done similar to mt7986 where base dts is sourced and only mmc node is changed

https://github.com/frank-w/u-boot/blob/2024-04-bpi/arch/arm/dts/mt7986a-bpi-r3-emmc.dts

Documented r4 pcie2 issue here

https://github.com/frank-w/u-boot/commit/29e7285fa1864c610afd4ed3cc3276a0bc62052a

But forgot the non-sd dts...sorry,changed now

[BtbN]

Yep, working now. The only issue I ran into is the lack of documentation on how to flash it to the NAND, but I ended up getting it to work in the end.

The BL2 image goes straight to 0x0 of the NAND. Linux using the 6.9-main kernel with its bpi-r4 defconfig has that mapped as /dev/mtd0. But I could not get Linux to write there whatsoever. (By now I found https://github.com/jclehner/mtd-rw which would probably enable it, but too late now, bl2 needs rarely ever re-flashed). So in the end I just ended up flashing it from u-boot:

ext4load nvme 0:3 $loadaddr /root/bpi-r4_spim-nand_bl2.img
ext4size nvme 0:3 /root/bpi-r4_spim-nand_bl2.img
mtd write spi-nand0 $loadaddr 0x0 $filesize

If I managed to unlock the mtd device, this wouldn't have been much more than dd if=/root/bpi-r4_spim-nand_bl2.img of=/dev/mtd0

Now the fip needs to be put in a place where bl2 finds it. There seem to be two ways of achieving this: Just plain writing it to a magic address in the NAND, or creating an UBI on it, which starts at its own magic address.

By poking in the source, I found the following two addresses:

For raw nand writing it's 0x580000: https://github.com/frank-w/u-boot/blob/mtk-atf/plat/mediatek/mt7988/bl2/bl2_dev_spi_nand.c#L11

For the UBI approach, the address is 0x200000: https://github.com/frank-w/u-boot/blob/mtk-atf/plat/mediatek/apsoc_common/bl2/bl2_boot_nand_ubi.c#L23 And then it'll be looking for a static volume named fip: https://github.com/frank-w/u-boot/blob/mtk-atf/plat/mediatek/apsoc_common/bl2/bl2_boot_nand_ubi.c#L68

To just dump the raw fib file into the NAND, the same method as for BL2 could be employed, just with the different file and offset.

I opted however for the UBI approach, since UBI should be able to deal with bad NAND blocks, whereas a fixed address in NAND is not. Conveniently, the default config of the 6.9.0-bpi-r4-main kernel already comes with the NAND pre-partitioned into bl2 starting at 0x0 and ubi starting at the correct offset 0x200000. The ubi partition is also not write protected.

First of all, this requires adding the UBI=1 makeflag to build.sh in the mtk-atf branch, otherwise it's only looking at the other magic offset.

To then create the actual ubi from Linux:

ubiformat /dev/mtd1
ubiattach -p /dev/mtd1
ubimkvol /dev/ubi0 -N fip -s 4MiB -t static
ubiupdatevol /dev/ubi0_0 /root/bpi-r4_spim-nand_fip.bin

The -t static is important. At first I omitted it, and BL2 could not read the volume. The commands might differ slightly if there already is an ubi on the NAND, which it might be if openwrt got its hand on the NAND at some point.

[frank-w]

Have not done anything with ubi yet

But here are my steps for nand...thought i added a wrnand function like for r3

https://wiki.fw-web.de/doku.php?id=en:bpi-r4:uboot

[frank-w]

So you got ubi working?

If i pass UBI=1 is it possible to load fip from fixed offset (like i currently do) so to support both? If i understand above it is not,so in my pipeline i have to build both,right? But based on your description my mtk-atf is already capable of ubi fip

More interesting will be adding kernel in the nand :) something with initrd...so about 25mb

[BtbN]

Yes, UBI is working fine for me. The fip is identical and does not need changed, just written to a static ubi volume named fip instead of straight into the NAND.

If I understand the Makefiles right, the "load from fixed offset" and "load from UBI" are mutually exclusive. Either the code for one or the other gets compiled in. Probably not impossible to re-program it to "check for UBI first, if that fails try the fixed offset", but would need to be done first obviously.

The way my board is booting now is via this bootcmd in the built in uEnv.txt:

bootcmd=pci enum; nvme scan; nvme info; if fatload nvme 0:1 $loadaddr boot.scr; then source $loadaddr; fi

So I just have a small ESP-like fat32 partition at the beginning of my NVMe SSD, which has a boot.scr script sitting there, which configures the actual boot.

Booting a kernel from nand via ubi should be straight forward, just dump the itb straight onto a static volume, ubi read it to the loadaddr, and bootm it.

[frank-w]

Need to find time to test ubi...but bl2 needs to be built additionally for ubi

Btw. Isn't filesize set with ext4load? It is for fat.

How does the boot.scr looks like? Afaik scripts have to be compiled with mkimage...thats why i used the uEnv.txt way

[frank-w]

added ubi support to r4 bl2 via second build...may i ask you to try this (https://github.com/frank-w/u-boot/releases/download/CI-BUILD-2024-04-bpi-2024.04-2024-06-30_1033/bpi-r4_spim-nand_ubi_bl2.img)?

[BtbN]

I have written down all the steps I ended up doing to get my BPI working here, including the boot.scr: https://gist.github.com/BtbN/9e5878d83816fb49d51d1f76c42d7945#boot-method

Edit: Will give it a test-run later tonight

[BtbN]

Can confirm the ubi image from the release works fine and chain-loads to my own fip file on the ubi partition.

I noticed that all the non-sd builds of u-boot print this message while interacting with the mtd command: jedec_spi_nor spi_nor@0: unrecognized JEDEC id bytes: ff, ff, ff

To my knowledge, the BPI-R4 has no NOR, so it somehow snuck into the mt7988-rfb.dts file?

[frank-w]

thought it is a relict from R3 uEnv.txt ;) but seems i have it dropped already

so maybe it is from dts which is not bpi-r4 specific one atm...it is for RFB which may have a nor

but i'm working with some others on OF_UPSTREAM conversion so BPI-R4 gets its own DTS soon (linux dts is only very basic and needs many overrides).

[frank-w]

Mhm, defconfig should be r4 specific and so i could drop the nor options if there...

But i do not see them... https://github.com/frank-w/u-boot/blob/2024-04-bpi/configs/mt7988a_bpir4_emmc_defconfig

Maybe nor is autoselected by one of the mtd options

[BtbN]

The mt7988(-sd)-rfb dts files aren't used by any other boards, are they? In that case, couldn't the two sections about the non-existent spi2/nor just be removed from mt7988-rfb.dts, like they already are from the mt7988-sd-rfb.dts one?

https://github.com/frank-w/u-boot/blob/2024-04-bpi/arch/arm/dts/mt7988-rfb.dts#L89 https://github.com/frank-w/u-boot/blob/2024-04-bpi/arch/arm/dts/mt7988-rfb.dts#L150

These two sections are nowhere to be found in the sd one, and the sd builds also don't print that warning.

[frank-w]

The dts is written for the reference-board which i do not have,but this is a second board which may have nor

But in my repo i can disable the second section,but need to do this for every version

[BtbN]

All working fine now, forgot to close.