Booting PINN from PCIe on CM4

[NumberOneGit]

Using the same CM4 carrier board with a dedicated PCIe slot, I am able to boot normally via a USB enclosure with an NVME drive into PINN but when booting with the same NVME drive in the PCIe slot I receive the error "Cannot find the drive with PINN files". Is this because PINN is looking for mmcblk, sda1, etc but not nvme0n1p1?

[procount]

Not really. PINN searches all devices in /sys/class/block that are not virtual devices by mounting them and trying to locate recovery.rfs. Does your drive show up in there?

[NumberOneGit]

At the moment, after clicking past the above message I get to: "Recovery application crashed Starting shell" etc When I have PINN installed on the eMMC it does not list the NVME over PCIe, but does over usb as sda

[procount]

Maybe it is missing a driver?

[procount]

@XECDesign - Do I need any additional drivers for NVME drives in the CM4 PCIe slot?

[XECDesign]

I don't know, but @pelwell may.

[lurch]

@procount Might be worth trying to compare the Linux kernel config used by PINN with that used by Raspberry Pi OS, to see if anything obvious jumps out?

[pelwell]

We have CONFIG_BLK_DEV_NVME=y and CONFIG_NVMEM_RMEM=m.

[procount]

Thanks @pelwell. I enabled CONFIG_NVMEM_RMEM after adding required CONFIG_PCI, but CONFIG_NVMEM_RMEM was not available. Guess I need a newer kernel for that. I'm using 5.4.51

[NumberOneGit]

I'll try to test this out tonight, but are there any potential updates with PINN 3.7?

[procount]

3.7.1 is the latest. It has a later 5.10 kernel but I haven't enabled the additional CONFIG_NVMEM_RMEM config yet so it may still not work. I'll check if it is now available.

[NumberOneGit]

Thanks again for all your great work here. I am wondering whether you ever had a chance to see whether the CONFIG_NVMEM_RMEM config was available in the current kernel. Thanks!

[pelwell]

$ sudo modprobe configs
$ zcat /proc/config.gz | grep NVMEM_RMEM

[procount]

Yes it seems to be available now, so I'll push up a version for you to test if you wouldn't mind.

[NumberOneGit]

Absolutely. Reply here once it's up and I'll give it a test.

[procount]

https://sourceforge.net/projects/pinn/files/testing/pinn-3753.zip/download

[procount]

I have no idea if it will work.....

[NumberOneGit]

From what I can tell so far it works! I wiped a CM4 so that only the NVME drive had PINN on it and am currently installing my OS's to PINN. I will update again once the installation is complete and I am able to test boot some OS's, but all looks good on my end so far.

[NumberOneGit]

No issues with PINN itself, but once the images are installed I will need to do quite a bit of research to figure out how to get them set up properly. The firmware that allows this to work is pretty new, so manually updating the firmware of the various installations is necessary to even get access to PCIE boot. I will keep experimenting with this. Do you expect it to be enabled in updates (I notice there was an update earlier today) or not until it is more proven?

[lurch]

AFAIK PCIe booting should "just work" with Raspberry Pi OS, but of course the situation with other OSes may vary... :wink:

@timg236 or @peterharperuk Does the partition-selection used by NOOBS (something to do with PM_RSTS IIRC) also work with booting from NVMe SSDs?

[procount]

@NumberOneGit - As you indicated that it basically worked, I have added the appropriate drivers to v3.8, so you can update to that version for your tests if you like. I have not announced that feature yet as I await your further investigations.

[NumberOneGit]

@procount I would definitely say that it can be officially announced. I have compiled a working kernel for OSMC using the config changes mentioned above and LibreElec has said they can add those changes to future builds. RPOS has it enabled by default and RetroPie can boot from NVME over PCIe if it has been installed on another drive (SD or eMMC) and fully updated then cloned over.

I don't expect Lineage to work with it any time soon.

[lurch]

@NumberOneGit Does that mean that PINN's json files ought to report which OSes are bootable over NVME, and only offer to install those ones? (Like it already does for OSes that are bootable over USB, IIRC?)

[NumberOneGit]

@lurch I'm not sure how that's decided, as many OS's that I would put on the CM4 functional list seem not to appear there. My only concern that if hiding functionality isn't working as expected, 'showall' may not work 100%. I tried to install Kali 64-bit to USB a few weeks ago after setting up my cmdline.txt on pinn.mjh.nz and even with 'showall' enabled, it would disappear when I selected my USB drive and re-appear when I selected my SD card (on RP4B). I use 'showall' by default when I'm using PINN with my CM4 because it's difficult to draw the line between "working" and "not working" in general at this stage. For example, to get USB to work on OSMC I put "i2c-dev" in /etc/modules which allows my "otg_mode=1" command to work. The dev may never make that change in the OS due to the way they choose to release it, but it takes very little effort to fix once one is aware that it's needed.

[procount]

Normally, PINN hides OSes that do not include the current RPi model in the supported_models field. The showall option shows all OSes regardless of RPi model. But this is independent of whether the OS can be installed to USB. The first is normally due to firmware and the second is generally due to kernel facilities.

I don't have a CM4 to test compatibility, so if you are able to do this I can add more OSes to the CM 4 family.

[procount]

@lurch - how would I know it was using nvme vs usb?

[lurch]

@procount I don't have a CM4 either, but I believe if you're booting off NVMe the boot device would be something like /dev/nvme0n1p1 (at least, that's how the NVMe SSD in my Linux laptop appears).

[paulwratt]

yes /dev/nvme?n?p? is also how NVME SSD is seen on RPi.

[NumberOneGit]

@procount Do you make any changes at all to modify the OS itself in cases where the OS is linked via sourceforge rather than linked directly to the dev's page? I ask because I'd be glad to point out minor changes like the one mentioned above for OSMC that would make CM4 fully functional, that the dev themself may not consider if they want to keep their OS vanilla. RPOS doesn't package a separate CM4 version, so any modules added to other OS's should mimic RPOS' approach. Not a suggestion, just curious because I could provide very minor tweaks that would not hurt people with RPi4B but save multiple steps for people with a CM4.
One immediate suggestion would be to add: " [cm4] dtoverlay=dwc2,dr_mode=host" to config.txt of future PINN releases. You could also switch over to XHCI as the RPi Foundation has because there is no use for "device mode" in PINN but due to the limited time spent in PINN this would just be to stay current and not in any way necessary.

[procount]

I try to keep the OS identical. The only changes I make are superficial ones to get the OS to multi-boot. For example: removing U-Boot, removing scripts that expand partitions & file systems etc. And of course, I convert the .img to a .tar.xz for each partition. Adding tweaks like [CM4] sections to config.txt of an OS would be ok, because they won't affect other models, but it makes it difficult to manage each time a new update to the OS it produced. An alternative is to capture the changes in a separate tar file and apply them at installation using a process that used to be called 'noobsconfig' I can add the section to PINN's config.txt easily if it helps CM4 users.

[procount]

BTW, I have a test version of PINN almost ready to test out the hiding of OSes that don't support NVMe, as per Lurch's suggestion. Would you be able to test it out for me if I provide some test instructions?

[NumberOneGit]

Of course! In general, I would say that RPOS and anything that is actually RPOS at the core (RetroPie) is at the moment the only successful OOTB NVME that can be added to the list. As for LibreElec, I will have to check the nightly builds to see whether functionality has been added or if it was meant for a later release (the test version I verified worked OOTB). OSMC I compiled myself and was asked to make a pull request, so not sure how long that will take. A 'noobsconfig' for CM$ that adds the line I mentioned above to /etc/modules in OSMC would be helpful if that is a system you already have in place. I'm glad to update any other finding I make as they come along. I'm interested in Kali, but am new to compiling and couldn't figure out how to update the kernel past 5.4. I'll try out Ubuntu soon, but in general had some issues with Ubuntu and PINN already.

[procount]

The test version for NVMe is available at: https://sourceforge.net/projects/pinn/files/testing/pinn-381.zip/download I shall write up some tests to do in due course.

[lurch]

I don't have a CM4, BUT I don't see how

For example, to get USB to work on OSMC I put "i2c-dev" in /etc/modules which allows my "otg_mode=1" command to work.

makes any sense? Aren't USB and I2C entirely separate subsystems?

[NumberOneGit]

I understand your confusion. I felt the same way. My first step was discovering that the way to enable DWC2 drivers in an OS was to add that under /etc/modules. Because I knew that RPi Foundation now preferred to use XHCI over DWC2, I went to /etc/modules to see what was currently being used to enable the "otg_mode" command. I wasn't expecting "i2c-dev" to enable the command either, but when I entered it it worked. An alternative would be to add dwc2, but that seems to be deprecated.

[NumberOneGit]

@procount Thanks, I've downloaded it and will be glad to test it out.

[procount]

@NumberOneGit - Here are some tests for you: Boot PINN from an SD card on a CM4

1. Go to Testing tab and check you can see Test1, Test2, Test3 and Test4
2. Change your Destination Drive to NVME and note which of the 4 above entries are now visible. Boot PINN from the NVME on a CM4
3. Go to Testing tab and note which of Test1, Test2, Test3 and Test4 are visible
4. (Optionally, if you have a spare blank USB device) Change your Destination Drive to your USB device and note which of the 4 above entries are now visible.

[NumberOneGit]

I only have eMMC version of the CM4 but I can try the same. I believe PINN can't tell the difference as they both appear as mmcblk. I'll try that out now.

[NumberOneGit]

Test 1 and Test 2 are visible. When the NVME is in USB enclosure all 4 show up. Glad to go through any other tests as needed.

[procount]

Probably I made a mistake calling the OS entries TestX cos this could get confusing :/ So, there are 4 numbered STEPS in my previous post. Please could you identify the STEP you are testing and which of the TestX OS entries are visible for each one.

[procount]

Not having a CM4 I've not really followed what it is or has. So the eMMC is in place of the SD card, and is still identified as /dev/mmcblk0 ? Your NVMe is in an external USB enclosure. What device does it show up as?

[NumberOneGit]

I did the following tests on the official CM4 development board for which I have a pcie adapter. The CM4 can be purchased with either SD slot or eMMC, I have two of the eMMC variety. The NVME over pcie shows up as "nvme0n1", the eMMC shows up as "mmcblk0", when in the USB enclosure the NVME shows up as "sda1". I installed PINN on only the eMMC for the first two steps of the pcie test with a blank NVME over pcie. For the 3rd and 4th steps of the NVME pcie test I formatted the eMMC and installed the test version on the NVME over pcie. Step 4 was the same regardless of which device PINN was installed on, as expected.

Step 1: all four are visible Step 2: 1 and 2 are visible Step 3: 1 and 3 are visible Step 4: all four are visible.

[NumberOneGit]

LineageOS update: I was able to update the 5.4 kernel for Lineage 18.1 with the CONFIG_BLK_DEV_NVME=y config and can report that with this change I have found full functionality including TWRP. I have also updated the 5.10 kernel for Lineage 19 to include CONFIG_NVMEM_RMEM=m. I'm not sure whether you have any interest in the compiled kernels, as the change should have no negative effect on operation of the devices. Lineage 19 actually worked already without the additional config. I hadn't realized I had still been trying on 18.1. That said, 18.1 is perfectly stable on the CM4 and has OpenGAPPS availability while Lineage 19 currently has functionality issues (specific to CM4 not Pi4B) and I am only using it to try to debug and report back to the developer. I'm glad to share any kernels that I compile for use in the /boot folder (with only one or both of the changes mentioned above depending on whether it's a later kernel that allows the RMEM change). Other than the kernel change, I also added the most recent start4 and fixup4 firmware files to achieve functionality.

(It would still be a great help to add the dtoverlay line in /boot of PINN to future releases to avoid having to attach the CM4 to a laptop every time PINN updates to get USB back)

[lurch]

(It would still be a great help to add the dtoverlay line in /boot of PINN to future releases to avoid having to attach the CM4 to a laptop every time PINN updates to get USB back)

From what I've read,

[cm4]
otg_mode=1

(which is what RPiOS does) is a better option than

[cm4]
dtoverlay=dwc2,dr_mode=host

?

[NumberOneGit]

I thought it would take effort to implement as it isn't enabled in all OS's, but I just tested and " otg_mode=1" does already work with PINN and that would be better.

[NumberOneGit]

I can also confirm that the latest version of Kali 64 bit works with a recompiled kernel.