Support nvme for roc-rk3588s-pc

[isolitude]

roc-rk3588s-pc has an m.2 interface which both support SATA and PCIe. But it has to modify in dts and recomplie the firmware . Can we support nvme through dtbo overlay or something that don't need compling the firmware?

https://wiki.t-firefly.com/en/ROC-RK3588S-PC/usage_sata.html

[Joshua-Riek]

I do not have this board so I'm hesitant to make changes to it. But right now I'm focusing on bugs for Ubuntu 24.04, I will come back to this later.

[shaunco]

On a roc-rk3588s-pc with an NVMe drive that works on the latest Armbian, here is what I get on the v2.3.0 release of 24.04:

root@ubuntu:~# dmesg | grep -i pcie
[   16.665514] reg-fixed-voltage vcc3v3-pcie20: Looking up vin-supply from device tree
[   16.665521] vcc3v3_pcie20: supplied by vcc12v_dcin
[   16.671182] vcc3v3_pcie20: 3300 mV, enabled
[   16.671254] reg-fixed-voltage vcc3v3-pcie20: vcc3v3_pcie20 supplying 3300000uV
[   19.227602] rockchip-pm-domain fd8d8000.power-management:power-controller: Looking up pcie-supply from device tree
[   19.227611] rockchip-pm-domain fd8d8000.power-management:power-controller: Looking up pcie-supply property in node /power-management@fd8d8000/power-controller failed

It would appear that Armbian did 2 updates _(NVMe default --> sata0 to overlay + remove unused hdptxphy_hdmiclk0/1) since https://github.com/armbian/linux-rockchip/blob/rk-6.1-rkr3/arch/arm64/boot/dts/rockchip/rk3588s-roc-pc.dts since it was forked to https://github.com/Joshua-Riek/linux-rockchip/blob/noble/arch/arm64/boot/dts/rockchip/rk3588s-roc-pc.dts ... moving the sata stuff to https://github.com/armbian/linux-rockchip/blob/rk-6.1-rkr3/arch/arm64/boot/dts/rockchip/overlay/rk3588s-roc-pc-sata0.dts

Looking at the changes, they changed: https://github.com/Joshua-Riek/linux-rockchip/blob/73c50d1756568e9c2b1da19b875ea112fa998c00/arch/arm64/boot/dts/rockchip/rk3588s-roc-pc.dts#L1011-L1023

to just: https://github.com/armbian/linux-rockchip/blob/9d47b29834fabc4edaa12c69c6770d78735928d3/arch/arm64/boot/dts/rockchip/rk3588s-roc-pc.dts#L1000-L1004 ...making the PCIe power supply always active vs only active if sata0 is not used

Similarly: https://github.com/Joshua-Riek/linux-rockchip/blob/73c50d1756568e9c2b1da19b875ea112fa998c00/arch/arm64/boot/dts/rockchip/rk3588s-roc-pc.dts#L1065-L1071 ... is now in the overlay

So, NVMe doesn't work in the current build because the M2_SATA_OR_PCIE is set to 1 which is setup for SATA instead of PCIe/NVMe.

Would appear the fix is either to have separate builds for roc-rk3588s-pc-sata and roc-rk3588s-pc-nvme or to update the device-tree files, pulling in the sata overlay, then having the default be PCIe/NVMe with manual work required to disable NVMe for SATA0:

1. sudo vi /etc/default/u-boot:

   U_BOOT_FDT_OVERLAYS_DIR="/lib/firmware/6.1.0-1021-rockchip/device-tree/rockchip/overlay/"
   U_BOOT_FDT_OVERLAYS="rk3588s-roc-pc-sata0.dtbo"
   U_BOOT_SYNC_DTBS="true"
   U_BOOT_UPDATE="true"

2. sudo u-boot-update

[shaunco]

To test this, I grabbed https://github.com/armbian/linux-rockchip/blob/rk-6.1-rkr3/arch/arm64/boot/dts/rockchip/rk3588s-roc-pc.dts and re-built it and replaced the original dtb:

dtc -I dts -O dtb -o rk3588s-roc-pc.dtb rk3588s-roc-pc.dts
sudo cp rk3588s-roc-pc.dtb /lib/firmware/6.1.0-1021-rockchip/device-tree/rockchip/rk3588s-roc-pc.dtb

Post reboot:

ubuntu@ubuntu:~$ dmesg | grep -i pcie
[   16.246782] reg-fixed-voltage vcc3v3-pcie20: Looking up vin-supply from device tree
[   16.246789] vcc3v3_pcie20: supplied by vcc12v_dcin
[   16.252434] vcc3v3_pcie20: 3300 mV, enabled
[   16.252505] reg-fixed-voltage vcc3v3-pcie20: vcc3v3_pcie20 supplying 3300000uV
[   16.324812] dw-pcie fe190000.pcie: invalid resource
[   16.324824] dw-pcie fe190000.pcie: Failed to initialize host
[   16.324833] dw-pcie: probe of fe190000.pcie failed with error -22
[   16.325387] rk-pcie fe190000.pcie: invalid prsnt-gpios property in node
[   16.325422] rk-pcie fe190000.pcie: Looking up vpcie3v3-supply from device tree
[   16.326394] rk-pcie fe190000.pcie: host bridge /pcie@fe190000 ranges:
[   16.326454] rk-pcie fe190000.pcie:       IO 0x00f4100000..0x00f41fffff -> 0x00f4100000
[   16.326499] rk-pcie fe190000.pcie:      MEM 0x00f4200000..0x00f4ffffff -> 0x00f4200000
[   16.326536] rk-pcie fe190000.pcie:      MEM 0x0a00000000..0x0a3fffffff -> 0x0a00000000
[   16.326626] rk-pcie fe190000.pcie: iATU unroll: enabled
[   16.326652] rk-pcie fe190000.pcie: iATU regions: 8 ob, 8 ib, align 64K, limit 8G
[   16.532578] rk-pcie fe190000.pcie: PCIe Linking... LTSSM is 0x3
[   16.614772] rk-pcie fe190000.pcie: PCIe Link up, LTSSM is 0x130011
[   16.614820] rk-pcie fe190000.pcie: PCIe Gen.2 x1 link up
[   16.615050] rk-pcie fe190000.pcie: PCI host bridge to bus 0004:40
[   16.629631] pci 0004:41:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0004:40:00.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
[   16.642100] pcieport 0004:40:00.0: PME: Signaling with IRQ 74
[   18.845383] rockchip-pm-domain fd8d8000.power-management:power-controller: Looking up pcie-supply from device tree
[   18.845391] rockchip-pm-domain fd8d8000.power-management:power-controller: Looking up pcie-supply property in node /power-management@fd8d8000/power-controller failed

ubuntu@ubuntu:~$ lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0          7:0    0    94M  1 loop /snap/lxd/29579
loop1          7:1    0  69.2M  1 loop /snap/core22/1383
loop2          7:2    0  33.7M  1 loop /snap/snapd/21761
mmcblk0      179:0    0 233.3G  0 disk 
└─mmcblk0p1  179:1    0 230.9G  0 part 
mmcblk0boot0 179:32   0     4M  1 disk 
mmcblk0boot1 179:64   0     4M  1 disk 
mmcblk1      179:96   0 238.3G  0 disk 
├─mmcblk1p1  179:97   0     4M  0 part 
└─mmcblk1p2  179:98   0 238.3G  0 part /
nvme0n1      259:0    0   1.9T  0 disk 
└─nvme0n1p1  259:1    0   1.9T  0 part 

ubuntu@ubuntu:~$ sudo mkdir /mnt/nvme
ubuntu@ubuntu:~$ sudo mount /dev/nvme0n1p1 /mnt/nvme
ubuntu@ubuntu:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           3.2G  2.1M  3.2G   1% /run
/dev/mmcblk1p2  235G  2.5G  223G   2% /
tmpfs            16G     0   16G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           3.2G   12K  3.2G   1% /run/user/1000
/dev/nvme0n1p1  1.9T   46G  1.7T   3% /mnt/nvme