NVMe Support - Githubissues

ftr3g commented 1 year ago

HELLO, Thank you for your port for Nanopi R5!

I encounter a problem with the management of the NVMe which does not seem to detect while it works perfectly with the FriendElec version of Debian Buster. Is there anything to do?

Thanks in advance

Friendlyelec debian 11

lspci 0000:00:00.0 PCI bridge: Fuzhou Rockchip Electronics Co., Ltd Device 3566 (rev 01) 0000:01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05) 0001:10:00.0 PCI bridge: Fuzhou Rockchip Electronics Co., Ltd Device 3566 (rev 01) 0001:11:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05) 0002:20:00.0 PCI bridge: Fuzhou Rockchip Electronics Co., Ltd Device 3566 (rev 01) 0002:21:00.0 Non-Volatile memory controller: Sandisk Corp WD Black SN750 / PC SN730 NVMe SSD

Debian 12 RC4

lspci 0000:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01) 0000:01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05) 0001:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01) 0001:01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05) 0002:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01)

arbv commented 1 year ago

Wow, that is too bad, if this is does not work. I wanted to use NVME on my device with a small boot partition on eMMC or SD.

Can you try this (beware that it might crash your system): echo 1 > /sys/bus/pci/rescan (root privileges are required) and rerun lspci?

ftr3g commented 1 year ago

lspci
0000:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01)
0000:01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
0001:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01)
0001:01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
0002:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01)
echo 1 > /sys/bus/pci/rescan
lspci
0000:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01)
0000:01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
0001:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01)
0001:01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
0002:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01)
0002:01:00.0 Non-Volatile memory controller: Sandisk Corp WD Black SN750 / PC SN730 NVMe SSD

Yes, NVMe is detected. However fdisk does not detect it and the system crashes on shutdown

inindev commented 1 year ago

I can confirm it works with a Samsung 980, no crashes.

On 5/25/23 4:47 PM, Artem Boldariev wrote:

Wow, that is too bad, if this is does not work. I wanted to use NVME on my device with a small boot partition on eMMC or SD.

Can you try this (beware that it might crash your system) echo 1 > /sys/bus/pci/rescan?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>

arbv commented 1 year ago

@ftr3g Cool, thanks for confirming that. Now you can try to somehow run this command from within initramfs on boot, but I am not sure that it is the right way.

Perhaps, @inindev could give us more information if something needs or could to be done on the side of bootloader.

arbv commented 1 year ago

I can confirm it works with a Samsung 980, no crashes.

@inindev Do you mean that it works just fine from the very beginning, or you still need to force the system to rescan the PCI bus in order to make SSD work?

I am wondering if pcie_aspm=off kernel parameter could help here, too. Also, I am wondering if it is possible to add a command to the boot script to force it to rescan the PCI bus before loading the OS.

inindev commented 1 year ago

There are two patches I want to apply to see if they help:

pci: pcie_dw_rockchip: Hide BARs of the root complex https://github.com/Kwiboo/u-boot-rockchip/commit/74dd38b0945b6f1dba9ad064241eb295e062bd87

rockchip: rk356x: Update PCIe config, IO and memory regions https://github.com/Kwiboo/u-boot-rockchip/commit/1fc2a3cc1248ebdaa16bd520e50d9da291d4b8b4

Give me a bit to recompile with these applied.

On 5/25/23 5:42 PM, Artem Boldariev wrote:

@.***(https://github.com/ftr3g) Cool, thanks for confirming that. Now you can try to somehow run this command from within initramfs on boot, but I am not sure that it is the right way.

Perhaps, @.***(https://github.com/inindev) could give us more information if something needs or could to be done on the side of bootloader.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

inindev commented 1 year ago

I have applied the above patches to this image: https://drive.google.com/file/d/1xK70fILu7jTe9C74Y9hog9iW0kW4vOSN/view?usp=sharing

I would be interested to see the dmesg output with your m.2 nvme

# dmesg | grep pci
[    5.233115] rockchip-dw-pcie 3c0000000.pcie: host bridge /pcie@fe260000 ranges:
[    5.233841] rockchip-dw-pcie 3c0000000.pcie:       IO 0x00f4100000..0x00f41fffff -> 0x00f4100000
[    5.234641] rockchip-dw-pcie 3c0000000.pcie:      MEM 0x00f4200000..0x00f5ffffff -> 0x00f4200000
[    5.235948] rockchip-dw-pcie 3c0000000.pcie:      MEM 0x0300000000..0x033fffffff -> 0x0040000000
[    5.237000] rockchip-dw-pcie 3c0000000.pcie: iATU unroll: enabled
[    5.237561] rockchip-dw-pcie 3c0000000.pcie: iATU regions: 8 ob, 8 ib, align 64K, limit 8G
[    5.443048] rockchip-dw-pcie 3c0000000.pcie: PCIe Gen.2 x1 link up
[    5.466503] rockchip-dw-pcie 3c0000000.pcie: PCI host bridge to bus 0000:00
[    5.467182] pci_bus 0000:00: root bus resource [bus 00-0f]
[    5.467686] pci_bus 0000:00: root bus resource [io  0x0000-0xfffff] (bus address [0xf4100000-0xf41fffff])
[    5.468533] pci_bus 0000:00: root bus resource [mem 0xf4200000-0xf5ffffff]
[    5.469146] pci_bus 0000:00: root bus resource [mem 0x300000000-0x33fffffff] (bus address [0x40000000-0x7fffffff])
[    5.470109] pci 0000:00:00.0: [1d87:3566] type 01 class 0x060400
[    5.470681] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[    5.471417] pci 0000:00:00.0: supports D1 D2
[    5.471816] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[    5.499542] pci_bus 0000:01: busn_res: can not insert [bus 01-ff] under [bus 00-0f] (conflicts with (null) [bus 00-0f])
[    5.500641] pci 0000:01:00.0: [10ec:8125] type 00 class 0x020000
[    5.501265] pci 0000:01:00.0: reg 0x10: initial BAR value 0x00000000 invalid
[    5.501904] pci 0000:01:00.0: reg 0x10: [io  size 0x0100]
[    5.502479] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x0000ffff 64bit]
[    5.503232] pci 0000:01:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit]
[    5.505782] pci 0000:01:00.0: supports D1 D2
[    5.506192] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    5.547203] pci 0000:00:00.0: BAR 14: assigned [mem 0xf4200000-0xf42fffff]
[    5.547852] pci 0000:00:00.0: BAR 6: assigned [mem 0xf4300000-0xf430ffff pref]
[    5.548500] pci 0000:00:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[    5.549068] pci 0000:01:00.0: BAR 2: assigned [mem 0xf4200000-0xf420ffff 64bit]
[    5.549777] pci 0000:01:00.0: BAR 4: assigned [mem 0xf4210000-0xf4213fff 64bit]
[    5.550483] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
[    5.551124] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    5.551613] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    5.552161] pci 0000:00:00.0:   bridge window [mem 0xf4200000-0xf42fffff]
[    5.591259] pcieport 0000:00:00.0: PME: Signaling with IRQ 37
[    5.598598] pcieport 0000:00:00.0: AER: enabled with IRQ 37
[    5.816917] rockchip-dw-pcie 3c0400000.pcie: host bridge /pcie@fe270000 ranges:
[    5.817621] rockchip-dw-pcie 3c0400000.pcie:       IO 0x00f2100000..0x00f21fffff -> 0x00f2100000
[    5.818424] rockchip-dw-pcie 3c0400000.pcie:      MEM 0x00f2200000..0x00f3ffffff -> 0x00f2200000
[    5.819403] rockchip-dw-pcie 3c0400000.pcie:      MEM 0x0340000000..0x037fffffff -> 0x0040000000
[    5.828704] rockchip-dw-pcie 3c0400000.pcie: iATU unroll: enabled
[    5.829253] rockchip-dw-pcie 3c0400000.pcie: iATU regions: 8 ob, 8 ib, align 64K, limit 8G
[    6.035071] rockchip-dw-pcie 3c0400000.pcie: PCIe Gen.2 x1 link up
[    6.035949] rockchip-dw-pcie 3c0400000.pcie: PCI host bridge to bus 0001:00
[    6.036579] pci_bus 0001:00: root bus resource [bus 00-0f]
[    6.037074] pci_bus 0001:00: root bus resource [io  0x100000-0x1fffff] (bus address [0xf2100000-0xf21fffff])
[    6.037940] pci_bus 0001:00: root bus resource [mem 0xf2200000-0xf3ffffff]
[    6.038552] pci_bus 0001:00: root bus resource [mem 0x340000000-0x37fffffff] (bus address [0x40000000-0x7fffffff])
[    6.039548] pci 0001:00:00.0: [1d87:3566] type 01 class 0x060400
[    6.040114] pci 0001:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[    6.040786] pci 0001:00:00.0: supports D1 D2
[    6.041172] pci 0001:00:00.0: PME# supported from D0 D1 D3hot
[    6.049752] pci_bus 0001:01: busn_res: can not insert [bus 01-ff] under [bus 00-0f] (conflicts with (null) [bus 00-0f])
[    6.050840] pci 0001:01:00.0: [10ec:8125] type 00 class 0x020000
[    6.051482] pci 0001:01:00.0: reg 0x10: [io  0x0000-0x00ff]
[    6.052074] pci 0001:01:00.0: reg 0x18: [mem 0x00000000-0x0000ffff 64bit]
[    6.052735] pci 0001:01:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit]
[    6.053868] pci 0001:01:00.0: supports D1 D2
[    6.054256] pci 0001:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    6.067214] pci 0001:00:00.0: BAR 14: assigned [mem 0xf2200000-0xf22fffff]
[    6.067851] pci 0001:00:00.0: BAR 6: assigned [mem 0xf2300000-0xf230ffff pref]
[    6.068497] pci 0001:00:00.0: BAR 13: assigned [io  0x100000-0x100fff]
[    6.069091] pci 0001:01:00.0: BAR 2: assigned [mem 0xf2200000-0xf220ffff 64bit]
[    6.069789] pci 0001:01:00.0: BAR 4: assigned [mem 0xf2210000-0xf2213fff 64bit]
[    6.070482] pci 0001:01:00.0: BAR 0: assigned [io  0x100000-0x1000ff]
[    6.071108] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[    6.071580] pci 0001:00:00.0:   bridge window [io  0x100000-0x100fff]
[    6.072154] pci 0001:00:00.0:   bridge window [mem 0xf2200000-0xf22fffff]
[    6.075227] pcieport 0001:00:00.0: PME: Signaling with IRQ 43
[    6.076315] pcieport 0001:00:00.0: AER: enabled with IRQ 43
[    6.113508] rockchip-dw-pcie 3c0800000.pcie: host bridge /pcie@fe280000 ranges:
[    6.114215] rockchip-dw-pcie 3c0800000.pcie:       IO 0x00f0100000..0x00f01fffff -> 0x00f0100000
[    6.115054] rockchip-dw-pcie 3c0800000.pcie:      MEM 0x00f0200000..0x00f1ffffff -> 0x00f0200000
[    6.115850] rockchip-dw-pcie 3c0800000.pcie:      MEM 0x0380000000..0x03bfffffff -> 0x0040000000
[    6.125447] rockchip-dw-pcie 3c0800000.pcie: iATU unroll: enabled
[    6.125997] rockchip-dw-pcie 3c0800000.pcie: iATU regions: 8 ob, 8 ib, align 64K, limit 8G
[    6.331049] rockchip-dw-pcie 3c0800000.pcie: PCIe Gen.3 x1 link up
[    6.331896] rockchip-dw-pcie 3c0800000.pcie: PCI host bridge to bus 0002:00
[    6.332526] pci_bus 0002:00: root bus resource [bus 00-0f]
[    6.333022] pci_bus 0002:00: root bus resource [io  0x200000-0x2fffff] (bus address [0xf0100000-0xf01fffff])
[    6.333888] pci_bus 0002:00: root bus resource [mem 0xf0200000-0xf1ffffff]
[    6.334502] pci_bus 0002:00: root bus resource [mem 0x380000000-0x3bfffffff] (bus address [0x40000000-0x7fffffff])
[    6.335499] pci 0002:00:00.0: [1d87:3566] type 01 class 0x060400
[    6.336056] pci 0002:00:00.0: reg 0x10: [mem 0x00000000-0x3fffffff]
[    6.336622] pci 0002:00:00.0: reg 0x14: [mem 0x00000000-0x3fffffff]
[    6.337183] pci 0002:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[    6.337851] pci 0002:00:00.0: supports D1 D2
[    6.338237] pci 0002:00:00.0: PME# supported from D0 D1 D3hot
[    6.346843] pci_bus 0002:01: busn_res: can not insert [bus 01-ff] under [bus 00-0f] (conflicts with (null) [bus 00-0f])
[    6.347953] pci 0002:01:00.0: [144d:a809] type 00 class 0x010802
[    6.348584] pci 0002:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
[    6.350094] pci 0002:01:00.0: 7.876 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x1 link at 0002:00:00.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
[    6.363188] pci 0002:00:00.0: BAR 0: assigned [mem 0x380000000-0x3bfffffff]
[    6.363849] pci 0002:00:00.0: BAR 1: no space for [mem size 0x40000000]
[    6.364440] pci 0002:00:00.0: BAR 1: failed to assign [mem size 0x40000000]
[    6.365062] pci 0002:00:00.0: BAR 14: assigned [mem 0xf0200000-0xf02fffff]
[    6.365678] pci 0002:00:00.0: BAR 6: assigned [mem 0xf0300000-0xf030ffff pref]
[    6.366332] pci 0002:01:00.0: BAR 0: assigned [mem 0xf0200000-0xf0203fff 64bit]
[    6.367076] pci 0002:00:00.0: PCI bridge to [bus 01-ff]
[    6.367557] pci 0002:00:00.0:   bridge window [mem 0xf0200000-0xf02fffff]
[    6.370753] pcieport 0002:00:00.0: PME: Signaling with IRQ 46
[    6.372035] pcieport 0002:00:00.0: AER: enabled with IRQ 46
[    6.458632] nvme nvme0: pci function 0002:01:00.0

for me, I see the nvme device either way:

$ ls -al /dev/nvme0*
crw------- 1 root root 235, 0 Feb 28 11:15 /dev/nvme0
brw-rw---- 1 root disk 259, 0 Feb 28 11:15 /dev/nvme0n1
brw-rw---- 1 root disk 259, 1 Feb 28 11:15 /dev/nvme0n1p1

ftr3g commented 1 year ago

Test with your patched image (https://drive.google.com/file/d/1xK70fILu7jTe9C74Y9hog9iW0kW4vOSN/view?usp=sharing)

dmesg | grep pci
[    5.386965] rockchip-dw-pcie 3c0000000.pcie: host bridge /pcie@fe260000 ranges:
[    5.387679] rockchip-dw-pcie 3c0000000.pcie:       IO 0x00f4100000..0x00f41fffff -> 0x00f4100000
[    5.388495] rockchip-dw-pcie 3c0000000.pcie:      MEM 0x00f4200000..0x00f5ffffff -> 0x00f4200000
[    5.389299] rockchip-dw-pcie 3c0000000.pcie:      MEM 0x0300000000..0x033fffffff -> 0x0040000000
[    5.390347] rockchip-dw-pcie 3c0000000.pcie: iATU unroll: enabled
[    5.390938] rockchip-dw-pcie 3c0000000.pcie: iATU regions: 8 ob, 8 ib, align 64K, limit 8G
[    5.598811] rockchip-dw-pcie 3c0000000.pcie: PCIe Gen.2 x1 link up
[    5.599679] rockchip-dw-pcie 3c0000000.pcie: PCI host bridge to bus 0000:00
[    5.600319] pci_bus 0000:00: root bus resource [bus 00-0f]
[    5.600817] pci_bus 0000:00: root bus resource [io  0x0000-0xfffff] (bus address [0xf4100000-0xf41fffff])
[    5.601663] pci_bus 0000:00: root bus resource [mem 0xf4200000-0xf5ffffff]
[    5.602278] pci_bus 0000:00: root bus resource [mem 0x300000000-0x33fffffff] (bus address [0x40000000-0x7fffffff])
[    5.603338] pci 0000:00:00.0: [1d87:3566] type 01 class 0x060400
[    5.603913] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[    5.604599] pci 0000:00:00.0: supports D1 D2
[    5.605004] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[    5.620344] pci_bus 0000:01: busn_res: can not insert [bus 01-ff] under [bus 00-0f] (conflicts with (null) [bus 00-0f])
[    5.621444] pci 0000:01:00.0: [10ec:8125] type 00 class 0x020000
[    5.622069] pci 0000:01:00.0: reg 0x10: initial BAR value 0x00000000 invalid
[    5.622699] pci 0000:01:00.0: reg 0x10: [io  size 0x0100]
[    5.623313] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x0000ffff 64bit]
[    5.623993] pci 0000:01:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit]
[    5.626055] pci 0000:01:00.0: supports D1 D2
[    5.626469] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    5.643762] pci 0000:00:00.0: BAR 14: assigned [mem 0xf4200000-0xf42fffff]
[    5.644412] pci 0000:00:00.0: BAR 6: assigned [mem 0xf4300000-0xf430ffff pref]
[    5.645061] pci 0000:00:00.0: BAR 13: assigned [io  0x1000-0x1fff]
[    5.645631] pci 0000:01:00.0: BAR 2: assigned [mem 0xf4200000-0xf420ffff 64bit]
[    5.646341] pci 0000:01:00.0: BAR 4: assigned [mem 0xf4210000-0xf4213fff 64bit]
[    5.647119] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
[    5.647703] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    5.648177] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    5.648724] pci 0000:00:00.0:   bridge window [mem 0xf4200000-0xf42fffff]
[    5.670396] pcieport 0000:00:00.0: PME: Signaling with IRQ 39
[    5.676801] pcieport 0000:00:00.0: AER: enabled with IRQ 39
[    5.904840] rockchip-dw-pcie 3c0400000.pcie: host bridge /pcie@fe270000 ranges:
[    5.905545] rockchip-dw-pcie 3c0400000.pcie:       IO 0x00f2100000..0x00f21fffff -> 0x00f2100000
[    5.906344] rockchip-dw-pcie 3c0400000.pcie:      MEM 0x00f2200000..0x00f3ffffff -> 0x00f2200000
[    5.907162] rockchip-dw-pcie 3c0400000.pcie:      MEM 0x0340000000..0x037fffffff -> 0x0040000000
[    5.916790] rockchip-dw-pcie 3c0400000.pcie: iATU unroll: enabled
[    5.917354] rockchip-dw-pcie 3c0400000.pcie: iATU regions: 8 ob, 8 ib, align 64K, limit 8G
[    6.122882] rockchip-dw-pcie 3c0400000.pcie: PCIe Gen.2 x1 link up
[    6.123765] rockchip-dw-pcie 3c0400000.pcie: PCI host bridge to bus 0001:00
[    6.124400] pci_bus 0001:00: root bus resource [bus 00-0f]
[    6.124898] pci_bus 0001:00: root bus resource [io  0x100000-0x1fffff] (bus address [0xf2100000-0xf21fffff])
[    6.125767] pci_bus 0001:00: root bus resource [mem 0xf2200000-0xf3ffffff]
[    6.126383] pci_bus 0001:00: root bus resource [mem 0x340000000-0x37fffffff] (bus address [0x40000000-0x7fffffff])
[    6.127416] pci 0001:00:00.0: [1d87:3566] type 01 class 0x060400
[    6.127990] pci 0001:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[    6.128672] pci 0001:00:00.0: supports D1 D2
[    6.129061] pci 0001:00:00.0: PME# supported from D0 D1 D3hot
[    6.137779] pci_bus 0001:01: busn_res: can not insert [bus 01-ff] under [bus 00-0f] (conflicts with (null) [bus 00-0f])
[    6.138896] pci 0001:01:00.0: [10ec:8125] type 00 class 0x020000
[    6.139513] pci 0001:01:00.0: reg 0x10: [io  0x0000-0x00ff]
[    6.140098] pci 0001:01:00.0: reg 0x18: [mem 0x00000000-0x0000ffff 64bit]
[    6.140757] pci 0001:01:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit]
[    6.141878] pci 0001:01:00.0: supports D1 D2
[    6.142268] pci 0001:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    6.154978] pci 0001:00:00.0: BAR 14: assigned [mem 0xf2200000-0xf22fffff]
[    6.155620] pci 0001:00:00.0: BAR 6: assigned [mem 0xf2300000-0xf230ffff pref]
[    6.156264] pci 0001:00:00.0: BAR 13: assigned [io  0x100000-0x100fff]
[    6.156857] pci 0001:01:00.0: BAR 2: assigned [mem 0xf2200000-0xf220ffff 64bit]
[    6.157556] pci 0001:01:00.0: BAR 4: assigned [mem 0xf2210000-0xf2213fff 64bit]
[    6.158249] pci 0001:01:00.0: BAR 0: assigned [io  0x100000-0x1000ff]
[    6.159586] pci 0001:00:00.0: PCI bridge to [bus 01-ff]
[    6.160059] pci 0001:00:00.0:   bridge window [io  0x100000-0x100fff]
[    6.160687] pci 0001:00:00.0:   bridge window [mem 0xf2200000-0xf22fffff]
[    6.163809] pcieport 0001:00:00.0: PME: Signaling with IRQ 43
[    6.166672] pcieport 0001:00:00.0: AER: enabled with IRQ 43
[    6.210312] rockchip-dw-pcie 3c0800000.pcie: host bridge /pcie@fe280000 ranges:
[    6.211081] rockchip-dw-pcie 3c0800000.pcie:       IO 0x00f0100000..0x00f01fffff -> 0x00f0100000
[    6.211899] rockchip-dw-pcie 3c0800000.pcie:      MEM 0x00f0200000..0x00f1ffffff -> 0x00f0200000
[    6.212694] rockchip-dw-pcie 3c0800000.pcie:      MEM 0x0380000000..0x03bfffffff -> 0x0040000000
[    6.222281] rockchip-dw-pcie 3c0800000.pcie: iATU unroll: enabled
[    6.222867] rockchip-dw-pcie 3c0800000.pcie: iATU regions: 8 ob, 8 ib, align 64K, limit 8G
[    6.330811] rockchip-dw-pcie 3c0800000.pcie: PCIe Gen.3 x1 link up
[    6.331670] rockchip-dw-pcie 3c0800000.pcie: PCI host bridge to bus 0002:00
[    6.332301] pci_bus 0002:00: root bus resource [bus 00-0f]
[    6.332797] pci_bus 0002:00: root bus resource [io  0x200000-0x2fffff] (bus address [0xf0100000-0xf01fffff])
[    6.333663] pci_bus 0002:00: root bus resource [mem 0xf0200000-0xf1ffffff]
[    6.334275] pci_bus 0002:00: root bus resource [mem 0x380000000-0x3bfffffff] (bus address [0x40000000-0x7fffffff])
[    6.335285] pci 0002:00:00.0: [1d87:3566] type 01 class 0x060400
[    6.335841] pci 0002:00:00.0: reg 0x10: [mem 0x00000000-0x3fffffff]
[    6.336406] pci 0002:00:00.0: reg 0x14: [mem 0x00000000-0x3fffffff]
[    6.336967] pci 0002:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[    6.337640] pci 0002:00:00.0: supports D1 D2
[    6.338027] pci 0002:00:00.0: PME# supported from D0 D1 D3hot
[    6.346675] pci_bus 0002:01: busn_res: can not insert [bus 01-ff] under [bus 00-0f] (conflicts with (null) [bus 00-0f])
[    6.348052] pci 0002:00:00.0: BAR 0: assigned [mem 0x380000000-0x3bfffffff]
[    6.348699] pci 0002:00:00.0: BAR 1: no space for [mem size 0x40000000]
[    6.349291] pci 0002:00:00.0: BAR 1: failed to assign [mem size 0x40000000]
[    6.349912] pci 0002:00:00.0: BAR 6: assigned [mem 0xf0200000-0xf020ffff pref]
[    6.350560] pci 0002:00:00.0: PCI bridge to [bus 01-ff]
[    6.353475] pcieport 0002:00:00.0: PME: Signaling with IRQ 46
[    6.354514] pcieport 0002:00:00.0: AER: enabled with IRQ 46

ls -al /dev/nvme0*
ls: cannot access '/dev/nvme0*': No such file or directory

echo 1 > /sys/bus/pci/rescan
ls -al /dev/nvme0*
crw------- 1 root root 509, 0 Feb 28 11:48 /dev/nvme0

arbv commented 1 year ago

@ftr3g FWIW, At least WD Blue SN570 (a generation newer than yours, but middle consumer end drive, compared to your higher end one) seems to work fine, at least with the latest official Fedora Workstation image booted in UEFI mode (see #6). I am not sure if UEFI mode changes anything PCI-related, making it to work.

ftr3g commented 1 year ago

I must receive Crucial P3 4To early next month, I will test.

arbv commented 1 year ago

@inindev I have noticed that you have updated the bootloader. So, does it support booting from NVME now?

inindev commented 1 year ago

Hi Artem,

No, I am not to that point yet. The update I posted is to get ready for the Debain 12 GA scheduled for 6/10/23. My focus has been to get the Nanopi R5S to boot to NVMe, which I did to my satisfaction: https://github.com/inindev/nanopi-r5#booting-from-m2-nvme

I do intend to look into this in the month of June and would value your feedback as I make progress. I ran centos for years and only switched to debian in 2019 as I wanted to start using netfilter instead of iptables (the fedora kernel at the time did not support netfilter).

arbv commented 1 year ago

Aha, I see. So I will stick to the initial plan of having /boot on eMMC. That should work.

Of course, I will provide you with the feedback should I spot something interesting (or "interesting"). At the same time, my plan is to bootstrap NixOS on the device, not Debian. Yesterday (late at night) I managed to produce a custom SD image which booted fine, so I am halfway there. However, I had to change a couple of kernel configuration options in order to get HDMI and PCI working. I have to report these to them, I guess. That is a bummer, because it means that kernel will automatically recompile on update (Gentoo style), but still - it is vanilla mainline kernel, so that is great anyway.

Thank you for your work!

Hi Artem,

No, I am not to that point yet. The update I posted is to get ready for the Debain 12 GA scheduled for 6/10/23. My focus has been to get the Nanopi R5S to boot to NVMe, which I did to my satisfaction: https://github.com/inindev/nanopi-r5#booting-from-m2-nvme

I do intend to look into this in the month of June and would value your feedback as I make progress. I ran centos for years and only switched to debian in 2019 as I wanted to start using netfilter instead of iptables (the fedora kernel at the time did not support netfilter).

arbv commented 1 year ago

@inindev BTW, I know that there is a Quartz64 UEFI port for this board (R5S).

https://github.com/S199pWa1k9r/rk356x-uefi/tree/nanopi-r5s

I haven't tested it yet, though.

Some boards supported by Quartz64 do have PCI support and they claim to have it working, too. Maybe you will find something what you may need to have support for booting from NVME in your patchset for u-boot.

arbv commented 1 year ago

@inindev

I had to remove nvme from boot targets in this u-boot version as when I boot with SD card inserted, eMMC is not writable from the loaded OS, making it impossible to update the bootloader (and write any data at to it) I am getting async write errors. That does not happen when I boot from USB, but in this version you prioritised NVME (which is not ready yet) over USB, making the device unbootable.

So, I bricked the device for now and have no idea how to flash the bootloader as I have no cable to do so over USB.

inindev commented 1 year ago

Thanks for the report, hold the mask button down and it will bootstrap u-boot from the SD card.

I know what is happening, the fix is here: https://github.com/inindev/nanopi-r5/commit/c16bb18e63daee1c87bf897edbfcd69070f88c63

Building an RC6 now...

arbv commented 1 year ago

Yeah, I know that when I boot from SD card, then mmcblk0 is unwritable - that is, I cannot replace the bootloader on eMMC.

arbv commented 1 year ago

@inindev can you consider prioritising USB over NVME. IMO, it makes more sense to try removable media first (for emergency purposes).

inindev commented 1 year ago

Yeah, I know that when I boot from SD card, then mmcblk0 is unwritable - that is, I cannot replace the bootloader on eMMC.

When you boot, the external mmc is /dev/mmcblk0 and the internal emmc is /dev/mmcblk1, right?

inindev commented 1 year ago

@inindev can you consider prioritising USB over NVME. IMO, it makes more sense to try removable media first (for emergency purposes).

I see what you are telling me now, this is messing you up: https://github.com/inindev/nanopi-r5/blob/main/uboot/patches/0011-add-nvme-boot-target.patch

I agree, changing this...

inindev commented 1 year ago

It will take me an hour or so to patch and publish an rc6 addressing this. In the mean time, using the u-boot files from rc4 should address your issue: https://github.com/inindev/nanopi-r5/releases/tag/v12-rc4

arbv commented 1 year ago

Yeah, I know that when I boot from SD card, then mmcblk0 is unwritable - that is, I cannot replace the bootloader on eMMC.

When you boot, the external mmc is /dev/mmcblk0 and the internal emmc is /dev/mmcblk1, right?

Hm, it does not seem to be the case (with mask pressed, I mean).

arbv commented 1 year ago

@inindev can you consider prioritising USB over NVME. IMO, it makes more sense to try removable media first (for emergency purposes).

I see what you are telling me now, this is messing you up: https://github.com/inindev/nanopi-r5/blob/main/uboot/patches/0011-add-nvme-boot-target.patch

I agree, changing this...

Yeah, I have patched it, but have problems to flash a new version of u-boot into eMMC as it does not boot from USB and when booting from SD, eMMC is not writeable. I think I need to buy USB-A to USB-A cable to fix the device (will likely take a couple of days).

inindev commented 1 year ago

Do you have a serial monitor on your r5s? The image I produce has an expand filesystem one-time step which triggers a reboot when the mmc is first used. This can be confusing as if you dont know it is rebooting then it will reboot using the internal emmc. I describe a work-around on the nvme instructions:

boot from mmc

Hold down the mask button while powering on. After 5 seconds the mask button can be released and the device will run initial setup then reboot (this will only happen one time). Note that without a serial terminal it will be difficult to know when the reboot has completed. Waiting two minutes then powering down and booting again with the mask button again should be sufficient to reach the second boot.

arbv commented 1 year ago

@inindev Aha, I see. That is, basically, boot the device with Mask button pressed and then reboot with Mask button pressed and reflash from SD card. Seems to do the trick!

I am not using your images right now, but my build of NixOS, which I intended to use for installation. I have patched up rc5 to exclude NVME, will try to flash it now.

arbv commented 1 year ago

@inindev Or, better yet, I would rather wait for rc6 then, and flash rc4 for the time being (patched with EFI support) and then start installation.

arbv commented 1 year ago

@inindev A bit unrelated topic: but can you also consider enabling UEFI support in your u-boot builds? That would make booting (some) official distro images much simpler.

inindev commented 1 year ago

Yes, I just need to get a chance to verify that it wont break anything else -- especially since I want to start supporting CentOS.

arbv commented 1 year ago

@inindev Sure thing. I am making the suggestion from my very limiting perspective and wearing my highly specialised tunnel vision glasses :)

inindev commented 1 year ago

@inindev Or, better yet, I would rather wait for rc6 then, and flash rc4 for the time being (patched with EFI support) and then start installation.

rc6 is published and no longer has nvme in the boot order

arbv commented 1 year ago

@inindev ~~Aaargh, I cannot seem to make eMMC writeable the second time.~~ Done.

arbv commented 1 year ago

@inindev Or, better yet, I would rather wait for rc6 then, and flash rc4 for the time being (patched with EFI support) and then start installation.

rc6 is published and no longer has nvme in the boot order

Thanks, will try to flash it, too.

arbv commented 1 year ago

@inindev Could you explain (in lame terms, at least) why eMMC was non-writeable? I think that it has happened because I was booting the device without the proper DTB specified, it seems to have worked fine once I specified it in the bootloader.

arbv commented 1 year ago

@inindev RC6 works fine. The device boots from both SD and USB just fine, just like before :+1: Thank you!

inindev / nanopi-r5

NVMe Support #7