inindev / nanopi-r5

stock debian arm64 linux for the nanopi r5c & r5s
GNU General Public License v3.0
100 stars 17 forks source link

NVME is not functional #23

Open svpcom opened 11 months ago

svpcom commented 11 months ago

After boot no NVME devices found. If I trigger pci rescan via echo 1 > /sys/bus/pci/rescan NVMe device is found but non-functional:

[   46.034631] pcieport 0002:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
[   46.035741] pci 0002:01:00.0: [15b7:5006] type 00 class 0x010802
[   46.036483] pci 0002:01:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
[   46.037312] pci 0002:01:00.0: reg 0x20: [mem 0x00000000-0x000000ff 64bit]
[   46.039350] pci 0002:01:00.0: 2.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x1 link at 0002:00:00.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
[   46.052298] pci_bus 0002:01: busn_res: [bus 01-ff] end is updated to 01
[   46.053043] pcieport 0002:00:00.0: BAR 14: assigned [mem 0xf0300000-0xf03fffff]
[   46.053764] pci 0002:01:00.0: BAR 0: assigned [mem 0xf0300000-0xf0303fff 64bit]
[   46.054522] pci 0002:01:00.0: BAR 4: assigned [mem 0xf0304000-0xf03040ff 64bit]
[   46.097927] nvme nvme0: pci function 0002:01:00.0
[   46.098444] nvme 0002:01:00.0: enabling device (0000 -> 0002)
[  107.508575] nvme nvme0: I/O 12 QID 0 timeout, completion polled
[  168.958145] nvme nvme0: I/O 8 QID 0 timeout, completion polled
[  230.391719] nvme nvme0: I/O 13 QID 0 timeout, completion polled
[  242.664114] INFO: task kworker/u8:4:167 blocked for more than 120 seconds.
[  242.664761]       Not tainted 6.1.0-12-arm64 #1 Debian 6.1.52-1
[  242.665290] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  242.665983] task:kworker/u8:4    state:D stack:0     pid:167   ppid:2      flags:0x00000008
[  242.666733] Workqueue: events_unbound async_run_entry_fn
[  242.667225] Call trace:
[  242.667449]  __switch_to+0xf0/0x170
[  242.667771]  __schedule+0x340/0x940
[  242.668129]  schedule+0x58/0xf0
[  242.668421]  schedule_timeout+0x14c/0x180
[  242.668785]  __wait_for_common+0xd4/0x254
[  242.669148]  wait_for_completion+0x28/0x3c
[  242.669519]  __flush_work.isra.0+0x180/0x2dc
[  242.669907]  flush_work+0x18/0x2c
[  242.670209]  nvme_async_probe+0x24/0x44 [nvme]
[  242.670636]  async_run_entry_fn+0x40/0x1e0
[  242.671010]  process_one_work+0x1f4/0x460
[  242.671375]  worker_thread+0x188/0x4e0
[  242.671715]  kthread+0xe0/0xe4
[  242.672025]  ret_from_fork+0x10/0x20
[  242.672407] INFO: task (udev-worker):706 blocked for more than 120 seconds.
[  242.673027]       Not tainted 6.1.0-12-arm64 #1 Debian 6.1.52-1
[  242.673554] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  242.674245] task:(udev-worker)   state:D stack:0     pid:706   ppid:277    flags:0x0000080d
[  242.674992] Call trace:
[  242.675214]  __switch_to+0xf0/0x170
[  242.675535]  __schedule+0x340/0x940
[  242.675851]  schedule+0x58/0xf0
[  242.676169]  async_synchronize_cookie_domain+0xe8/0x150
[  242.676645]  async_synchronize_full+0x20/0x30
[  242.677041]  do_init_module+0x160/0x1f4
[  242.677390]  load_module+0x1cb8/0x2220
[  242.677730]  __do_sys_finit_module+0xac/0x130
[  242.678123]  __arm64_sys_finit_module+0x28/0x34
[  242.678530]  invoke_syscall+0x78/0x100
[  242.678874]  el0_svc_common.constprop.0+0xd4/0xf4
[  242.679299]  do_el0_svc+0x34/0xd0
[  242.679601]  el0_svc+0x34/0xd4
[  242.679905]  el0t_64_sync_handler+0xf4/0x120
[  242.680298]  el0t_64_sync+0x18c/0x190
[  263.763473] systemd-journald[243]: Oldest entry in /var/log/journal/69fd1ff5339ee7c1931e562d64ac7bcc/system.journal is older than the configured file retention duration (1month), suggesting rotation.
[  263.765169] systemd-journald[243]: /var/log/journal/69fd1ff5339ee7c1931e562d64ac7bcc/system.journal: Journal header limits reached or header out-of-date, rotating.
[  291.833349] nvme nvme0: I/O 9 QID 0 timeout, completion polled
[  353.274913] nvme nvme0: I/O 14 QID 0 timeout, completion polled
[  353.276365] hwmon hwmon2: temp1_input not attached to any thermal zone
[  363.499229] INFO: task kworker/u8:4:167 blocked for more than 241 seconds.
[  363.499892]       Not tainted 6.1.0-12-arm64 #1 Debian 6.1.52-1
[  363.500427] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  363.501129] task:kworker/u8:4    state:D stack:0     pid:167   ppid:2      flags:0x00000008
[  363.501890] Workqueue: events_unbound async_run_entry_fn
[  363.502394] Call trace:
[  363.502623]  __switch_to+0xf0/0x170
[  363.502956]  __schedule+0x340/0x940
[  363.503329]  schedule+0x58/0xf0
[  363.503631]  schedule_timeout+0x14c/0x180
[  363.504004]  __wait_for_common+0xd4/0x254
[  363.504377]  wait_for_completion+0x28/0x3c
[  363.504756]  __flush_work.isra.0+0x180/0x2dc
[  363.505154]  flush_work+0x18/0x2c
[  363.505465]  nvme_async_probe+0x24/0x44 [nvme]
[  363.505910]  async_run_entry_fn+0x40/0x1e0
[  363.506294]  process_one_work+0x1f4/0x460
[  363.506669]  worker_thread+0x188/0x4e0
[  363.507059]  kthread+0xe0/0xe4
[  363.507353]  ret_from_fork+0x10/0x20
[  363.507742] INFO: task (udev-worker):706 blocked for more than 241 seconds.
[  363.508370]       Not tainted 6.1.0-12-arm64 #1 Debian 6.1.52-1
[  363.508903] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  363.509602] task:(udev-worker)   state:D stack:0     pid:706   ppid:277    flags:0x0000080d
[  363.510359] Call trace:
[  363.510588]  __switch_to+0xf0/0x170
[  363.510916]  __schedule+0x340/0x940
[  363.511289]  schedule+0x58/0xf0
[  363.511590]  async_synchronize_cookie_domain+0xe8/0x150
[  363.512073]  async_synchronize_full+0x20/0x30
[  363.512478]  do_init_module+0x160/0x1f4
[  363.512834]  load_module+0x1cb8/0x2220
[  363.513182]  __do_sys_finit_module+0xac/0x130
[  363.513584]  __arm64_sys_finit_module+0x28/0x34
[  363.514002]  invoke_syscall+0x78/0x100
[  363.514355]  el0_svc_common.constprop.0+0xd4/0xf4
[  363.514791]  do_el0_svc+0x34/0xd0
[  363.515144]  el0_svc+0x34/0xd4
[  363.515442]  el0t_64_sync_handler+0xf4/0x120
[  363.515841]  el0t_64_sync+0x18c/0x190
[  414.724449] nvme nvme0: I/O 10 QID 0 timeout, completion polled
[  476.157964] nvme nvme0: I/O 15 QID 0 timeout, completion polled
[  484.334241] INFO: task kworker/u8:4:167 blocked for more than 362 seconds.
[  484.334908]       Not tainted 6.1.0-12-arm64 #1 Debian 6.1.52-1
[  484.335443] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  484.336141] task:kworker/u8:4    state:D stack:0     pid:167   ppid:2      flags:0x00000008
[  484.336903] Workqueue: events_unbound async_run_entry_fn
[  484.337409] Call trace:
[  484.337638]  __switch_to+0xf0/0x170
[  484.337968]  __schedule+0x340/0x940
[  484.338345]  schedule+0x58/0xf0
[  484.338647]  schedule_timeout+0x14c/0x180
[  484.339020]  __wait_for_common+0xd4/0x254
[  484.339394]  wait_for_completion+0x28/0x3c
[  484.339773]  __flush_work.isra.0+0x180/0x2dc
[  484.340170]  flush_work+0x18/0x2c
[  484.340481]  nvme_async_probe+0x24/0x44 [nvme]
[  484.340926]  async_run_entry_fn+0x40/0x1e0
[  484.341311]  process_one_work+0x1f4/0x460
[  484.341684]  worker_thread+0x188/0x4e0
[  484.342078]  kthread+0xe0/0xe4
[  484.342373]  ret_from_fork+0x10/0x20
[  484.342766] INFO: task (udev-worker):706 blocked for more than 362 seconds.
[  484.343397]       Not tainted 6.1.0-12-arm64 #1 Debian 6.1.52-1
[  484.343930] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  484.344629] task:(udev-worker)   state:D stack:0     pid:706   ppid:277    flags:0x0000080d
[  484.345387] Call trace:
[  484.345616]  __switch_to+0xf0/0x170
[  484.345944]  __schedule+0x340/0x940
[  484.346321]  schedule+0x58/0xf0
[  484.346624]  async_synchronize_cookie_domain+0xe8/0x150
[  484.347109]  async_synchronize_full+0x20/0x30
[  484.347514]  do_init_module+0x160/0x1f4
[  484.347870]  load_module+0x1cb8/0x2220
[  484.348217]  __do_sys_finit_module+0xac/0x130
[  484.348616]  __arm64_sys_finit_module+0x28/0x34
[  484.349033]  invoke_syscall+0x78/0x100
[  484.349386]  el0_svc_common.constprop.0+0xd4/0xf4
[  484.349820]  do_el0_svc+0x34/0xd0
[  484.350175]  el0_svc+0x34/0xd4
[  484.350473]  el0t_64_sync_handler+0xf4/0x120
[  484.350874]  el0t_64_sync+0x18c/0x190
[  537.599512] nvme nvme0: I/O 11 QID 0 timeout, completion polled

NVME device is WD Red SN700

inindev commented 11 months ago

Others have reported problems with Western Digital NVMe's as well: https://github.com/inindev/nanopi-r5/issues/7

So far the Samsung NVMe are the only known working devices with this build.

svpcom commented 11 months ago

@inindev Is the problem in kernel or u-boot?

inindev commented 11 months ago

Do you have a serial console to test u-boot? As I recall the previous report did not. I would like to see the u-boot output of this:

pci enum
nvme scan
nvme dev 0

Device 0: Vendor: 0x15b7 Rev: 21113012 Prod: 21442U804561
            Type: Hard Disk
            Capacity: 244198.3 MB = 238.4 GB (500118192 x 512)
... is now current device
svpcom commented 11 months ago

@inindev Yes, I'll test and report this evening

svpcom commented 11 months ago

Image was build from master branch (commit 1c61cfba309e605025d1f5ca73c00b79e15ce97d) Kernel 6.1.0-12-arm64

svpcom commented 11 months ago

@inindev

DDR V1.15 ec2fae0c96 huan.he 22/11/08-20:06:42
In
ddrconfig:0
LPDDR4X, 324MHz
BW=32 Col=10 Bk=8 CS0 Row=17 CS=1 Die BW=16 Size=4096MB
tdqss: cs0 dqs0: 24ps, dqs1: -24ps, dqs2: 0ps, dqs3: -96ps, 

change to: 324MHz
PHY drv:clk:36,ca:36,DQ:29,odt:0
vrefinner:24%, vrefout:41%
dram drv:40,odt:0
clk skew:0x61

change to: 528MHz
PHY drv:clk:36,ca:36,DQ:29,odt:0
vrefinner:24%, vrefout:41%
dram drv:40,odt:0
clk skew:0x58

change to: 780MHz
PHY drv:clk:36,ca:36,DQ:29,odt:0
vrefinner:24%, vrefout:41%
dram drv:40,odt:0
clk skew:0x58

change to: 1560MHz(final freq)
PHY drv:clk:36,ca:36,DQ:29,odt:60
vrefinner:16%, vrefout:22%
dram drv:40,odt:80
vref_ca:00000071
clk skew:0x13
cs 0:
the read training result:
DQS0:0x2e, DQS1:0x35, DQS2:0x36, DQS3:0x32, 
min  : 0xc  0xb  0xd  0xa  0x2  0x7  0x9  0x3 , 0xd  0x9  0x3  0x2  0xf  0xc  0xf  0xa ,
      0x12 0x14  0xc  0xc  0x4  0x1  0x2  0x5 , 0xc  0x9  0x9  0x3  0xc  0xe  0xc  0xc ,
mid  :0x28 0x27 0x29 0x26 0x1e 0x23 0x25 0x1e ,0x28 0x25 0x1f 0x1f 0x2b 0x27 0x29 0x25 ,
      0x30 0x31 0x28 0x28 0x22 0x1f 0x20 0x22 ,0x29 0x26 0x25 0x21 0x2a 0x2b 0x2a 0x2a ,
max  :0x45 0x43 0x45 0x43 0x3a 0x3f 0x42 0x3a ,0x43 0x42 0x3c 0x3c 0x47 0x42 0x44 0x41 ,
      0x4e 0x4e 0x45 0x45 0x40 0x3d 0x3e 0x40 ,0x47 0x44 0x42 0x3f 0x48 0x48 0x48 0x48 ,
range:0x39 0x38 0x38 0x39 0x38 0x38 0x39 0x37 ,0x36 0x39 0x39 0x3a 0x38 0x36 0x35 0x37 ,
      0x3c 0x3a 0x39 0x39 0x3c 0x3c 0x3c 0x3b ,0x3b 0x3b 0x39 0x3c 0x3c 0x3a 0x3c 0x3c ,
the write training result:
DQS0:0x17, DQS1:0xf, DQS2:0x13, DQS3:0x0, 
min  :0x63 0x67 0x69 0x67 0x5d 0x60 0x61 0x60 0x60 ,0x57 0x55 0x4e 0x4c 0x5c 0x58 0x57 0x57 0x52 ,
      0x5f 0x5e 0x5b 0x5a 0x51 0x4f 0x4f 0x54 0x55 ,0x47 0x44 0x45 0x40 0x4b 0x4b 0x47 0x4d 0x43 ,
mid  :0x7e 0x81 0x83 0x81 0x77 0x7a 0x7c 0x79 0x79 ,0x71 0x6e 0x66 0x65 0x76 0x71 0x71 0x6e 0x6b ,
      0x7a 0x7a 0x74 0x73 0x6c 0x68 0x6a 0x6d 0x6f ,0x62 0x5f 0x5e 0x5a 0x65 0x65 0x61 0x66 0x5d ,
max  :0x9a 0x9c 0x9d 0x9b 0x91 0x95 0x97 0x92 0x93 ,0x8b 0x88 0x7f 0x7f 0x90 0x8a 0x8b 0x86 0x84 ,
      0x96 0x97 0x8d 0x8d 0x88 0x81 0x85 0x87 0x8a ,0x7d 0x7a 0x77 0x74 0x80 0x7f 0x7b 0x7f 0x77 ,
range:0x37 0x35 0x34 0x34 0x34 0x35 0x36 0x32 0x33 ,0x34 0x33 0x31 0x33 0x34 0x32 0x34 0x2f 0x32 ,
      0x37 0x39 0x32 0x33 0x37 0x32 0x36 0x33 0x35 ,0x36 0x36 0x32 0x34 0x35 0x34 0x34 0x32 0x34 ,
CA Training result:
cs:0 min  :0x3b 0x37 0x3f 0x37 0x3e 0x35 0x40 ,0x40 0x37 0x3d 0x38 0x3e 0x37 0x41 ,
cs:0 mid  :0x77 0x76 0x7a 0x76 0x7a 0x77 0x6c ,0x7b 0x77 0x79 0x76 0x7a 0x78 0x6f ,
cs:0 max  :0xb3 0xb5 0xb5 0xb5 0xb7 0xb9 0x98 ,0xb7 0xb7 0xb5 0xb5 0xb7 0xb9 0x9d ,
cs:0 range:0x78 0x7e 0x76 0x7e 0x79 0x84 0x58 ,0x77 0x80 0x78 0x7d 0x79 0x82 0x5c ,
out

U-Boot SPL 2023.07.02-00042-gfb4063c8ad (Sep 21 2023 - 11:40:19 +0000)
Trying to boot from MMC1
## Checking hash(es) for config config-1 ... OK
## Checking hash(es) for Image atf-1 ... sha256+ OK
## Checking hash(es) for Image u-boot ... sha256+ OK
## Checking hash(es) for Image fdt-1 ... sha256+ OK
## Checking hash(es) for Image atf-2 ... sha256+ OK
## Checking hash(es) for Image atf-3 ... sha256+ OK
INFO:    Preloader serial: 2
NOTICE:  BL31: v2.3():v2.3-152-g4e725b15f:cl
NOTICE:  BL31: Built : 10:51:13, Jul 15 2021
INFO:    GICv3 without legacy support detected.
INFO:    ARM GICv3 driver initialized in EL3
INFO:    pmu v1 is valid
INFO:    dfs DDR fsp_param[0].freq_mhz= 1560MHz
INFO:    dfs DDR fsp_param[1].freq_mhz= 324MHz
INFO:    dfs DDR fsp_param[2].freq_mhz= 528MHz
INFO:    dfs DDR fsp_param[3].freq_mhz= 780MHz
INFO:    Using opteed sec cpu_context!
INFO:    boot cpu mask: 0
INFO:    BL31: Initializing runtime services
WARNING: No OPTEE provided by BL2 boot loader, Booting device without OPTEE initialization. SMC`s destined for OPTEE will return SMC_UNK
ERROR:   Error initializing runtime service opteed_fast
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0xa00000
INFO:    SPSR = 0x3c9

U-Boot 2023.07.02-00042-gfb4063c8ad (Sep 21 2023 - 11:40:19 +0000)

Model: FriendlyElec NanoPi R5S
DRAM:  4 GiB (effective 3.7 GiB)
PMIC:  RK8090 (on=0x40, off=0x00)
Core:  318 devices, 26 uclasses, devicetree: separate
MMC:   mmc@fe2b0000: 1, mmc@fe310000: 0
Loading Environment from nowhere... OK
In:    serial@fe660000
Out:   serial@fe660000
Err:   serial@fe660000
Model: FriendlyElec NanoPi R5S
Net:   No ethernet found.
Hit any key to stop autoboot:  0 
=> 
=> pci enum
=> 
=> nvme scan
=> 
=> nvme dev 0

Device 0: Vendor: 0x15b7 Rev: 112000WD Prod: 23041F800808        
            Type: Hard Disk
            Capacity: 1907729.0 MB = 1863.0 GB (3907029168 x 512)
... is now current device
=> 

Device 0: Vendor: 0x15b7 Rev: 112000WD Prod: 23041F800808        
            Type: Hard Disk
            Capacity: 1907729.0 MB = 1863.0 GB (3907029168 x 512)
... is now current device
=> 
svpcom commented 11 months ago

@inindev boot of debian 11 from friendlyarm where NVME function properly:

DDR V1.18 f366f69a7d typ 23/07/17-15:48:58
In
LP4/4x derate en, other dram:1x trefi
ddrconfig:0
LPDDR4X, 324MHz
BW=32 Col=10 Bk=8 CS0 Row=17 CS=1 Die BW=16 Size=4096MB
tdqss: cs0 dqs0: 24ps, dqs1: -24ps, dqs2: -24ps, dqs3: -96ps, 

change to: 324MHz
clk skew:0x61

change to: 528MHz
clk skew:0x58

change to: 780MHz
clk skew:0x58

change to: 1560MHz(final freq)
PHY drv:clk:36,ca:36,DQ:29,odt:60
vrefinner:16%, vrefout:22%
dram drv:40,odt:80
vref_ca:00000071
clk skew:0x13
cs 0:
the read training result:
DQS0:0x2f, DQS1:0x36, DQS2:0x38, DQS3:0x32, 
min  : 0xc  0xa  0xd  0xa  0x2  0x7  0xa  0x4 , 0xd  0x9  0x1  0x1  0xf  0xc  0xf  0xa ,
      0x14 0x14  0xd  0xc  0x4  0x2  0x3  0x5 , 0xa  0x6  0x7  0x1  0xd  0xd  0xa  0xb ,
mid  :0x29 0x27 0x29 0x27 0x1f 0x23 0x27 0x20 ,0x28 0x26 0x1e 0x1e 0x2c 0x27 0x2a 0x25 ,
      0x32 0x32 0x2a 0x29 0x23 0x20 0x21 0x23 ,0x28 0x24 0x24 0x1f 0x2a 0x2a 0x29 0x2a ,
max  :0x46 0x45 0x46 0x44 0x3c 0x40 0x44 0x3c ,0x44 0x43 0x3c 0x3c 0x49 0x43 0x45 0x41 ,
      0x50 0x51 0x48 0x47 0x43 0x3f 0x40 0x42 ,0x47 0x43 0x42 0x3e 0x48 0x48 0x49 0x49 ,
range:0x3a 0x3b 0x39 0x3a 0x3a 0x39 0x3a 0x38 ,0x37 0x3a 0x3b 0x3b 0x3a 0x37 0x36 0x37 ,
      0x3c 0x3d 0x3b 0x3b 0x3f 0x3d 0x3d 0x3d ,0x3d 0x3d 0x3b 0x3d 0x3b 0x3b 0x3f 0x3e ,
the write training result:
DQS0:0x17, DQS1:0xf, DQS2:0xf, DQS3:0x0, 
min  :0x64 0x67 0x69 0x67 0x5d 0x60 0x61 0x60 0x60 ,0x57 0x54 0x4e 0x4d 0x5d 0x57 0x57 0x58 0x52 ,
      0x5c 0x5b 0x57 0x56 0x4e 0x4b 0x4c 0x50 0x51 ,0x48 0x44 0x45 0x40 0x4b 0x4c 0x47 0x4e 0x44 ,
mid  :0x7f 0x81 0x83 0x81 0x77 0x7b 0x7c 0x79 0x79 ,0x70 0x6e 0x66 0x66 0x76 0x70 0x70 0x6f 0x6a ,
      0x77 0x77 0x70 0x70 0x69 0x65 0x66 0x6a 0x6b ,0x63 0x5f 0x5e 0x5a 0x66 0x65 0x61 0x66 0x5e ,
max  :0x9b 0x9c 0x9e 0x9c 0x91 0x96 0x97 0x93 0x93 ,0x8a 0x88 0x7f 0x7f 0x8f 0x89 0x8a 0x86 0x83 ,
      0x93 0x93 0x89 0x8a 0x85 0x7f 0x81 0x84 0x86 ,0x7e 0x7a 0x78 0x75 0x81 0x7f 0x7c 0x7f 0x78 ,
range:0x37 0x35 0x35 0x35 0x34 0x36 0x36 0x33 0x33 ,0x33 0x34 0x31 0x32 0x32 0x32 0x33 0x2e 0x31 ,
      0x37 0x38 0x32 0x34 0x37 0x34 0x35 0x34 0x35 ,0x36 0x36 0x33 0x35 0x36 0x33 0x35 0x31 0x34 ,
CA Training result:
cs:0 min  :0x3b 0x37 0x3e 0x37 0x3e 0x36 0x40 ,0x40 0x37 0x3e 0x37 0x3f 0x37 0x42 ,
cs:0 mid  :0x77 0x76 0x7a 0x76 0x7a 0x77 0x6c ,0x7b 0x77 0x79 0x76 0x7b 0x77 0x6f ,
cs:0 max  :0xb3 0xb5 0xb6 0xb5 0xb7 0xb9 0x99 ,0xb7 0xb7 0xb5 0xb5 0xb7 0xb8 0x9d ,
cs:0 range:0x78 0x7e 0x78 0x7e 0x79 0x83 0x59 ,0x77 0x80 0x77 0x7e 0x78 0x81 0x5b ,
out
U-Boot SPL board init
U-Boot SPL 2017.09-g82417168bf-230508 #root (Sep 08 2023 - 21:25:39)
unknown raw ID 0 0 0
unrecognized JEDEC id bytes: 00, 00, 00
Trying to boot from MMC2
SPL: A/B-slot: _a, successful: 0, tries-remain: 7
Trying fit image at 0x4000 sector
## Verified-boot: 0
## Checking atf-1 0x00040000 ... sha256(0d5225a4ab...) + OK
## Checking uboot 0x00a00000 ... sha256(7af96391a0...) + OK
## Checking fdt 0x00b3f518 ... sha256(8afdf6bf4d...) + OK
## Checking atf-2 0xfdcc1000 ... sha256(3e94d16e6a...) + OK
## Checking atf-3 0x0006b000 ... sha256(fde0ef262b...) + OK
## Checking atf-4 0xfdcce000 ... sha256(c9eb312bf2...) + OK
## Checking atf-5 0xfdcd0000 ... sha256(befba422b8...) + OK
## Checking atf-6 0x00069000 ... sha256(6ede7a3b44...) + OK
## Checking optee 0x08400000 ... sha256(4fcbcd3870...) + OK
Jumping to U-Boot(0x00a00000) via ARM Trusted Firmware(0x00040000)
Total: 284.903/401.525 ms

INFO:    Preloader serial: 2
NOTICE:  BL31: v2.3():v2.3-607-gbf602aff1:cl
NOTICE:  BL31: Built : 10:16:03, Jun  5 2023
INFO:    GICv3 without legacy support detected.
INFO:    ARM GICv3 driver initialized in EL3
INFO:    pmu v1 is valid 220114
INFO:    dfs DDR fsp_param[0].freq_mhz= 1560MHz
INFO:    dfs DDR fsp_param[1].freq_mhz= 324MHz
INFO:    dfs DDR fsp_param[2].freq_mhz= 528MHz
INFO:    dfs DDR fsp_param[3].freq_mhz= 780MHz
INFO:    Using opteed sec cpu_context!
INFO:    boot cpu mask: 0
INFO:    BL31: Initializing runtime services
INFO:    BL31: Initializing BL32
I/TC: 
I/TC: OP-TEE version: 3.13.0-723-gdcfdd61d0 #hisping.lin (gcc version 10.2.1 20201103 (GNU Toolchain for the A-profile Architecture 10.2-2020.11 (arm-10.16))) #2 Wed Jun  7 09:43:57 CST 2023 aarch64
I/TC: Primary CPU initializing
I/TC: Primary CPU switching to normal world boot
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0xa00000
INFO:    SPSR = 0x3c9

U-Boot 2017.09-g82417168bf-230508 #root (Sep 08 2023 - 21:28:24 +0800)

Board: NanoPi R5S
PreSerial: 2, raw, 0xfe660000
DRAM:  4 GiB
Sysmem: init
Relocation Offset: ed220000
Relocation fdt: eb9f91e8 - eb9fecd0
CR: M/C/I
Using default environment

DM: v1
no mmc device at slot 1
dwmmc@fe2b0000: 1 (SD), dwmmc@fe2c0000: 2, sdhci@fe310000: 0
Bootdev(atags): mmc 1
MMC1: Legacy, 52Mhz
PartType: EFI
boot mode: None
RESC: 'resource', blk@0x0000a000
optee api revision: 2.0
TEEC: Waring: Could not find security partition
lib/avb/libavb_user/avb_ops_user.c: trusty_read_lock_state failed
Error determining whether device is unlocked.
Device is: UNLOCKED
DTB: rk3568-nanopi5-rev01.dtb (167486)
HASH(c): OK
ANDROID: fdt overlay OK
I2c0 speed: 400000Hz
PMIC:  RK8090 (on=0x40, off=0x00)
vdd_logic init 900000 uV
vdd_gpu init 900000 uV
vdd_npu init 900000 uV
io-domain: OK
INFO:    ddr dmc_fsp already initialized in loader.
get vp0 plane mask:0xf, primary id:2, cursor_plane:3, from dts
get vp1 plane mask:0x30, primary id:4, cursor_plane:-1, from dts
Could not find baseparameter partition
I2c5 speed: 400000Hz
Rockchip UBOOT DRM driver version: v1.0.1
vp0 have layer nr:4[0 1 2 3 ], primary plane: 2
vp1 have layer nr:2[4 5 ], primary plane: 4
vp2 have layer nr:0[], primary plane: 0
xfer: num: 2, addr: 0x50
xfer: num: 2, addr: 0x50
Monitor has basic audio support
Could not find baseparameter partition
hdmi@fe0a0000:  detailed mode clock 76000 kHz, flags[5]
    H: 1280 1352 1480 1680
    V: 0720 0723 0729 0751
bus_format: 2025
VOP update mode to: 1280x720p60, type: HDMI0 for VP0
VP0 set crtc_clock to 76000KHz
VOP VP0 enable Esmart0[800x480->800x480@240x120] fmt[1] addr[0xee01a000]
Non-CEA mode used in HDMI
final pixclk = 76000000 tmdsclk = 76000000
PHY powered down in 0 iterations
PHY PLL locked 1 iterations
PHY powered down in 1 iterations
PHY PLL locked 1 iterations
sink has audio support
Rate 76000000 missing; compute N dynamically
hdmi_set_clk_regenerator: fs=48000Hz ftdms=76.000MHz N=6144 cts=76000
CLK: (sync kernel. arm: enter 816000 KHz, init 816000 KHz, kernel 0N/A)
  apll 1104000 KHz
  dpll 780000 KHz
  gpll 1188000 KHz
  cpll 1000000 KHz
  npll 1200000 KHz
  vpll 24000 KHz
  hpll 76000 KHz
  ppll 200000 KHz
  armclk 1104000 KHz
  aclk_bus 150000 KHz
  pclk_bus 100000 KHz
  aclk_top_high 500000 KHz
  aclk_top_low 400000 KHz
  hclk_top 150000 KHz
  pclk_top 100000 KHz
  aclk_perimid 300000 KHz
  hclk_perimid 150000 KHz
  pclk_pmu 100000 KHz
vdd_usbc 5314 mV

Net:   eth0: ethernet@fe2a0000
Hit key to stop autoboot('CTRL+C'):  0 
## Booting FIT Image FIT: No fit blob
FIT: No FIT image
ANDROID: reboot reason: "(none)"
Not AVB images, AVB skip
No valid android hdr
Android image load failed
Android boot failed, error -1.

## Booting Rockchip Format Image
fdt  @ 0x08300000 (0x000290ce)
kernel   @ 0x00280000 (0x01fd8808)
ramdisk  @ 0x0a200000 (0x007b2bc0)
Fdt Ramdisk skip relocation
## Flattened Device Tree blob at 0x08300000
   Booting using the fdt blob at 0x08300000
   Using Device Tree in place at 0000000008300000, end 000000000832c0cd
WARNING: could not set reg FDT_ERR_BADOFFSET.
## reserved-memory:
  drm-logo@00000000: addr=edf00000 size=468000
  drm-cubic-lut@00000000: addr=eff00000 size=8000
  linux,cma: addr=10000000 size=800000
  ramoops@110000: addr=110000 size=f0000
Adding bank: 0x00200000 - 0x08400000 (size: 0x08200000)
Adding bank: 0x09400000 - 0xf0000000 (size: 0xe6c00000)
Adding bank: 0x1f0000000 - 0x200000000 (size: 0x10000000)
Total: 5239.604/5282.375 ms

Starting kernel ...

[    5.288230] Booting Linux on physical CPU 0x0000000000 [0x412fd050]
[    5.288255] Linux version 5.10.160 (root@jensen) (aarch64-linux-gcc (ctng-1.25.0-119g-FA) 11.3.0, GNU ld (GNU Binutils) 2.38) #2 SMP Fri Sep 1 10:30:30 CST 2023
[    5.293554] Machine model: FriendlyElec NanoPi R5S
[    5.316644] earlycon: uart8250 at MMIO32 0x00000000fe660000 (options '')
[    5.319673] printk: bootconsole [uart8250] enabled
[    5.321710] Reserved memory: created CMA memory pool at 0x0000000010000000, size 8 MiB
[    5.322458] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    5.382039] Zone ranges:
[    5.382293]   DMA      [mem 0x0000000000200000-0x00000000ffffffff]
[    5.382882]   DMA32    empty
[    5.383157]   Normal   [mem 0x0000000100000000-0x00000001ffffffff]
[    5.383741] Movable zone start for each node
[    5.384144] Early memory node ranges
[    5.384482]   node   0: [mem 0x0000000000200000-0x00000000083fffff]
[    5.385074]   node   0: [mem 0x0000000009400000-0x00000000efffffff]
[    5.385668]   node   0: [mem 0x00000001f0000000-0x00000001ffffffff]
[    5.386261] Initmem setup node 0 [mem 0x0000000000200000-0x00000001ffffffff]
[    5.414407] psci: probing for conduit method from DT.
[    5.414898] psci: PSCIv1.1 detected in firmware.
[    5.415338] psci: Using standard PSCI v0.2 function IDs
[    5.415834] psci: Trusted OS migration not required
[    5.416297] psci: SMC Calling Convention v1.2
[    5.417115] percpu: Embedded 31 pages/cpu s88808 r8192 d29976 u126976
[    5.417843] Detected VIPT I-cache on CPU0
[    5.418259] CPU features: detected: GIC system register CPU interface
[    5.418867] CPU features: detected: Virtualization Host Extensions
[    5.419452] CPU features: detected: Qualcomm erratum 1009, or ARM erratum 1286807
[    5.420161] CPU features: detected: ARM errata 1165522, 1319367, or 1530923
[    5.420838] alternatives: patching kernel code
[    5.422799] Built 1 zonelists, mobility grouping on.  Total pages: 1027656
[    5.423458] Kernel command line: storagemedia=sd androidboot.storagemedia=sd androidboot.mode=normal androidboot.dtbo_idx=1 androidboot.verifiedbootstate=orange earlycon=uart8250,mmio32,0xfe660000 console=ttyFIQ0 coherent_pool=1m rw root=/dev/mmcblk0p8 rootfstype=ext4 data=/dev/mmcblk0p9 consoleblank=0 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory swapaccount=1
[    5.427890] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear)
[    5.429133] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    5.429865] mem auto-init: stack:off, heap alloc:off, heap free:off
[    5.446652] software IO TLB: mapped [mem 0x00000000e9f00000-0x00000000edf00000] (64MB)
[    5.496821] Memory: 3974416K/4175872K available (16510K kernel code, 3362K rwdata, 5876K rodata, 6784K init, 880K bss, 193264K reserved, 8192K cma-reserved)
[    5.498265] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    5.498916] ftrace: allocating 54055 entries in 212 pages
[    5.577588] ftrace: allocated 212 pages with 4 groups
[    5.578406] rcu: Hierarchical RCU implementation.
[    5.578859] rcu:     RCU event tracing is enabled.
[    5.579288] rcu:     RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=4.
[    5.579905]  Rude variant of Tasks RCU enabled.
[    5.580332]  Tracing variant of Tasks RCU enabled.
[    5.580786] rcu: RCU calculated value of scheduler-enlistment delay is 30 jiffies.
[    5.581500] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
[    5.585091] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    5.587229] GICv3: GIC: Using split EOI/Deactivate mode
[    5.587735] GICv3: 320 SPIs implemented
[    5.588098] GICv3: 0 Extended SPIs implemented
[    5.588556] GICv3: Distributor has no Range Selector support
[    5.589098] GICv3: 16 PPIs implemented
[    5.589489] GICv3: CPU0: found redistributor 0 region 0:0x00000000fd460000
[    5.590257] ITS [mem 0xfd440000-0xfd45ffff]
[    5.590723] ITS@0x00000000fd440000: allocated 8192 Devices @2810000 (indirect, esz 8, psz 64K, shr 0)
[    5.591623] ITS@0x00000000fd440000: allocated 32768 Interrupt Collections @2820000 (flat, esz 2, psz 64K, shr 0)
[    5.592585] ITS: using cache flushing for cmd queue
[    5.593396] GICv3: using LPI property table @0x0000000002830000
[    5.594070] GIC: using cache flushing for LPI property table
[    5.594610] GICv3: CPU0: using allocated LPI pending table @0x0000000002840000
[    5.623830] arch_timer: cp15 timer(s) running at 24.00MHz (phys).
[    5.624420] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x588fe9dc0, max_idle_ns: 440795202592 ns
[    5.625442] sched_clock: 56 bits at 24MHz, resolution 41ns, wraps every 4398046511097ns
[    5.627272] Console: colour dummy device 80x25
[    5.627738] Calibrating delay loop (skipped), value calculated using timer frequency.. 48.00 BogoMIPS (lpj=80000)
[    5.628711] pid_max: default: 32768 minimum: 301
[    5.629236] LSM: Security Framework initializing
[    5.629739] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    5.630448] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[    5.632942] rcu: Hierarchical SRCU implementation.
[    5.634106] Platform MSI: interrupt-controller@fd440000 domain created
[    5.635192] PCI/MSI: /interrupt-controller@fd400000/interrupt-controller@fd440000 domain created
[    5.636875] smp: Bringing up secondary CPUs ...
I/TC: Secondary CPU 1 initializing
I/TC: Secondary CPU 1 switching to normal world boot
I/TC: Secondary CPU 2 initializing
I/TC: Secondary CPU 2 switching to normal world boot
I/TC: Secondary CPU 3 initializing
I/TC: Secondary CPU 3 switching to normal world boot
[    5.638611] Detected VIPT I-cache on CPU1
[    5.638645] GICv3: CPU1: found redistributor 100 region 0:0x00000000fd480000
[    5.638665] GICv3: CPU1: using allocated LPI pending table @0x0000000002850000
[    5.638720] CPU1: Booted secondary processor 0x0000000100 [0x412fd050]
[    5.640079] Detected VIPT I-cache on CPU2
[    5.640104] GICv3: CPU2: found redistributor 200 region 0:0x00000000fd4a0000
[    5.640120] GICv3: CPU2: using allocated LPI pending table @0x0000000002860000
[    5.640158] CPU2: Booted secondary processor 0x0000000200 [0x412fd050]
[    5.641484] Detected VIPT I-cache on CPU3
[    5.641509] GICv3: CPU3: found redistributor 300 region 0:0x00000000fd4c0000
[    5.641524] GICv3: CPU3: using allocated LPI pending table @0x0000000002870000
[    5.641560] CPU3: Booted secondary processor 0x0000000300 [0x412fd050]
[    5.641658] smp: Brought up 1 node, 4 CPUs
[    5.649061] SMP: Total of 4 processors activated.
[    5.649508] CPU features: detected: Privileged Access Never
[    5.650033] CPU features: detected: LSE atomic instructions
[    5.650559] CPU features: detected: User Access Override
[    5.651060] CPU features: detected: 32-bit EL0 Support
[    5.651545] CPU features: detected: Common not Private translations
[    5.652151] CPU features: detected: RAS Extension Support
[    5.652662] CPU features: detected: Data cache clean to the PoU not required for I/D coherence
[    5.653474] CPU features: detected: CRC32 instructions
[    5.653958] CPU features: detected: Speculative Store Bypassing Safe (SSBS)
[    5.654615] CPU features: detected: RCpc load-acquire (LDAPR)
[    5.680518] CPU: All CPU(s) started at EL2
[    5.682394] devtmpfs: initialized
[    5.704642] Trying to unpack rootfs image as initramfs...
[    5.705168] Registered cp15_barrier emulation handler
[    5.705183] Registered setend emulation handler
[    5.705190] KASLR disabled due to lack of seed
[    5.705384] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 6370867519511994 ns
[    5.707462] futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
[    5.708562] pinctrl core: initialized pinctrl subsystem
[    5.710164] NET: Registered protocol family 16
[    5.719759] DMA: preallocated 1024 KiB GFP_KERNEL pool for atomic allocations
[    5.729118] DMA: preallocated 1024 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    5.729927] audit: initializing netlink subsys (disabled)
[    5.730701] audit: type=2000 audit(0.103:1): state=initialized audit_enabled=0 res=1
[    5.732817] Registered FIQ tty driver
[    5.733571] thermal_sys: Registered thermal governor 'fair_share'
[    5.733576] thermal_sys: Registered thermal governor 'step_wise'
[    5.734154] thermal_sys: Registered thermal governor 'user_space'
[    5.735612] cpuidle: using governor menu
[    5.736876] hw-breakpoint: found 6 breakpoint and 4 watchpoint registers.
[    5.737849] ASID allocator initialised with 65536 entries
[    5.740805] ramoops: dmesg-0 0x20000@0x0000000000110000
[    5.741326] ramoops: console 0x80000@0x0000000000130000
[    5.741831] ramoops: pmsg    0x50000@0x00000000001b0000
[    5.742683] printk: console [ramoops-1] enabled
[    5.743376] pstore: Registered ramoops as persistent store backend
[    5.743966] ramoops: using 0xf0000@0x110000, ecc: 0
[    5.785516] rockchip-gpio fdd60000.gpio0: probed /pinctrl/gpio0@fdd60000
[    5.786736] rockchip-gpio fe740000.gpio1: probed /pinctrl/gpio1@fe740000
[    5.788036] rockchip-gpio fe750000.gpio2: probed /pinctrl/gpio2@fe750000
[    5.789196] rockchip-gpio fe760000.gpio3: probed /pinctrl/gpio3@fe760000
[    5.790353] rockchip-gpio fe770000.gpio4: probed /pinctrl/gpio4@fe770000
[    5.791185] rockchip-pinctrl pinctrl: probed pinctrl
[    5.809310] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    5.809957] HugeTLB registered 32.0 MiB page size, pre-allocated 0 pages
[    5.810592] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    5.811257] HugeTLB registered 64.0 KiB page size, pre-allocated 0 pages
[    5.813778] cryptd: max_cpu_qlen set to 1000
[    5.820120] fiq_debugger fiq_debugger.0: IRQ fiq not found
[    5.820664] fiq_debugger fiq_debugger.0: IRQ wakeup not found
[    5.821216] fiq_debugger_probe: could not install nmi irq handler
[[    5.821918] printk: console [ttyFIQ0] enabled
    5.821918] printk: console [ttyFIQ0] enabled
[    5.822723] printk: bootconsole [uart8250] disabled
[    5.822723] printk: bootconsole [uart8250] disabled
[    5.823395] Registered fiq debugger ttyFIQ0
[    5.824477] vcc3v3_sys: supplied by vdd_usbc
[    5.824840] vcc3v3_sysp: supplied by vdd_usbc
[    5.825195] vcc5v0_sysp: supplied by vcc3v3_sysp
[    5.825847] vcc5v0_host: supplied by vcc5v0_sysp
[    5.826472] vcc5v0_otg: supplied by vcc5v0_sysp
[    5.826701] pcie30_avdd0v9: supplied by vcc3v3_sys
[    5.827023] pcie30_avdd1v8: supplied by vcc3v3_sys
[    5.827495] vcc3v3_pcie: supplied by vcc3v3_sysp
[    5.828153] iommu: Default domain type: Translated 
[    5.831741] SCSI subsystem initialized
[    5.832016] usbcore: registered new interface driver usbfs
[    5.832064] usbcore: registered new interface driver hub
[    5.832107] usbcore: registered new device driver usb
[    5.832528] mc: Linux media interface: v0.10
[    5.832562] videodev: Linux video capture interface: v2.00
[    5.832653] pps_core: LinuxPPS API ver. 1 registered
[    5.832664] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    5.832682] PTP clock support registered
[    5.833441] arm-scmi firmware:scmi: SCMI Notifications - Core Enabled.
[    5.833510] arm-scmi firmware:scmi: SCMI Protocol v2.0 'rockchip:' Firmware version 0x0
[    5.835752] Advanced Linux Sound Architecture Driver Initialized.
[    5.836331] Bluetooth: Core ver 2.22
[    5.836382] NET: Registered protocol family 31
[    5.836391] Bluetooth: HCI device and connection manager initialized
[    5.836405] Bluetooth: HCI socket layer initialized
[    5.836415] Bluetooth: L2CAP socket layer initialized
[    5.836435] Bluetooth: SCO socket layer initialized
[    5.836455] NetLabel: Initializing
[    5.836463] NetLabel:  domain hash size = 128
[    5.836468] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
[    5.836523] NetLabel:  unlabeled traffic allowed by default
[    5.836924] rockchip-cpuinfo cpuinfo: SoC        : 35682000
[    5.836937] rockchip-cpuinfo cpuinfo: Serial     : d668a897af7d5bce
[    5.837991] clocksource: Switched to clocksource arch_sys_counter
[    5.946593] Freeing initrd memory: 7880K
[    6.387196] VFS: Disk quotas dquot_6.6.0
[    6.387277] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    6.387398] FS-Cache: Loaded
[    6.395497] NET: Registered protocol family 2
[    6.395764] IP idents hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    6.397257] tcp_listen_portaddr_hash hash table entries: 2048 (order: 4, 81920 bytes, linear)
[    6.397407] TCP established hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    6.397573] TCP bind hash table entries: 32768 (order: 8, 1048576 bytes, linear)
[    6.398394] TCP: Hash tables configured (established 32768 bind 32768)
[    6.398548] UDP hash table entries: 2048 (order: 5, 196608 bytes, linear)
[    6.398721] UDP-Lite hash table entries: 2048 (order: 5, 196608 bytes, linear)
[    6.399039] NET: Registered protocol family 1
[    6.399560] RPC: Registered named UNIX socket transport module.
[    6.399572] RPC: Registered udp transport module.
[    6.399579] RPC: Registered tcp transport module.
[    6.399586] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    6.399937] PCI: CLS 0 bytes, default 64
[    6.401891] rockchip-thermal fe710000.tsadc: tsadc is probed successfully!
[    6.402878] hw perfevents: enabled with armv8_cortex_a55 PMU driver, 7 counters available
[    6.409112] Initialise system trusted keyrings
[    6.409348] workingset: timestamp_bits=46 max_order=20 bucket_order=0
[    6.415143] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    6.415520] FS-Cache: Netfs 'nfs' registered for caching
[    6.415980] NFS: Registering the id_resolver key type
[    6.416025] Key type id_resolver registered
[    6.416035] Key type id_legacy registered
[    6.416117] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    6.416135] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[    6.416324] fuse: init (API version 7.32)
[    6.445041] Key type asymmetric registered
[    6.445068] Asymmetric key parser 'x509' registered
[    6.445110] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)
[    6.445247] io scheduler mq-deadline registered
[    6.445265] io scheduler kyber registered
[    6.445376] io scheduler bfq registered
[    6.453563] rockchip-snps-pcie3-phy fe8c0000.phy: failed to find rockchip,pipe_grf regmap
[    6.456138] rk-pcie 3c0000000.pcie: invalid prsnt-gpios property in node
[    6.456215] rk-pcie 3c0000000.pcie: no vpcie3v3 regulator found
[    6.456408] rk-pcie 3c0400000.pcie: invalid prsnt-gpios property in node
[    6.456637] rk-pcie 3c0000000.pcie: IRQ msi not found
[    6.456650] rk-pcie 3c0000000.pcie: use outband MSI support
[    6.456659] rk-pcie 3c0000000.pcie: Missing *config* reg space
[    6.456678] rk-pcie 3c0000000.pcie: host bridge /pcie@fe260000 ranges:
[    6.456709] rk-pcie 3c0000000.pcie:      err 0x00f4000000..0x00f40fffff -> 0x00f4000000
[    6.456790] rk-pcie 3c0000000.pcie:       IO 0x00f4100000..0x00f41fffff -> 0x00f4100000
[    6.456825] rk-pcie 3c0000000.pcie:      MEM 0x00f4200000..0x00f5ffffff -> 0x00f4200000
[    6.456841] rk-pcie 3c0800000.pcie: invalid prsnt-gpios property in node
[    6.456846] rk-pcie 3c0000000.pcie:      MEM 0x0300000000..0x033fffffff -> 0x0300000000
[    6.456876] rk-pcie 3c0000000.pcie: Missing *config* reg space
[    6.457021] rk-pcie 3c0000000.pcie: invalid resource
[    6.457629] iep: Module initialized.
[    6.457708] mpp_service mpp-srv: 6deff11d7ce73 author: Jensen Huang 2023-06-26 Merge remote-tracking branch 'rksdk/develop-5.10'
[    6.457719] mpp_service mpp-srv: probe start
[    6.459774] mpp_vdpu2 fdea0400.vdpu: Adding to iommu group 1
[    6.459990] mpp_vdpu2 fdea0400.vdpu: probe device
[    6.460438] mpp_vdpu2 fdea0400.vdpu: probing finish
[    6.460940] mpp_vepu2 fdee0000.vepu: Adding to iommu group 3
[    6.461126] mpp_vepu2 fdee0000.vepu: probing start
[    6.461593] mpp_vepu2 fdee0000.vepu: probing finish
[    6.462126] mpp-iep2 fdef0000.iep: Adding to iommu group 4
[    6.462306] mpp-iep2 fdef0000.iep: probe device
[    6.462594] mpp-iep2 fdef0000.iep: allocate roi buffer failed
[    6.462798] mpp-iep2 fdef0000.iep: probing finish
[    6.463246] mpp_jpgdec fded0000.jpegd: Adding to iommu group 2
[    6.463426] mpp_jpgdec fded0000.jpegd: probe device
[    6.463914] mpp_jpgdec fded0000.jpegd: probing finish
[    6.465392] mpp_service mpp-srv: probe success
[    6.470020] dma-pl330 fe530000.dmac: Loaded driver for PL330 DMAC-241330
[    6.470043] dma-pl330 fe530000.dmac:     DBUFF-128x8bytes Num_Chans-8 Num_Peri-32 Num_Events-16
[    6.471163] snps pcie3phy FW update! size 8192
[    6.472012] dma-pl330 fe550000.dmac: Loaded driver for PL330 DMAC-241330
[    6.472026] dma-pl330 fe550000.dmac:     DBUFF-128x8bytes Num_Chans-8 Num_Peri-32 Num_Events-16
[    6.472451] rockchip-pvtm fde00000.pvtm: pvtm@0 probed
[    6.472593] rockchip-pvtm fde80000.pvtm: pvtm@1 probed
[    6.472713] rockchip-pvtm fde90000.pvtm: pvtm@2 probed
[    6.473312] rockchip-system-monitor rockchip-system-monitor: system monitor probe
[    6.473972] arm-scmi firmware:scmi: Failed. SCMI protocol 22 not active.
[    6.474226] Serial: 8250/16550 driver, 10 ports, IRQ sharing disabled
[    6.475060] fe6d0000.serial: ttyS9 at MMIO 0xfe6d0000 (irq = 70, base_baud = 1500000) is a 16550A
[    6.477348] random: crng init done
[    6.477698] rockchip-vop2 fe040000.vop: Adding to iommu group 7
[    6.484023] rockchip-vop2 fe040000.vop: [drm:vop2_bind] vp0 assign plane mask: 0xf, primary plane phy id: 2
[    6.484068] rockchip-vop2 fe040000.vop: [drm:vop2_bind] vp1 assign plane mask: 0x30, primary plane phy id: 4
[    6.484081] rockchip-vop2 fe040000.vop: [drm:vop2_bind] vp2 assign plane mask: 0x0, primary plane phy id: -1
[    6.484197] rockchip-vop2 fe040000.vop: [drm:vop2_bind] Esmart1-win0 as cursor plane for vp0
[    6.484800] rk-pcie 3c0800000.pcie: IRQ msi not found
[    6.484812] rk-pcie 3c0800000.pcie: use outband MSI support
[    6.484822] rk-pcie 3c0800000.pcie: Missing *config* reg space
[    6.484843] rk-pcie 3c0800000.pcie: host bridge /pcie@fe280000 ranges:
[    6.484872] rk-pcie 3c0800000.pcie:      err 0x00f0000000..0x00f00fffff -> 0x00f0000000
[    6.484896] rockchip-drm display-subsystem: bound fe040000.vop (ops 0xffffffc0091215c0)
[    6.484900] rk-pcie 3c0800000.pcie:       IO 0x00f0100000..0x00f01fffff -> 0x00f0100000
[    6.484915] rk-pcie 3c0800000.pcie:      MEM 0x00f0200000..0x00f1ffffff -> 0x00f0200000
[    6.484989] rk-pcie 3c0800000.pcie:      MEM 0x0380000000..0x03bfffffff -> 0x0380000000
[    6.485030] rk-pcie 3c0800000.pcie: Missing *config* reg space
[    6.485074] rk-pcie 3c0800000.pcie: invalid resource
[    6.485208] dwhdmi-rockchip fe0a0000.hdmi: Detected HDMI TX controller v2.11a with HDCP (DWC HDMI 2.0 TX PHY)
[    6.485758] dwhdmi-rockchip fe0a0000.hdmi: registered DesignWare HDMI I2C bus driver
[    6.486000] dwhdmi-rockchip fe0a0000.hdmi: IRQ index 1 not found
[    6.486409] rockchip-drm display-subsystem: bound fe0a0000.hdmi (ops 0xffffffc00912f828)
[    6.561399] rk-pcie 3c0400000.pcie: no vpcie3v3 regulator found
[    6.575383] rk-pcie 3c0400000.pcie: IRQ msi not found
[    6.575404] rk-pcie 3c0400000.pcie: use outband MSI support
[    6.575415] rk-pcie 3c0400000.pcie: Missing *config* reg space
[    6.575433] rk-pcie 3c0400000.pcie: host bridge /pcie@fe270000 ranges:
[    6.575461] rk-pcie 3c0400000.pcie:      err 0x00f2000000..0x00f20fffff -> 0x00f2000000
[    6.575480] rk-pcie 3c0400000.pcie:       IO 0x00f2100000..0x00f21fffff -> 0x00f2100000
[    6.575500] rk-pcie 3c0400000.pcie:      MEM 0x00f2200000..0x00f3ffffff -> 0x00f2200000
[    6.575517] rk-pcie 3c0400000.pcie:      MEM 0x0340000000..0x037fffffff -> 0x0340000000
[    6.575555] rk-pcie 3c0400000.pcie: Missing *config* reg space
[    6.575594] rk-pcie 3c0400000.pcie: invalid resource
[    6.692453] rk-pcie 3c0800000.pcie: PCIe Linking... LTSSM is 0x3
[    6.718025] rk-pcie 3c0000000.pcie: PCIe Link up, LTSSM is 0x130011
[    6.718176] rk-pcie 3c0000000.pcie: PCI host bridge to bus 0000:00
[    6.718186] pci_bus 0000:00: root bus resource [bus 00-0f]
[    6.718192] pci_bus 0000:00: root bus resource [??? 0xf4000000-0xf40fffff flags 0x0]
[    6.718198] pci_bus 0000:00: root bus resource [io  0x0000-0xfffff] (bus address [0xf4100000-0xf41fffff])
[    6.718202] pci_bus 0000:00: root bus resource [mem 0xf4200000-0xf5ffffff]
[    6.718206] pci_bus 0000:00: root bus resource [mem 0x300000000-0x33fffffff pref]
[    6.718241] pci 0000:00:00.0: [1d87:3566] type 01 class 0x060400
[    6.718265] pci 0000:00:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[    6.718324] pci 0000:00:00.0: supports D1 D2
[    6.718328] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[    6.728044] pci_bus 0000:01: busn_res: can not insert [bus 01-ff] under [bus 00-0f] (conflicts with (null) [bus 00-0f])
[    6.728185] pci 0000:01:00.0: [10ec:8125] type 00 class 0x020000
[    6.728253] pci 0000:01:00.0: reg 0x10: initial BAR value 0x00000000 invalid
[    6.728257] pci 0000:01:00.0: reg 0x10: [io  size 0x0100]
[    6.728330] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x0000ffff 64bit]
[    6.728379] pci 0000:01:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit]
[    6.728812] pci 0000:01:00.0: supports D1 D2
[    6.728815] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    6.747430] pci 0000:00:00.0: BAR 8: assigned [mem 0xf4200000-0xf42fffff]
[    6.747443] pci 0000:00:00.0: BAR 6: assigned [mem 0xf4300000-0xf430ffff pref]
[    6.747449] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    6.747460] pci 0000:01:00.0: BAR 2: assigned [mem 0xf4200000-0xf420ffff 64bit]
[    6.747505] pci 0000:01:00.0: BAR 4: assigned [mem 0xf4210000-0xf4213fff 64bit]
[    6.747544] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
[    6.747560] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    6.747565] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    6.747571] pci 0000:00:00.0:   bridge window [mem 0xf4200000-0xf42fffff]
[    6.749861] pcieport 0000:00:00.0: PME: Signaling with IRQ 83
[    6.750678] Console: switching to colour frame buffer device 160x45
[    6.767261] rockchip-drm display-subsystem: [drm] fb0: rockchipdrmfb frame buffer device
[    6.768113] [drm] Initialized rockchip 3.0.0 20140818 for display-subsystem on minor 0
[    6.773329] input: hdmi_cec_key as /devices/platform/fe0a0000.hdmi/dw-hdmi-cec.1.auto/input/input0
[    6.774215] cacheinfo: Unable to detect cache hierarchy for CPU 0
[    6.774692] rk-pcie 3c0800000.pcie: PCIe Link up, LTSSM is 0x130011
[    6.774860] rk-pcie 3c0800000.pcie: PCI host bridge to bus 0002:20
[    6.774878] pci_bus 0002:20: root bus resource [bus 20-2f]
[    6.774890] pci_bus 0002:20: root bus resource [??? 0xf0000000-0xf00fffff flags 0x0]
[    6.774901] pci_bus 0002:20: root bus resource [io  0x100000-0x1fffff] (bus address [0xf0100000-0xf01fffff])
[    6.774911] pci_bus 0002:20: root bus resource [mem 0xf0200000-0xf1ffffff]
[    6.774921] pci_bus 0002:20: root bus resource [mem 0x380000000-0x3bfffffff pref]
[    6.774962] pci 0002:20:00.0: [1d87:3566] type 01 class 0x060400
[    6.774984] pci 0002:20:00.0: reg 0x10: [mem 0x00000000-0x3fffffff]
[    6.774997] pci 0002:20:00.0: reg 0x14: [mem 0x00000000-0x3fffffff]
[    6.775009] pci 0002:20:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[    6.775078] pci 0002:20:00.0: supports D1 D2
[    6.775087] pci 0002:20:00.0: PME# supported from D0 D1 D3hot
[    6.779377] brd: module loaded
[    6.786462] loop: module loaded
[    6.786763] lkdtm: No crash points registered, enable through debugfs
[    6.787054] system_heap: orders[0] = 6
[    6.787068] system_heap: orders[1] = 4
[    6.787075] system_heap: orders[2] = 0
[    6.788497] pci 0002:20:00.0: Primary bus is hard wired to 0
[    6.788524] pci 0002:20:00.0: bridge configuration invalid ([bus 01-ff]), reconfiguring
[    6.788763] pci 0002:21:00.0: [15b7:5006] type 00 class 0x010802
[    6.788862] pci 0002:21:00.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit]
[    6.788968] pci 0002:21:00.0: reg 0x20: [mem 0x00000000-0x000000ff 64bit]
[    6.789439] pci 0002:21:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0002:20:00.0 (capable of 31.504 Gb/s with 8.0 GT/s PCIe x4 link)
[    6.791782] spi-nand spi4.0: unknown raw ID 00000000
[    6.791814] spi-nand: probe of spi4.0 failed with error -524
[    6.794067] rk_gmac-dwmac fe2a0000.ethernet: IRQ eth_lpi not found
[    6.794388] rk_gmac-dwmac fe2a0000.ethernet: no regulator found
[    6.794401] rk_gmac-dwmac fe2a0000.ethernet: clock input or output? (output).
[    6.794411] rk_gmac-dwmac fe2a0000.ethernet: TX delay(0x3c).
[    6.794420] rk_gmac-dwmac fe2a0000.ethernet: RX delay(0x2f).
[    6.794436] rk_gmac-dwmac fe2a0000.ethernet: integrated PHY? (no).
[    6.794703] rk_gmac-dwmac fe2a0000.ethernet: init for RGMII
[    6.794922] rk_gmac-dwmac fe2a0000.ethernet: User ID: 0x30, Synopsys ID: 0x51
[    6.794937] rk_gmac-dwmac fe2a0000.ethernet:     DWMAC4/5
[    6.794948] rk_gmac-dwmac fe2a0000.ethernet: DMA HW capability register supported
[    6.794956] rk_gmac-dwmac fe2a0000.ethernet: RX Checksum Offload Engine supported
[    6.794964] rk_gmac-dwmac fe2a0000.ethernet: TX Checksum insertion supported
[    6.794971] rk_gmac-dwmac fe2a0000.ethernet: Wake-Up On Lan supported
[    6.795035] rk_gmac-dwmac fe2a0000.ethernet: TSO supported
[    6.795045] rk_gmac-dwmac fe2a0000.ethernet: Enable RX Mitigation via HW Watchdog Timer
[    6.795054] rk_gmac-dwmac fe2a0000.ethernet: TSO feature enabled
[    6.795064] rk_gmac-dwmac fe2a0000.ethernet: Using 32 bits DMA width
[    6.807674] pci_bus 0002:21: busn_res: [bus 21-2f] end is updated to 21
[    6.807730] pci 0002:20:00.0: BAR 0: no space for [mem size 0x40000000]
[    6.807742] pci 0002:20:00.0: BAR 0: failed to assign [mem size 0x40000000]
[    6.807752] pci 0002:20:00.0: BAR 1: no space for [mem size 0x40000000]
[    6.807761] pci 0002:20:00.0: BAR 1: failed to assign [mem size 0x40000000]
[    6.807772] pci 0002:20:00.0: BAR 8: assigned [mem 0xf0200000-0xf02fffff]
[    6.807782] pci 0002:20:00.0: BAR 6: assigned [mem 0xf0300000-0xf030ffff pref]
[    6.807798] pci 0002:21:00.0: BAR 0: assigned [mem 0xf0200000-0xf0203fff 64bit]
[    6.807852] pci 0002:21:00.0: BAR 4: assigned [mem 0xf0204000-0xf02040ff 64bit]
[    6.807898] pci 0002:20:00.0: PCI bridge to [bus 21]
[    6.807910] pci 0002:20:00.0:   bridge window [mem 0xf0200000-0xf02fffff]
[    6.810397] pcieport 0002:20:00.0: PME: Signaling with IRQ 93
[    6.810849] nvme nvme0: pci function 0002:21:00.0
[    6.810909] rk-pcie 3c0800000.pcie: missing "busno" property
[    6.810959] nvme 0002:21:00.0: enabling device (0000 -> 0002)
[    6.838014] rk-pcie 3c0400000.pcie: PCIe Link up, LTSSM is 0x130011
[    6.838180] rk-pcie 3c0400000.pcie: PCI host bridge to bus 0001:10
[    6.838197] pci_bus 0001:10: root bus resource [bus 10-1f]
[    6.838209] pci_bus 0001:10: root bus resource [??? 0xf2000000-0xf20fffff flags 0x0]
[    6.838220] pci_bus 0001:10: root bus resource [io  0x200000-0x2fffff] (bus address [0xf2100000-0xf21fffff])
[    6.838229] pci_bus 0001:10: root bus resource [mem 0xf2200000-0xf3ffffff]
[    6.838238] pci_bus 0001:10: root bus resource [mem 0x340000000-0x37fffffff pref]
[    6.838277] pci 0001:10:00.0: [1d87:3566] type 01 class 0x060400
[    6.838304] pci 0001:10:00.0: reg 0x38: [mem 0x00000000-0x0000ffff pref]
[    6.838369] pci 0001:10:00.0: supports D1 D2
[    6.838378] pci 0001:10:00.0: PME# supported from D0 D1 D3hot
[    6.848085] pci 0001:10:00.0: Primary bus is hard wired to 0
[    6.848130] pci 0001:10:00.0: bridge configuration invalid ([bus 01-ff]), reconfiguring
[    6.848430] pci 0001:11:00.0: [10ec:8125] type 00 class 0x020000
[    6.848508] pci 0001:11:00.0: reg 0x10: [io  0x0000-0x00ff]
[    6.848588] pci 0001:11:00.0: reg 0x18: [mem 0x00000000-0x0000ffff 64bit]
[    6.848641] pci 0001:11:00.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit]
[    6.849077] pci 0001:11:00.0: supports D1 D2
[    6.849088] pci 0001:11:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    6.866770] pci_bus 0001:11: busn_res: [bus 11-1f] end is updated to 11
[    6.866848] pci 0001:10:00.0: BAR 8: assigned [mem 0xf2200000-0xf22fffff]
[    6.866864] pci 0001:10:00.0: BAR 6: assigned [mem 0xf2300000-0xf230ffff pref]
[    6.866876] pci 0001:10:00.0: BAR 7: assigned [io  0x200000-0x200fff]
[    6.866893] pci 0001:11:00.0: BAR 2: assigned [mem 0xf2200000-0xf220ffff 64bit]
[    6.866945] pci 0001:11:00.0: BAR 4: assigned [mem 0xf2210000-0xf2213fff 64bit]
[    6.866991] pci 0001:11:00.0: BAR 0: assigned [io  0x200000-0x2000ff]
[    6.867012] pci 0001:10:00.0: PCI bridge to [bus 11]
[    6.867022] pci 0001:10:00.0:   bridge window [io  0x200000-0x200fff]
[    6.867033] pci 0001:10:00.0:   bridge window [mem 0xf2200000-0xf22fffff]
[    6.869436] pcieport 0001:10:00.0: PME: Signaling with IRQ 104
[    6.885009] PPP generic driver version 2.4.2
[    6.885287] usbcore: registered new interface driver asix
[    6.885347] usbcore: registered new interface driver ax88179_178a
[    6.885386] usbcore: registered new interface driver cdc_ether
[    6.885421] usbcore: registered new interface driver rndis_host
[    6.885460] usbcore: registered new interface driver qmi_wwan
[    6.894229] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    6.894279] ehci-pci: EHCI PCI platform driver
[    6.894361] ehci-platform: EHCI generic platform driver
[    6.894789] phy phy-fe8b0000.usb2-phy.3: illegal mode
[    6.896832] ehci-platform fd800000.usb: EHCI Host Controller
[    6.896875] ehci-platform fd800000.usb: new USB bus registered, assigned bus number 1
[    6.896998] ehci-platform fd800000.usb: irq 19, io mem 0xfd800000
[    6.908009] ehci-platform fd800000.usb: USB 2.0 started, EHCI 1.00
[    6.908210] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.10
[    6.908222] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    6.908231] usb usb1: Product: EHCI Host Controller
[    6.908239] usb usb1: Manufacturer: Linux 5.10.160 ehci_hcd
[    6.908246] usb usb1: SerialNumber: fd800000.usb
[    6.908711] hub 1-0:1.0: USB hub found
[    6.908748] hub 1-0:1.0: 1 port detected
[    6.911450] ehci-platform fd880000.usb: EHCI Host Controller
[    6.911478] ehci-platform fd880000.usb: new USB bus registered, assigned bus number 2
[    6.911594] ehci-platform fd880000.usb: irq 21, io mem 0xfd880000
[    6.924670] ehci-platform fd880000.usb: USB 2.0 started, EHCI 1.00
[    6.924868] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.10
[    6.924880] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    6.924889] usb usb2: Product: EHCI Host Controller
[    6.924897] usb usb2: Manufacturer: Linux 5.10.160 ehci_hcd
[    6.924904] usb usb2: SerialNumber: fd880000.usb
[    6.925302] hub 2-0:1.0: USB hub found
[    6.925339] hub 2-0:1.0: 1 port detected
[    6.925992] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    6.926017] ohci-platform: OHCI generic platform driver
[    6.926330] phy phy-fe8b0000.usb2-phy.3: illegal mode
[    6.926344] ohci-platform fd840000.usb: Generic Platform OHCI controller
[    6.926367] ohci-platform fd840000.usb: new USB bus registered, assigned bus number 3
[    6.926469] ohci-platform fd840000.usb: irq 20, io mem 0xfd840000
[    6.985473] usb usb3: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 5.10
[    6.985493] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    6.985502] usb usb3: Product: Generic Platform OHCI controller
[    6.985510] usb usb3: Manufacturer: Linux 5.10.160 ohci_hcd
[    6.985518] usb usb3: SerialNumber: fd840000.usb
[    6.985939] hub 3-0:1.0: USB hub found
[    6.985977] hub 3-0:1.0: 1 port detected
[    6.986510] ohci-platform fd8c0000.usb: Generic Platform OHCI controller
[    6.986535] ohci-platform fd8c0000.usb: new USB bus registered, assigned bus number 4
[    6.986665] ohci-platform fd8c0000.usb: irq 22, io mem 0xfd8c0000
[    6.993874] nvme nvme0: 4/0/0 default/read/poll queues
[    6.997680]  nvme0n1: p1
[    7.045501] usb usb4: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 5.10
[    7.045529] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    7.045539] usb usb4: Product: Generic Platform OHCI controller
[    7.045547] usb usb4: Manufacturer: Linux 5.10.160 ohci_hcd
[    7.045555] usb usb4: SerialNumber: fd8c0000.usb
[    7.046015] hub 4-0:1.0: USB hub found
[    7.046059] hub 4-0:1.0: 1 port detected
[    7.047307] xhci-hcd xhci-hcd.4.auto: xHCI Host Controller
[    7.047340] xhci-hcd xhci-hcd.4.auto: new USB bus registered, assigned bus number 5
[    7.047501] xhci-hcd xhci-hcd.4.auto: hcc params 0x0220fe64 hci version 0x110 quirks 0x0000600002010010
[    7.047562] xhci-hcd xhci-hcd.4.auto: irq 107, io mem 0xfd000000
[    7.047852] usb usb5: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.10
[    7.047864] usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    7.047873] usb usb5: Product: xHCI Host Controller
[    7.047881] usb usb5: Manufacturer: Linux 5.10.160 xhci-hcd
[    7.047888] usb usb5: SerialNumber: xhci-hcd.4.auto
[    7.048374] hub 5-0:1.0: USB hub found
[    7.048413] hub 5-0:1.0: 1 port detected
[    7.048720] xhci-hcd xhci-hcd.4.auto: xHCI Host Controller
[    7.048740] xhci-hcd xhci-hcd.4.auto: new USB bus registered, assigned bus number 6
[    7.048758] xhci-hcd xhci-hcd.4.auto: Host supports USB 3.0 SuperSpeed
[    7.048824] usb usb6: We don't know the algorithms for LPM for this host, disabling LPM.
[    7.048931] usb usb6: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.10
[    7.048943] usb usb6: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    7.048952] usb usb6: Product: xHCI Host Controller
[    7.048960] usb usb6: Manufacturer: Linux 5.10.160 xhci-hcd
[    7.048968] usb usb6: SerialNumber: xhci-hcd.4.auto
[    7.049355] hub 6-0:1.0: USB hub found
[    7.049392] hub 6-0:1.0: 1 port detected
[    7.049853] xhci-hcd xhci-hcd.5.auto: xHCI Host Controller
[    7.049877] xhci-hcd xhci-hcd.5.auto: new USB bus registered, assigned bus number 7
[    7.050005] xhci-hcd xhci-hcd.5.auto: hcc params 0x0220fe64 hci version 0x110 quirks 0x0000602002010010
[    7.050061] xhci-hcd xhci-hcd.5.auto: irq 106, io mem 0xfcc00000
[    7.050335] usb usb7: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.10
[    7.050348] usb usb7: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    7.050356] usb usb7: Product: xHCI Host Controller
[    7.050364] usb usb7: Manufacturer: Linux 5.10.160 xhci-hcd
[    7.050371] usb usb7: SerialNumber: xhci-hcd.5.auto
[    7.050757] hub 7-0:1.0: USB hub found
[    7.050793] hub 7-0:1.0: 1 port detected
[    7.051080] xhci-hcd xhci-hcd.5.auto: xHCI Host Controller
[    7.051100] xhci-hcd xhci-hcd.5.auto: new USB bus registered, assigned bus number 8
[    7.051117] xhci-hcd xhci-hcd.5.auto: Host supports USB 3.0 SuperSpeed
[    7.051179] usb usb8: We don't know the algorithms for LPM for this host, disabling LPM.
[    7.051294] usb usb8: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 5.10
[    7.051332] usb usb8: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    7.051343] usb usb8: Product: xHCI Host Controller
[    7.051351] usb usb8: Manufacturer: Linux 5.10.160 xhci-hcd
[    7.051358] usb usb8: SerialNumber: xhci-hcd.5.auto
[    7.051728] hub 8-0:1.0: USB hub found
[    7.051763] hub 8-0:1.0: 1 port detected
[    7.052195] usbcore: registered new interface driver cdc_acm
[    7.052205] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
[    7.052259] usbcore: registered new interface driver cdc_wdm
[    7.052499] usbcore: registered new interface driver uas
[    7.052593] usbcore: registered new interface driver usb-storage
[    7.052677] usbcore: registered new interface driver usbserial_generic
[    7.052708] usbserial: USB Serial support registered for generic
[    7.052753] usbcore: registered new interface driver ch341
[    7.052777] usbserial: USB Serial support registered for ch341-uart
[    7.052827] usbcore: registered new interface driver cp210x
[    7.052850] usbserial: USB Serial support registered for cp210x
[    7.052916] usbcore: registered new interface driver ftdi_sio
[    7.052940] usbserial: USB Serial support registered for FTDI USB Serial Device
[    7.053078] usbcore: registered new interface driver option
[    7.053103] usbserial: USB Serial support registered for GSM modem (1-port)
[    7.053288] usbcore: registered new interface driver pl2303
[    7.053314] usbserial: USB Serial support registered for pl2303
[    7.053369] usbcore: registered new interface driver qcserial
[    7.053394] usbserial: USB Serial support registered for Qualcomm USB modem
[    7.053858] mousedev: PS/2 mouse device common for all mice
[    7.054600] <<-GTP-INFO->> GTP driver installing
[    7.055326] i2c /dev entries driver
[    7.056706] fan53555-regulator 0-001c: FAN53555 Option[12] Rev[15] Detected!
[    7.057263] vdd_cpu: supplied by vcc3v3_sys
[    7.060531] rk808 0-0020: chip id: 0x8090
[    7.060591] rk808 0-0020: No cache defaults, reading back from HW
[    7.066762] rk808 0-0020: source: on=0x40, off=0x00
[    7.066779] rk808 0-0020: support dcdc3 fb mode:-22, 1
[    7.066790] rk808 0-0020: support pmic reset mode:0,0
[    7.070291] rk808-regulator rk808-regulator: there is no dvs0 gpio
[    7.070326] rk808-regulator rk808-regulator: there is no dvs1 gpio
[    7.070525] vdd_logic: supplied by vcc3v3_sys
[    7.071102] vdd_gpu: supplied by vcc3v3_sys
[    7.071552] vcc_ddr: supplied by vcc3v3_sys
[    7.071968] vdd_npu: supplied by vcc3v3_sys
[    7.072488] vcc_1v8: supplied by vcc3v3_sys
[    7.072871] vdda0v9_image: supplied by vcc3v3_sys
[    7.073880] vdda_0v9: supplied by vcc3v3_sys
[    7.074814] vdda0v9_pmu: supplied by vcc3v3_sys
[    7.075700] vccio_acodec: supplied by vcc3v3_sys
[    7.076701] vccio_sd: supplied by vcc3v3_sys
[    7.077609] vcc3v3_pmu: supplied by vcc3v3_sys
[    7.078629] vcca_1v8: supplied by vcc3v3_sys
[    7.079537] vcca1v8_pmu: supplied by vcc3v3_sys
[    7.080432] vcca1v8_image: supplied by vcc3v3_sys
[    7.081512] vcc_3v3: supplied by vcc3v3_sys
[    7.082001] vcc3v3_sd: supplied by vcc3v3_sys
[    7.082384] rk817-battery: Failed to locate of_node [id: -1]
[    7.082598] rk817-charger: Failed to locate of_node [id: -1]
[    7.083851] input: rk805 pwrkey as /devices/platform/fdd40000.i2c/i2c-0/0-0020/rk805-pwrkey/input/input1
[    7.088626] rtc-hym8563 5-0051: rtc information is invalid
[    7.091042] rtc-hym8563 5-0051: registered as rtc0
[    7.091376] rtc-hym8563 5-0051: setting system clock to 2021-01-01T12:00:00 UTC (1609502400)
[    7.092607] IR NEC protocol handler initialized
[    7.094147] __power_supply_register: Expected proper parent device for 'test_ac'
[    7.094439] __power_supply_register: Expected proper parent device for 'test_battery'
[    7.094854] __power_supply_register: Expected proper parent device for 'test_usb'
[    7.097111] device-mapper: uevent: version 1.0.3
[    7.097364] device-mapper: ioctl: 4.44.0-ioctl (2021-02-01) initialised: dm-devel@redhat.com
[    7.097899] cpu cpu0: leakage=26
[    7.098006] cpu cpu0: pvtm = 86420, from nvmem
[    7.098025] cpu cpu0: pvtm-volt-sel=1
[    7.099586] cpu cpu0: avs=0
[    7.099936] cpu cpu0: EM: created perf domain
[    7.100004] cpu cpu0: l=0 h=2147483647 hyst=5000 l_limit=1608000000 h_limit=0 h_table=0
[    7.102936] sdhci: Secure Digital Host Controller Interface driver
[    7.102965] sdhci: Copyright(c) Pierre Ossman
[    7.102974] Synopsys Designware Multimedia Card Interface Driver
[    7.103517] sdhci-pltfm: SDHCI platform and OF driver helper
[    7.104204] dwmmc_rockchip fe2b0000.dwmmc: No normal pinctrl state
[    7.104223] dwmmc_rockchip fe2b0000.dwmmc: No idle pinctrl state
[    7.104351] dwmmc_rockchip fe2b0000.dwmmc: IDMAC supports 32-bit address mode.
[    7.104388] dwmmc_rockchip fe2b0000.dwmmc: Using internal DMA controller.
[    7.104404] dwmmc_rockchip fe2b0000.dwmmc: Version ID is 270a
[    7.104476] dwmmc_rockchip fe2b0000.dwmmc: DW MMC controller at irq 60,32 bit host data width,256 deep fifo
[    7.106576] ledtrig-cpu: registered to indicate activity on CPUs
[    7.106623] arm-scmi firmware:scmi: Failed. SCMI protocol 17 not active.
[    7.106704] SMCCC: SOC_ID: ARCH_SOC_ID not implemented, skipping ....
[    7.107376] hid: raw HID events driver (C) Jiri Kosina
[    7.107846] usbcore: registered new interface driver usbhid
[    7.107858] usbhid: USB HID core driver
[    7.108557] ashmem: initialized
[    7.109119] rockchip,bus bus-npu: Failed to get leakage
[    7.109183] rockchip,bus bus-npu: pvtm = 86420, from nvmem
[    7.109203] rockchip,bus bus-npu: pvtm-volt-sel=1
[    7.109528] rockchip,bus bus-npu: avs=0
[    7.114978] optee: probing for conduit method.
[    7.115029] optee: revision 3.13 (dcfdd61d)
[    7.115402] optee: dynamic shared memory is enabled
[    7.115619] optee: initialized driver
[    7.118276] mmc_host mmc0: Bus speed (slot 0) = 375000Hz (slot req 400000Hz, actual 375000HZ div = 0)
[    7.122736] Initializing XFRM netlink socket
[    7.122792] IPsec XFRM device driver
[    7.123236] NET: Registered protocol family 10
[    7.124328] Segment Routing with IPv6
[    7.126613] bpfilter: Loaded bpfilter_umh pid 192
[    7.127308] NET: Registered protocol family 17
[    7.127584] Bluetooth: RFCOMM TTY layer initialized
[    7.127603] Bluetooth: RFCOMM socket layer initialized
[    7.127635] Bluetooth: RFCOMM ver 1.11
[    7.127653] Bluetooth: HIDP (Human Interface Emulation) ver 1.2
[    7.127669] Bluetooth: HIDP socket layer initialized
[    7.128220] [BT_RFKILL]: Enter rfkill_rk_init
[    7.128233] [WLAN_RFKILL]: Enter rfkill_wlan_init
[    7.128773] Key type dns_resolver registered
[    7.130082] registered taskstats version 1
[    7.130108] Loading compiled-in X.509 certificates
[    7.130464] pstore: Using crash dump compression: deflate
[    7.131816] rga: rga2, irq = 30, match scheduler
[    7.132387] rga: rga2 hardware loaded successfully, hw_version:3.2.63318.
[    7.132424] rga: rga2 probe successfully
[    7.132750] rga_iommu: IOMMU binding successfully, default mapping core[0x4]
[    7.133012] rga: Module initialized. v1.2.26
[    7.133027] 
[    7.133035] ********************************************************************
[    7.133043] **     NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE           **
[    7.133050] **                                                                **
[    7.133057] **  WRITEABLE clk DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL **
[    7.133064] **                                                                **
[    7.133071] ** This means that this kernel is built to expose clk operations  **
[    7.133077] ** such as parent or rate setting, enabling, disabling, etc.      **
[    7.133083] ** to userspace, which may compromise security on your system.    **
[    7.133090] **                                                                **
[    7.133096] ** If you see this message and you are not debugging the          **
[    7.133102] ** kernel, report this immediately to your vendor!                **
[    7.133108] **                                                                **
[    7.133115] **     NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE           **
[    7.133120] ********************************************************************
[    7.135431] mmc2: SDHCI controller on fe310000.sdhci [fe310000.sdhci] using ADMA
[    7.164542] friendlyelec-board board: Serial     : d668a897af7d5bce
[    7.165753] mpp_rkvenc fdf40000.rkvenc: Adding to iommu group 5
[    7.165962] mpp_rkvenc fdf40000.rkvenc: probing start
[    7.166355] mpp_rkvenc fdf40000.rkvenc: Failed to get leakage
[    7.166402] mpp_rkvenc fdf40000.rkvenc: pvtm = 86420, from nvmem
[    7.166415] mpp_rkvenc fdf40000.rkvenc: pvtm-volt-sel=1
[    7.166574] mpp_rkvenc fdf40000.rkvenc: avs=0
[    7.166679] mpp_rkvenc fdf40000.rkvenc: failed to find power_model node
[    7.166687] mpp_rkvenc fdf40000.rkvenc: failed to initialize power model
[    7.166694] mpp_rkvenc fdf40000.rkvenc: failed to get dynamic-coefficient
[    7.167080] mpp_rkvenc fdf40000.rkvenc: probing finish
[    7.167378] mpp_rkvdec2 fdf80200.rkvdec: Adding to iommu group 6
[    7.167648] mpp_rkvdec2 fdf80200.rkvdec: rkvdec, probing start
[    7.167943] mpp_rkvdec2 fdf80200.rkvdec: shared_niu_a is not found!
[    7.167953] rkvdec2_init:1022: No niu aclk reset resource define
[    7.167961] mpp_rkvdec2 fdf80200.rkvdec: shared_niu_h is not found!
[    7.167967] rkvdec2_init:1025: No niu hclk reset resource define
[    7.168269] mpp_rkvdec2 fdf80200.rkvdec: leakage=46
[    7.168283] mpp_rkvdec2 fdf80200.rkvdec: leakage-volt-sel=0
[    7.168316] mpp_rkvdec2 fdf80200.rkvdec: pvtm = 86420, from nvmem
[    7.168328] mpp_rkvdec2 fdf80200.rkvdec: pvtm-volt-sel=1
[    7.168494] mpp_rkvdec2 fdf80200.rkvdec: avs=0
[    7.168601] mpp_rkvdec2 fdf80200.rkvdec: failed to find power_model node
[    7.168610] mpp_rkvdec2 fdf80200.rkvdec: failed to initialize power model
[    7.168616] mpp_rkvdec2 fdf80200.rkvdec: failed to get dynamic-coefficient
[    7.168722] mpp_rkvdec2 fdf80200.rkvdec: sram_start 0x00000000fdcc0000
[    7.168730] mpp_rkvdec2 fdf80200.rkvdec: rcb_iova 0x0000000010000000
[    7.168736] mpp_rkvdec2 fdf80200.rkvdec: sram_size 45056
[    7.168742] mpp_rkvdec2 fdf80200.rkvdec: rcb_size 65536
[    7.168749] mpp_rkvdec2 fdf80200.rkvdec: min_width 512
[    7.168800] mpp_rkvdec2 fdf80200.rkvdec: link mode probe finish
[    7.168867] mpp_rkvdec2 fdf80200.rkvdec: probing finish
[    7.169586] rockchip-iodomain fdc20000.syscon:io-domains: pmuio2(3300000 uV) supplied by vcc3v3_pmu
[    7.169710] mali fde60000.gpu: Kernel DDK version g18p0-01eac0
[    7.169724] rockchip-iodomain fdc20000.syscon:io-domains: vccio1(3300000 uV) supplied by vccio_acodec
[    7.169861] rockchip-iodomain fdc20000.syscon:io-domains: vccio3(3300000 uV) supplied by vccio_sd
[    7.169989] rockchip-iodomain fdc20000.syscon:io-domains: vccio4(1800000 uV) supplied by vcc_1v8
[    7.170091] mali fde60000.gpu: leakage=7
[    7.170094] rockchip-iodomain fdc20000.syscon:io-domains: vccio5(3300000 uV) supplied by vcc_3v3
[    7.170135] mali fde60000.gpu: pvtm = 86420, from nvmem
[    7.170150] mali fde60000.gpu: pvtm-volt-sel=1
[    7.170154] rockchip-iodomain fdc20000.syscon:io-domains: vccio6(1800000 uV) supplied by vcc_1v8
[    7.170205] rockchip-iodomain fdc20000.syscon:io-domains: vccio7(3300000 uV) supplied by vcc_3v3
[    7.170750] mali fde60000.gpu: avs=0
[    7.170777] W : [File] : drivers/gpu/arm/bifrost/platform/rk/mali_kbase_config_rk.c; [Line] : 143; [Func] : kbase_platform_rk_init(); power-off-delay-ms not available.
[    7.171156] mali fde60000.gpu: GPU identified as 0x2 arch 7.4.0 r1p0 status 0
[    7.171233] mali fde60000.gpu: No priority control manager is configured
[    7.171434] mali fde60000.gpu: No memory group manager is configured
[    7.172696] mmc2: Host Software Queue enabled
[    7.172725] mmc2: new HS200 MMC card at address 0001
[    7.172929] mali fde60000.gpu: l=0 h=2147483647 hyst=5000 l_limit=700000000 h_limit=0 h_table=0
[    7.173345] mmcblk2: mmc2:0001 AJTD4R 14.6 GiB 
[    7.173649] mmcblk2boot0: mmc2:0001 AJTD4R partition 1 4.00 MiB
[    7.173939] mmcblk2boot1: mmc2:0001 AJTD4R partition 2 4.00 MiB
[    7.174117] mmcblk2rpmb: mmc2:0001 AJTD4R partition 3 4.00 MiB, chardev (237:0)
[    7.174880] mali fde60000.gpu: Probed as mali0
[    7.176412] input: gpio-keys as /devices/platform/gpio-keys/input/input2
[    7.177098]  mmcblk2: p1
[    7.179393] rk808-rtc rk808-rtc: registered as rtc1
[    7.180363] RKNPU fde40000.npu: Adding to iommu group 0
[    7.180800] RKNPU fde40000.npu: RKNPU: rknpu iommu is enabled, using iommu mode
[    7.181080] RKNPU fde40000.npu: can't request region for resource [mem 0xfde40000-0xfde4ffff]
[    7.181551] [drm] Initialized rknpu 0.8.8 20230428 for fde40000.npu on minor 1
[    7.181833] RKNPU fde40000.npu: leakage=3
[    7.181871] RKNPU fde40000.npu: pvtm = 86420, from nvmem
[    7.181887] RKNPU fde40000.npu: pvtm-volt-sel=1
[    7.182479] RKNPU fde40000.npu: avs=0
[    7.182638] RKNPU fde40000.npu: l=0 h=2147483647 hyst=5000 l_limit=0 h_limit=0 h_table=0
[    7.182899] RKNPU fde40000.npu: failed to find power_model node
Loading, please wait...
[    7.182908] RKNPU fde40000.npu: RKNPU: failed to initialize power model
[    7.182915] RKNPU fde40000.npu: RKNPU: failed to get dynamic-coefficient
[    7.184782] rockchip-pm rockchip-suspend: not set pwm-regulator-config
[    7.184807] rockchip-suspend not set sleep-mode-config for mem-lite
[    7.184813] rockchip-suspend not set wakeup-config for mem-lite
[    7.184821] rockchip-suspend not set sleep-mode-config for mem-ultra
[    7.184826] rockchip-suspend not set wakeup-config for mem-ultra
[    7.185507] ALSA device list:
[    7.185519]   #0: rockchip,hdmi
[    7.192441] Freeing unused kernel memory: 6784K
[    7.192587] Run /init as init process
[    7.209530] mmc_host mmc0: Bus speed (slot 0) = 50000000Hz (slot req 100000000Hz, actual 50000000HZ div = 0)
[    7.221413] dwmmc_rockchip fe2b0000.dwmmc: Successfully tuned phase to 360
[    7.221461] mmc0: new ultra high speed SDR50 SDHC card at address aaaa
[    7.222225] mmcblk0: mmc0:aaaa SL08G 7.40 GiB 
[    7.229784]  mmcblk0: p1 p2 p3 p4 p5 p6 p7 p8 p9
Starting version 249.11-0ubuntu3.7
[    7.251478] vendor storage:20190527 ret = 0
[    7.304739] usb 7-1: new full-speed USB device number 2 using xhci-hcd
[    7.447676] usb 7-1: New USB device found, idVendor=04b3, idProduct=3016, bcdDevice= 2.01
[    7.447707] usb 7-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[    7.447714] usb 7-1: Product: USB 1.1 2 port downstream low-power hub
[    7.447721] usb 7-1: Manufacturer: Lite-On Tech
[    7.691555] hub 7-1:1.0: USB hub found
[    7.692189] hub 7-1:1.0: 4 ports detected
[    7.739093] Freeing drm_logo memory: 4512K
[    7.978072] usb 7-1.3: new low-speed USB device number 3 using xhci-hcd
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
ajayramaswamy commented 11 months ago

Have you tried with pcie_aspm=off on the kernel command line. I have to use that to make my WD Blue drive work in my NanoPI R5S

svpcom commented 11 months ago

@ajayramaswamy Yes, I've tried - no success

ajayramaswamy commented 11 months ago

I am currently running u-boot from https://github.com/Kwiboo/u-boot-rockchip/commits/rk3568-2023.10 at commit a821e84f39ed32f150cb8c6aeced836a2cdfee0a This is installed on the internal emmc and my OS Fedora 38 and grub etc is started via efi from the WD Blue NVMe disk

svpcom commented 11 months ago

Also it is interesting that booting kernel from friendlyarm (I've build it from their sources) using stock u-boot (from @inindev) results in partial working hardware:

It seems that friendlyarm's u-boot do a lot of work for hardware initialization. But is it old (2017) and intended mostly for android (a lot of crap android features like ten emmc partitions, etc)

svpcom commented 11 months ago

@ajayramaswamy How to configure, build and install this version of u-boot for R5S? Do any additional patches required?

ajayramaswamy commented 11 months ago

!/bin/bash

#

cd ~/scratch/kwiboo/u-boot-rockchip/ || exit 99

git clean -fdx

git switch rk3568-2023.07

export CROSS_COMPILE=aarch64-linux-gnu- export BL31=~/scratch/kwiboo/rkbin-2/bin/rk35/rk3568_bl31_v1.43.elf export ROCKCHIP_TPL=~/scratch/kwiboo/rkbin-2/bin/rk35/rk3568_ddr_1560MHz_v1.18.bin

git am ../patches/0*.patch

make -j"$(nproc)" distclean

make -j"$(nproc)" nanopi-r5s-rk3568_defconfig

make -j"$(nproc)" olddefconfig make -j"$(nproc)" savedefconfig make -j"$(nproc)"

This is the shell script I am using to build on my fedora x86_64 laptop

On Fri, 22 Sept 2023 at 18:28, Vasily Evseenko @.***> wrote:

@ajayramaswamy https://github.com/ajayramaswamy How to configure, build and install this version of u-boot for R5S? Do any additional patches required?

— Reply to this email directly, view it on GitHub https://github.com/inindev/nanopi-r5/issues/23#issuecomment-1731372726, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAPZLF65OXQ4ZR22BQBYOO3X3WDPRANCNFSM6AAAAAA5BNJ2VY . You are receiving this because you were mentioned.Message ID: @.***>

--

Thanks & best regards

Ajay

svpcom commented 11 months ago

@ajayramaswamy I've tried following branches: rk3568-2023.10, rk3568-2023.07.02 and tag rk3568-2023.07. No success. Behavior still the same - No NVMe detect on boot and device freeze after pci rescan. Also I've tried to enable pcie and nvme in u-boot config. Also no success.

What device tree do you use for fedora kernel and what kernel version is?

svpcom commented 11 months ago

@inindev Do you use debian 12 kernel 6.1 or build custom 6.4? I ask this because dtb is from 6.4 and 6.1 doesn't have dtb files for R5S

svpcom commented 11 months ago

I've tried linux-6.5.0 from debian-sid. Also no success. It seems the problem is in kernel (or dtb) because u-boot see NVMe drive and can read a partition table.

inindev commented 11 months ago

Device 0: Vendor: 0x15b7 Rev: 112000WD Prod: 23041F800808
Type: Hard Disk Capacity: 1907729.0 MB = 1863.0 GB (3907029168 x 512) ... is now current device

The issue is not u-boot, we see it is properly being detected. ☝️

The issue looks to be a timeout with linux:

[   46.097927] nvme nvme0: pci function 0002:01:00.0
[   46.098444] nvme 0002:01:00.0: enabling device (0000 -> 0002)
[  107.508575] nvme nvme0: I/O 12 QID 0 timeout, completion polled
[  168.958145] nvme nvme0: I/O 8 QID 0 timeout, completion polled
[  230.391719] nvme nvme0: I/O 13 QID 0 timeout, completion polled

I have increased the timeout in this dtb, can you replace the one in /boot? https://drive.google.com/file/d/1hyjFWuMCrhI_wlOSDiFkD-vLOQVr2Qgb/view?usp=sharing

svpcom commented 11 months ago

I've tried and behavior was the same - no nvme device during boot and stuck device after pci rescan

Device 0: Vendor: 0x15b7 Rev: 112000WD Prod: 23041F800808 Type: Hard Disk Capacity: 1907729.0 MB = 1863.0 GB (3907029168 x 512) ... is now current device

The issue is not u-boot, we see it is properly being detected. ☝️

The issue looks to be a timeout with linux:

[   46.097927] nvme nvme0: pci function 0002:01:00.0
[   46.098444] nvme 0002:01:00.0: enabling device (0000 -> 0002)
[  107.508575] nvme nvme0: I/O 12 QID 0 timeout, completion polled
[  168.958145] nvme nvme0: I/O 8 QID 0 timeout, completion polled
[  230.391719] nvme nvme0: I/O 13 QID 0 timeout, completion polled

I have increased the timeout in this dtb, can you replace the one in /boot? https://drive.google.com/file/d/1hyjFWuMCrhI_wlOSDiFkD-vLOQVr2Qgb/view?usp=sharing

inindev commented 11 months ago

This is the shell script I am using to build on my fedora x86_64 laptop

Also note that Jonas Karlman has builds for his repositories here: https://github.com/Kwiboo/u-boot-build/tree/main

Here is the list of targets: https://github.com/Kwiboo/u-boot-build/actions

And the outputs from the most recent u-boot rk3568-2023.07.02 for RK356x: https://github.com/Kwiboo/u-boot-build/actions/runs/6100093756 (note that you will need to login to download)

inindev commented 11 months ago

I've tried and behavior was the same - no nvme device during boot and stuck device after pci rescan

I have ordered a WD NVMe to see if I can reproduce this issue locally.

svpcom commented 11 months ago

I've tried his u-boot builds and no success for nvme support in subsequent linux kernel boot (see my replies to @ajayramaswamy above)

This is the shell script I am using to build on my fedora x86_64 laptop

Also note that Jonas Karlman has builds for his repositories here: https://github.com/Kwiboo/u-boot-build/tree/main

Here is the list of targets: https://github.com/Kwiboo/u-boot-build/actions

And the outputs from the most recent u-boot rk3568-2023.07.02 for RK356x: https://github.com/Kwiboo/u-boot-build/actions/runs/6100093756 (note that you will need to login to download)

svpcom commented 11 months ago

My drive is wd red sn700 2TB It works with friendlyarm's system (based on debian 11 with custom 5.10 kernel and u-boot)

I've tried and behavior was the same - no nvme device during boot and stuck device after pci rescan

I have ordered a WD NVMe to see if I can reproduce this issue locally.

inindev commented 11 months ago

I've tried his u-boot builds and no success for nvme support in subsequent linux kernel boot (see my replies to @ajayramaswamy above)

As I noted above, the issue is with linux, not with u-boot. The reference to the build artifacts were for anyone reading this to know that local u-boot builds were unnecessary.

With some luck, maybe multiple WD NVMEs have problems and I might be able to figure out why.

inindev commented 11 months ago

My WD Blue SN570 came in today and failed in the same way you report. Placing pcie_aspm=off on the kernel command line gets it to work:

debian@nanopi-r5s-arm64:~$ ll /dev/nvme0*
crw------- 1 root root 235, 0 Sep 24 19:41 /dev/nvme0
brw-rw---- 1 root disk 259, 0 Sep 24 19:41 /dev/nvme0n1

These are the steps I followed:

wget https://github.com/inindev/nanopi-r5/releases/download/v12.0.3/nanopi-r5s_bookworm-1203.img.xz
sudo su
xzcat nanopi-r5s_bookworm-1203.img.xz > /dev/sdX

Note: replace /dev/sdX with your MMC device name, then boot using this MMC

Once booted, edit the kernel command line:

sudo nano /boot/mk_extlinux
  EXLT_CMD_LINE='ro rootwait pcie_aspm=off'  # optional: ipv6.disable=1

sudo /boot/mk_extlinux
  file /boot/extlinux/extlinux.conf updated successfully

cat file /boot/extlinux/extlinux.conf
  ...
  append root=UUID=f17fb872-5b13-11ee-8c99-0242ac120002 ro rootwait pcie_aspm=off
  ...

sudo reboot
inindev commented 11 months ago

To boot to Western Digital NVMe:

While booted from MMC:

wget https://github.com/inindev/nanopi-r5/releases/download/v12.0.3/nanopi-r5s_bookworm-1203.img.xz
sudo su
xzcat nanopi-r5s_bookworm-1203.img.xz > /dev/nvme0n1

mount /dev/nvme0n1p1 /mnt
nano /mnt/boot/mk_extlinux
   EXLT_CMD_LINE='ro rootwait pcie_aspm=off'  # optional: ipv6.disable=1

chroot /mnt
/boot/mk_extlinux
exit

cat /mnt/boot/extlinux/extlinux.conf
   ...
   append root=UUID=2482ce5c-5b15-11ee-8c99-0242ac120002 ro rootwait pcie_aspm=off
   ...

umount /mnt

Remove the MMC boot partition so it boots to NVMe:

sudo fdisk /dev/mmcblk0
d (to delete partition)
w (to write partition table)
sudo reboot

After reboot:

debian@nanopi-r5s-arm64:~$ df
Filesystem     1K-blocks   Used Available Use% Mounted on
udev             1863616      0   1863616   0% /dev
tmpfs             382804    380    382424   1% /run
/dev/nvme0n1p1 480718012 696840 460450364   1% /
tmpfs            1914000      0   1914000   0% /dev/shm
tmpfs               5120      0      5120   0% /run/lock
tmpfs             382800      0    382800   0% /run/user/1000

Note: You can also write u-boot to the internal eMMC which is: /dev/mmcblk1

svpcom commented 11 months ago

@inindev Unfortunately this doesn't help with WD Red:

$ cat /proc/cmdline 
root=UUID=6bbd00fc-aec1-45cc-a200-19e3acfd81a0 ro pcie_aspm=off rootwait
$ uname -a
Linux nanopi-r5s-arm64 6.1.0-10-arm64 #1 SMP Debian 6.1.38-1 (2023-07-14) aarch64 GNU/Linux
$ lspci
0000:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01)
0000:01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
0001:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01)
0001:01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
0002:00:00.0 PCI bridge: Rockchip Electronics Co., Ltd RK3568 Remote Signal Processor (rev 01)

dmesg.txt

linux from frienlyarm:

$ lspci
0000:00:00.0 PCI bridge: Fuzhou Rockchip Electronics Co., Ltd Device 3566 (rev 01)
0000:01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
0001:10:00.0 PCI bridge: Fuzhou Rockchip Electronics Co., Ltd Device 3566 (rev 01)
0001:11:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
0002:20:00.0 PCI bridge: Fuzhou Rockchip Electronics Co., Ltd Device 3566 (rev 01)
0002:21:00.0 Non-Volatile memory controller: Sandisk Corp WD Black SN750 / PC SN730 NVMe SSD

It is interesting that PCI bridge has different address

inindev commented 11 months ago

@inindev Unfortunately this doesn't help with WD Red:

Any idea if it expected that it identifies itself as a WD Black under frienlyarm?

Sandisk Corp WD Black SN750 / PC SN730 NVMe SSD

svpcom commented 11 months ago

It seems that only differs by firmware. WD Red designed for write-intensive tasks like CCTV cameras or NAS

@inindev Unfortunately this doesn't help with WD Red:

Any idea if it expected that it identifies itself as a WD Black under frienlyarm?

Sandisk Corp WD Black SN750 / PC SN730 NVMe SSD

inindev commented 11 months ago

I am afraid I probably will not be much help in working out NVMe issues with PCI in the debian kernel. Looking around at others experiencing such issues, using nvme_core.default_ps_max_latency_us=0 pcie_aspm=off might be worth trying. I will leave the issue open in case someone comes along with other suggestions.

svpcom commented 11 months ago

@inindev I think the root of problem is invalid PCIe bridge setup, because device is not visible by lspci and after force pci bus rescan it show warning about invalid bridge config

inindev commented 11 months ago

There are a few differences between the friendlyarm device tree and the upstream device tree that might be worth trying. Are you setup to compile the device tree?

inindev commented 11 months ago

One difference is the max-link-speed = <2>; parameter. Adding it to the device tree allows me to boot my WD_BLUE without the kernel parameter. This is encouraging if you want to try it: https://drive.google.com/file/d/16QwGvavvObl0k5Nt94NhnzN5S2S3a7fN/view?usp=sharing

Note: I had to remove the pcie_aspm=off kernel command line parameter to get this new device tree above to boot / detect the NVMe.

svpcom commented 11 months ago

No change in case of NVMe, but with max-link-speed= <2>; and without pcie_aspm=off several interesting happened:

  1. disappears r8169 0000:01:00.0: can't read MAC address, setting random one and r8169 0001:01:00.0: can't read MAC address, setting random one
  2. rockchip-dw-pcie 3c0800000.pcie: PCIe Gen.3 x1 link up --> rockchip-dw-pcie 3c0800000.pcie: PCIe Gen.2 x1 link up
  3. rockchip-rga fdeb0000.rga: Registered rockchip-rga as /dev/video0
  4. rk808-rtc rk808-rtc: setting system clock to 2017-08-06T02:56:14 UTC (1501988174) -> rtc-hym8563 5-0051: setting system clock to 2023-09-26T14:22:07 UTC (1695738127)
  5.  - vcca1v8_image: Bringing 600000uV into 1800000-1800000uV
     - vccio_acodec: Bringing 600000uV into 3300000-3300000uV
     - vdda0v9_image: Bringing 600000uV into 950000-950000uV
     + vccio_acodec: disabling
     + vdd_npu: disabling
     + vdda0v9_image: Bringing 900000uV into 950000-950000uV

dmesg.sorted.diff.txt

inindev commented 11 months ago

I am unsure if this is even a valid node, but another difference is the num-viewport = <4>; node. This dtb still works for me if you want to try it.

https://drive.google.com/file/d/1akXUibWhxobbKVMGOJMBi3WI2LhvvalB/view?usp=sharing

Here are the additions at this point:

&pcie3x1 {
        num-lanes = <1>;
        num-viewport = <4>;
        reset-gpios = <&gpio0 RK_PA0 GPIO_ACTIVE_HIGH>;
        vpcie3v3-supply = <&vcc3v3_pcie>;
        status = "okay";
};

&pcie3x2 {
        max-link-speed = <2>;
        num-lanes = <1>;
        num-ib-windows = <8>;
        num-ob-windows = <8>;
        num-viewport = <4>;
        reset-gpios = <&gpio2 RK_PD6 GPIO_ACTIVE_HIGH>;
        vpcie3v3-supply = <&vcc3v3_pcie>;
        status = "okay";
};
svpcom commented 11 months ago

NVMe still not visible. Also no changes in dmesg compared to previous verison

inindev commented 11 months ago

Comparing the friendlyarm device tree to the upstream device tree I noticed a large difference between the startup-delay-us of the two pci voltage regulator definitions. I reduced the upstream one to match friendlyarm if you want to try another iteration:

https://drive.google.com/file/d/1XyMA-GEzNKGhbSCLRw7tOEGv8c93E7XM/view?usp=sharing

svpcom commented 11 months ago

NVMe still not visible. Also no changes in dmesg compared to previous verison. After pci rescan nvme detected and hangs as before.

Kn0ax commented 11 months ago

Hi, I just wanted to thank for your work and report that, Crucial P2 boots from NVMe.

image
arbv commented 10 months ago

Just want to report that SK Hynix Gold P31 2TB also works fine.

inindev commented 8 months ago

not all NVMe devices appear to work with the nanopi r5s

this topic lists several known working options

known working

known not working

welovewebs commented 3 weeks ago

Just noting the following here in case it helps (sorry if it's bad form to dig this up)

dmesg -l 0,1,2,3 returns

pci 0002:01:00.0: calc_l12_pwron: Invalid T_PwrOn scale: 3 pci 0002:01:00.0: BAR0: error updating (0xf0200004 != 0xffffffff) pci 0002:01:00.0: BAR0: error updating (high 0x00000000 != 0xffffffff) pci 0002:01:00.0: BAR4: error updating (0xf0204004 != 0xffffffff) pci 0002:01:00.0: BAR4: error updating (high 0x00000000 != 0xffffffff)

lspci returns (from a working OS)

0002:01:00.0 Non-Volatile memory controller: Sandisk Corp HD Blue SN570 NVMe SSD 1TB

The problem is not present on FriendlyWrt (or OpenWRT 23.05) or the Ubuntu builds based on Focal, both of which initialise the NVMe successfully. I note that adding pcie_aspm=off would work for this particular device. It's strange how Ubuntu and the WRTs are both OK

inindev commented 3 weeks ago

can you try this version: https://github.com/inindev/debian-image/releases/tag/v12.6

welovewebs commented 3 weeks ago

Writing to SD...

welovewebs commented 3 weeks ago

Bah, same problem :/ Going to try pcie_aspm=off, as I haven't actually tried that yet

inindev commented 3 weeks ago

Bah, same problem :/ Going to try pcie_aspm=off, as I haven't actually tried that yet

Can you post a listing of the /boot directory?

ll /boot
welovewebs commented 3 weeks ago

Can confirm pcie_aspm=off works

Will just get you that listing (does it make any difference now pcie_aspm=off is there or should I remove that and reboot first?)

inindev commented 3 weeks ago

I wanted to be sure it is using the correct .dtb (the flag wont matter).

-rw-r--r-- 1 root root    58986 Jul  2 21:46 rk3568-nanopi-r5s.dtb
lrwxrwxrwx 1 root root       21 Jul  2 21:46 rk3568-nanopi-r5s.dtb-6.1.0-22-arm64 -> rk3568-nanopi-r5s.dtb
welovewebs commented 3 weeks ago

Shows rk3568-nanopi-r5s.dtb 58986 bytes rk3568-nanopi-r5s.dtb-6.1.0-22-arm64 > rk3568-nanopi-r5s.dtb 21 bytes

So as you posted above

welovewebs commented 3 weeks ago

How are Ubuntu and the WRTs working? Different DTBs?

welovewebs commented 3 weeks ago

Would it help if I flashed Ubuntu back on it and got the DTB from that?

EDIT: Realised I can edit :) I have Ubuntu still on the MMC, so can just boot from that if it helps