Getting an Nvidia GPU working on ARM (aarch64) Rockchip RK3568

[HeyMeco]

Progress status is documented here: https://github.com/HeyMeco/Rockchip-pcie-devices/blob/main/RK3568/_cards_gpu/Nvidia_RTX-3090-Founders-Edition.md

What works:

• [x] Detect the GPU
• [x] Installing the driver

Current issues that need to be solved:

• [x] Increase PCI-E Bar space so GPU can get valid memory addresses assigned. → so GPU can be enabled by the driver

Closed because of 32 MB PCI-E outbound memory limit on RK3568

Any ideas, contributions or feedback is appreciated.

[HeyMeco]

This is the iomem result:

ubuntu@nanopir5s:~$ sudo cat /proc/iomem 
0010f000-0010f0ff : 10f000.sram sram@10f000
00200000-efffffff : System RAM
  02210000-038cffff : Kernel code
  038d0000-03c8ffff : reserved
  03c90000-03eaffff : Kernel data
  e5800000-e97fffff : reserved
  e9a00000-ed5fffff : reserved
  ed7ad000-ed7cafff : reserved
  ed7cd000-edee3fff : reserved
  ef17f000-ef77ffff : reserved
  ef780000-ef7cffff : reserved
  ef7d2000-ef7d4fff : reserved
  ef7d5000-ef7e7fff : reserved
  ef7e8000-efffffff : reserved
f0000000-f00fffff : 3c0800000.pcie config
f0200000-f1ffffff : pcie@fe280000
  f0200000-f020ffff : 0002:20:00.0
  f0800000-f1ffffff : PCI Bus 0002:21
    f0800000-f087ffff : 0002:21:00.0
    f0880000-f0883fff : 0002:21:00.1
    f1000000-f1ffffff : 0002:21:00.0
f2000000-f20fffff : 3c0400000.pcie config
f2200000-f3ffffff : pcie@fe270000
  f2200000-f22fffff : PCI Bus 0001:11
    f2200000-f220ffff : 0001:11:00.0
      f2200000-f220ffff : r8169
    f2210000-f2213fff : 0001:11:00.0
  f2300000-f230ffff : 0001:10:00.0
f4000000-f40fffff : 3c0000000.pcie config
f4200000-f5ffffff : pcie@fe260000
  f4200000-f42fffff : PCI Bus 0000:01
    f4200000-f420ffff : 0000:01:00.0
      f4200000-f420ffff : r8169
    f4210000-f4213fff : 0000:01:00.0
  f4300000-f430ffff : 0000:00:00.0
fcc00000-fcc07fff : usb@fcc00000
  fcc00000-fcc07fff : xhci-hcd.0.auto usb@fcc00000
fcc0c100-fcffffff : fcc00000.usb usb@fcc00000
fd000000-fd007fff : usb@fd000000
  fd000000-fd007fff : xhci-hcd.1.auto usb@fd000000
fd00c100-fd3fffff : fd000000.usb usb@fd000000
fd400000-fd40ffff : GICD
fd460000-fd51ffff : GICR
fd800000-fd83ffff : fd800000.usb usb@fd800000
fd840000-fd87ffff : fd840000.usb usb@fd840000
fd880000-fd8bffff : fd880000.usb usb@fd880000
fd8c0000-fd8fffff : fd8c0000.usb usb@fd8c0000
fdd40000-fdd40fff : fdd40000.i2c i2c@fdd40000
fdd60000-fdd600ff : fdd60000.gpio gpio@fdd60000
fde60000-fde63fff : fde60000.gpu gpu@fde60000
fdea0000-fdea07ff : fdea0000.video-codec video-codec@fdea0400
fdea0800-fdea083f : fdea0800.iommu iommu@fdea0800
fdee0000-fdee07ff : fdee0000.video-codec video-codec@fdee0000
fdee0800-fdee083f : fdee0800.iommu iommu@fdee0800
fe040000-fe042fff : fe040000.vop vop
fe043e00-fe043eff : fe043e00.iommu iommu@fe043e00
fe043f00-fe043fff : fe043e00.iommu iommu@fe043e00
fe044000-fe044fff : fe040000.vop gamma-lut
fe0a0000-fe0bffff : fe0a0000.hdmi hdmi@fe0a0000
fe260000-fe26ffff : 3c0000000.pcie apb
fe270000-fe27ffff : 3c0400000.pcie apb
fe280000-fe28ffff : 3c0800000.pcie apb
fe2a0000-fe2affff : fe2a0000.ethernet ethernet@fe2a0000
fe2b0000-fe2b3fff : fe2b0000.mmc mmc@fe2b0000
fe310000-fe31ffff : fe310000.mmc mmc@fe310000
fe400000-fe400fff : fe400000.i2s i2s@fe400000
fe530000-fe533fff : dma-controller@fe530000
  fe530000-fe533fff : fe530000.dma-controller dma-controller@fe530000
fe550000-fe553fff : dma-controller@fe550000
  fe550000-fe553fff : fe550000.dma-controller dma-controller@fe550000
fe5e0000-fe5e0fff : fe5e0000.i2c i2c@fe5e0000
fe600000-fe6000ff : fe600000.watchdog watchdog@fe600000
fe660000-fe66001f : serial
fe710000-fe7100ff : fe710000.tsadc tsadc@fe710000
fe720000-fe7200ff : fe720000.saradc saradc@fe720000
fe740000-fe7400ff : fe740000.gpio gpio@fe740000
fe750000-fe7500ff : fe750000.gpio gpio@fe750000
fe760000-fe7600ff : fe760000.gpio gpio@fe760000
fe770000-fe7700ff : fe770000.gpio gpio@fe770000
fe820000-fe8200ff : fe820000.phy phy@fe820000
fe830000-fe8300ff : fe830000.phy phy@fe830000
fe840000-fe8400ff : fe840000.phy phy@fe840000
fe8c0000-fe8dffff : fe8c0000.phy phy@fe8c0000

This Part had the first address marked red and also should be the GPU:

300000000-33fffffff : pcie@fe260000
340000000-37fffffff : pcie@fe270000
380000000-3bfffffff : pcie@fe280000
  380000000-3bfffffff : 0002:20:00.0

3c0000000-3c03fffff : 3c0000000.pcie dbi
3c0400000-3c07fffff : 3c0400000.pcie dbi
3c0800000-3c0bfffff : 3c0800000.pcie dbi

[HeyMeco]

Well per Page 1751 of the RK3568 Technical Reference Manual the PCI-E Memory Space is limited to 32mb (per pcie device) so I guess this is definitely not possible.

[mtx512]

@HeyMeco I think your confusing RK3568 register mapping space with PCI BAR space. If you require PCI BAR space for your card then you can try allocating want you required in your dts file.

[  +0.000060] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid:
              NVRM: BAR1 is 0M @ 0x0 (PCI:0002:21:00.0)
[  +0.000008] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid:
              NVRM: BAR2 is 0M @ 0x0 (PCI:0002:21:00.0)
[  +0.000005] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid:
              NVRM: BAR3 is 0M @ 0x0 (PCI:0002:21:00.0)
[  +0.000005] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid:
              NVRM: BAR4 is 0M @ 0x0 (PCI:0002:21:00.0)