Test GPU (ASRock Rack M2_VGA, based on SM750)

[geerlingguy]

This is a fun one—it's definitely not the kind of GPU where people would ask "will it run Crysis", especially since a headline feature is 2D graphics acceleration (not even 3D yet!).

But the ASRock Rack M2_VGA is an M.2 form factor graphics card that sports a lone VGA port and 16MB (yeah, MB, not GB) of DDR graphics memory.

I doubt it will even be as fast as the built-in graphics on the Pi, but it would be interesting to see if it works. It uses the SiliconMotion SM750 graphics chip, which actually supports up to two DVI/HDMI/VGA displays, as well as two video inputs which can be overlaid on those outputs.

The chip is mostly known for being helpful in embedded or server graphics situations, and is not a 'powerhouse' by any means. Just a little utilitarian chip that sips less than 2W of power maximum (making it suitable for lower-power scenarios where you still need a display or two, but don't do gaming or ML/AI applications on it).

It seems like there's a mainline driver since a few years ago (SM750), and it would be interesting to see if it 'just works' (compared to the other cards). It seems like the chip itself uses BIOS (and was designed in 2012), so that gives me a little pause.

But the chip is simple enough and documented enough that I wonder if we could bring it up manually if the driver starts barfing on memory allocations like all the other GPUs I've tested have (AMD and Nvidia).

[geerlingguy]

Other server vendors also build the same chip (SM750) into full PCIe cards, e.g. Sunix's VGA0419L PCI-E card.

And in this video, a rep demos a USB version of the video card and a few other form factors.

Finally, some commentary from Phoronix seems to indicate the early versions of the driver were a bit rough (but hopefully adequate for ARM64... we'll see!).

[geerlingguy]

I was hoping the card would leech power off the PCI-E bus but it looks like you have to use the included 4-pin molex to 3-pin jumper to power the board, otherwise it doesn't show up on the Pi at all.

I'm using an external / separate PSU and it looks like it puts 5v on the outer pins and GND in the middle.

Anyways, here's the deets:

pi@raspberrypi:~ $ sudo lspci -vvvv
...
01:00.0 VGA compatible controller: Silicon Motion, Inc. SM750 (rev a1) (prog-if 00 [VGA controller])
    Subsystem: Silicon Motion, Inc. SM750
    Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 255
    Region 0: Memory at <unassigned> (32-bit, prefetchable) [disabled]
    Capabilities: [40] Power Management version 3
        Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
        Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [70] Express (v2) Legacy Endpoint, MSI 00
        DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
            ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
        DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
            RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
            MaxPayload 128 bytes, MaxReadReq 512 bytes
        DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
        LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <16us
            ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp-
        LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
        LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
             EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
    Capabilities: [b0] MSI-X: Enable- Count=1 Masked-
        Vector table: BAR=5 offset=00000000
        PBA: BAR=5 offset=00000000
    Capabilities: [100 v1] Advanced Error Reporting
        UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
        CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
        AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
    Capabilities: [140 v1] Virtual Channel
        Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
        Arb:    Fixed- WRR32- WRR64- WRR128-
        Ctrl:   ArbSelect=Fixed
        Status: InProgress-
        VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
            Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
            Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
            Status: NegoPending- InProgress-

[geerlingguy]

$ dmesg
...
[    1.013251] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[    1.013270] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[    1.013327] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x0603ffffff -> 0x00f8000000
[    1.013383] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x00ffffffff -> 0x0100000000
[    1.047197] brcm-pcie fd500000.pcie: link up, 2.5 GT/s x1 (SSC)
[    1.047488] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
[    1.047503] pci_bus 0000:00: root bus resource [bus 00-ff]
[    1.047520] pci_bus 0000:00: root bus resource [mem 0x600000000-0x603ffffff] (bus address [0xf8000000-0xfbffffff])
[    1.047572] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
[    1.047792] pci 0000:00:00.0: PME# supported from D0 D3hot
[    1.051200] pci 0000:00:00.0: bridge configuration invalid ([bus ff-ff]), reconfiguring
[    1.051391] pci 0000:01:00.0: [126f:0750] type 00 class 0x030000
[    1.051459] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x03ffffff pref]
[    1.051487] pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x001fffff]
[    1.051576] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[    1.051768] pci 0000:01:00.0: supports D1
[    1.051779] pci 0000:01:00.0: PME# supported from D0 D1 D3hot
[    1.051973] pci 0000:01:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[    1.055078] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[    1.055115] pci 0000:00:00.0: BAR 8: no space for [mem size 0x06000000]
[    1.055127] pci 0000:00:00.0: BAR 8: failed to assign [mem size 0x06000000]
[    1.055143] pci 0000:01:00.0: BAR 0: no space for [mem size 0x04000000 pref]
[    1.055155] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x04000000 pref]
[    1.055166] pci 0000:01:00.0: BAR 1: no space for [mem size 0x00200000]
[    1.055178] pci 0000:01:00.0: BAR 1: failed to assign [mem size 0x00200000]
[    1.055190] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00010000 pref]
[    1.055201] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00010000 pref]
[    1.055214] pci 0000:00:00.0: PCI bridge to [bus 01]

So might need to expand the BAR. Good thing I don't see any IO BAR in there 😅

Anyways, that's all for now, I'm going to recompile the kernel later (maybe tomorrow, maybe next week) and add in the SM750 driver, and see where we end up from there. All using the 64-bit Pi OS beta for now.

[geerlingguy]

Enabled:

Device Drivers > Staging drivers > Silicon Motion SM750 framebuffer support

Cross-compiling on kernel rpi-5.10.y branch.

[PixlRainbow]

as well as two video inputs which can be overlaid on those outputs.

Pi HUD? 🤔

[69K-ram]

I was just about to suggest this to "GPU" to you after seeing the LTT video about it, but you already have one lol. Good luck! Maybe this one is so low spec it will just work?

[geerlingguy]

Reference for anyone who doesn't subscribe to LTT: THIS is a Graphics Card..??

(I wish I had a team of tech nuts who could do all the testing and script writing for me... just jelly :)

[geerlingguy]

Well that's annoying. After recompiling the kernel and booting with the VGA card powered up and plugged in, the Pi starts booting to the point where the display starts flashing the cursor... then completely freezes.

Here's what that looks like:

(That signal's coming through the HDMI port on the IO Board, not from the VGA output.)

I've tried rebooting multiple times, but it always gets stuck a few seconds in and then there's no response from anything. If I turn off power to the VGA card, it boots just fine.

[geerlingguy]

I used sudo raspi-config to boot to console (instead of GUI), and also to disable the splash screen, and then rebooted with the VGA card enabled so I could see what was coming out in the console.

Every single time it gets stuck around here:

[   OK   ] Started Forward Password Requests to Plymouth Directory Watch.

So maybe something that's trying to initialize some graphics rendering (the Plymouth Boot Screen...). The annoying thing is there doesn't seem to be a separate driver for the SM750 so I can't just disable it using a modprobe blacklist.

This could be the shortest test cycle ever—I can't yet think of anything else to try :/

[geerlingguy]

Just noting that without the card powered up, when it boots, it hits that 'Show Plymouth Boot Screen' in the log, then the screen flashes to black, then the screen comes back, outputs some more startup data, then ends with:

           Starting Terminate Plymouth Boot Screen...

So something in the Plymouth Boot Screen process seems to kinda explode when this card is active.

[geerlingguy]

Uninstalling plymouth in light of the fact that I don't see how to completely disable it otherwise (all the online guides assume you're using GRUB, but Pi OS does not):

sudo apt-get purge -y plymouth*

[geerlingguy]

Well, now it locks up completely at:

[   OK   ] Started udev Kernel Device Manager

So at least it doesn't lock up at the Plymouth stage anymore :D

[geerlingguy]

I'm recompiling the kernel without Silicon Motion SM750 framebuffer support to see if it makes a difference.

[geerlingguy]

Okay, so without that compiled in, the thing boots with the board powered up... and even better, with the updated kernel (which has the better BAR space mapping committed a few months ago), the registers all look good:

[    1.245812] brcm-pcie fd500000.pcie: host bridge /scb/pcie@7d500000 ranges:
[    1.247456] brcm-pcie fd500000.pcie:   No bus range found for /scb/pcie@7d500000, using [bus 00-ff]
[    1.249244] brcm-pcie fd500000.pcie:      MEM 0x0600000000..0x063fffffff -> 0x00c0000000
[    1.251088] brcm-pcie fd500000.pcie:   IB MEM 0x0000000000..0x00ffffffff -> 0x0100000000
[    1.286525] brcm-pcie fd500000.pcie: link up, 2.5 GT/s PCIe x1 (SSC)
[    1.287844] brcm-pcie fd500000.pcie: PCI host bridge to bus 0000:00
[    1.288874] pci_bus 0000:00: root bus resource [bus 00-ff]
[    1.289826] pci_bus 0000:00: root bus resource [mem 0x600000000-0x63fffffff] (bus address [0xc0000000-0xffffffff])
[    1.291763] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
[    1.292968] pci 0000:00:00.0: PME# supported from D0 D3hot
[    1.297601] pci 0000:00:00.0: bridge configuration invalid ([bus ff-ff]), reconfiguring
[    1.299700] pci 0000:01:00.0: [126f:0750] type 00 class 0x030000
[    1.300732] pci 0000:01:00.0: reg 0x10: [mem 0x00000000-0x03ffffff pref]
[    1.302611] pci 0000:01:00.0: reg 0x14: [mem 0x00000000-0x001fffff]
[    1.303687] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[    1.305835] pci 0000:01:00.0: supports D1
[    1.306794] pci 0000:01:00.0: PME# supported from D0 D1 D3hot
[    1.307945] pci 0000:01:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
[    1.323777] pci_bus 0000:01: busn_res: [bus 01-ff] end is updated to 01
[    1.324778] pci 0000:00:00.0: BAR 8: assigned [mem 0x600000000-0x605ffffff]
[    1.326596] pci 0000:01:00.0: BAR 0: assigned [mem 0x600000000-0x603ffffff pref]
[    1.328471] pci 0000:01:00.0: BAR 1: assigned [mem 0x604000000-0x6041fffff]
[    1.330393] pci 0000:01:00.0: BAR 6: assigned [mem 0x604200000-0x60420ffff pref]
[    1.332347] pci 0000:00:00.0: PCI bridge to [bus 01]
[    1.333381] pci 0000:00:00.0:   bridge window [mem 0x600000000-0x605ffffff]

Going to do some testing and see what happens with X.org.

[geerlingguy]

Well, nothing. It only outputs over HDMI.

[geerlingguy]

Hmm... driver being worked on over here: https://gitlab.com/sudipm/sm750/-/tree/sm750/drivers/gpu/drm/sm750 (from 2 years ago, ha!).

See: https://github.com/raspberrypi/linux/blob/rpi-5.10.y/drivers/staging/sm750fb/TODO

I emailed one of the contacts listed in the driver docs... who knows if I'll get a response.

[geerlingguy]

So... I tried pulling the patch from that drm driver and popping it into the 5.10.y branch—but when I build I get:

  CC [M]  drivers/gpu/drm/sm750/smi_drv.o
drivers/gpu/drm/sm750/smi_drv.c:12:10: fatal error: drm/drmP.h: No such file or directory
 #include <drm/drmP.h>
          ^~~~~~~~~~~~
compilation terminated.
make[4]: *** [scripts/Makefile.build:279: drivers/gpu/drm/sm750/smi_drv.o] Error 1
make[3]: *** [scripts/Makefile.build:496: drivers/gpu/drm/sm750] Error 2
make[3]: *** Waiting for unfinished jobs....

So it looks like that driver may be built for an older kernel version and isn't compatible with the latest kernel :/

[geerlingguy]

Looks like that change was in kernel 5.5—so maybe running the current stable kernel version (5.4) would let it work? It's worth a try... See https://github.com/DisplayLink/evdi/issues/185

[geerlingguy]

Yeah... so doing a few manual patches to that patch to try to make it compile leads to dozens more compile errors, so I'm going to drop trying to get it working on 5.10.y, and jump back to 5.4.y.

[geerlingguy]

With the patch in place, the kernel build on 5.4.y seemed to work. Copied it over to the Pi, rebooted and...

[    1.324708] pci 0000:00:00.0: BAR 8: no space for [mem size 0x06000000]
[    1.325653] pci 0000:00:00.0: BAR 8: failed to assign [mem size 0x06000000]
[    1.327425] pci 0000:01:00.0: BAR 0: no space for [mem size 0x04000000 pref]
[    1.329274] pci 0000:01:00.0: BAR 0: failed to assign [mem size 0x04000000 pref]
[    1.331175] pci 0000:01:00.0: BAR 1: no space for [mem size 0x00200000]
[    1.332164] pci 0000:01:00.0: BAR 1: failed to assign [mem size 0x00200000]
[    1.334106] pci 0000:01:00.0: BAR 6: no space for [mem size 0x00010000 pref]
[    1.336053] pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00010000 pref]

Have to adjust the BAR space manually on this old kernel.

With 1 GB BAR space, getting:

[    1.324861] pci 0000:00:00.0: BAR 8: assigned [mem 0x600000000-0x605ffffff]
[    1.326682] pci 0000:01:00.0: BAR 0: assigned [mem 0x600000000-0x603ffffff pref]
[    1.328536] pci 0000:01:00.0: BAR 1: assigned [mem 0x604000000-0x6041fffff]
[    1.330481] pci 0000:01:00.0: BAR 6: assigned [mem 0x604200000-0x60420ffff pref]

...but nothing in dmesg mentioning SM750, nor anything out of the VGA output. So... how do I get Linux/RPi to recognize the card using the new driver? Maybe I have to manually load it in somehow?

[geerlingguy]

Oh... well... that's not right:

$ uname -r
5.10.9-v8+

[geerlingguy]

Ugh... I can't get these drivers to build for the life of me!

In file included from drivers/gpu/drm/sm750/smi_drv.c:15:
drivers/gpu/drm/sm750/smi_drv.h:178:32: error: field ‘mem_global_ref’ has incomplete type
    struct drm_global_reference mem_global_ref;
                                ^~~~~~~~~~~~~~
drivers/gpu/drm/sm750/smi_drv.h:179:29: error: field ‘bo_global_ref’ has incomplete type
    struct ttm_bo_global_ref bo_global_ref;
                             ^~~~~~~~~~~~~
drivers/gpu/drm/sm750/smi_drv.h:296:50: warning: ‘struct reservation_object’ declared inside parameter list will not be visible outside of this definition or declaration
      uint32_t flags, struct sg_table *sg, struct reservation_object *resv, struct smi_bo **psmibo);
                                                  ^~~~~~~~~~~~~~~~~~
drivers/gpu/drm/sm750/smi_drv.c: In function ‘smi_drm_freeze’:
drivers/gpu/drm/sm750/smi_drv.c:107:2: error: implicit declaration of function ‘drm_kms_helper_poll_disable’; did you mean ‘drm_fb_helper_pan_display’? [-Werror=implicit-function-declaration]
  drm_kms_helper_poll_disable(dev);
  ^~~~~~~~~~~~~~~~~~~~~~~~~~~
  drm_fb_helper_pan_display
drivers/gpu/drm/sm750/smi_drv.c: In function ‘smi_drm_resume’:
drivers/gpu/drm/sm750/smi_drv.c:154:2: error: implicit declaration of function ‘drm_kms_helper_poll_enable’; did you mean ‘drm_fb_helper_fill_info’? [-Werror=implicit-function-declaration]
  drm_kms_helper_poll_enable(dev);
  ^~~~~~~~~~~~~~~~~~~~~~~~~~
  drm_fb_helper_fill_info
drivers/gpu/drm/sm750/smi_drv.c:144:21: warning: unused variable ‘sdev’ [-Wunused-variable]
  struct smi_device *sdev = dev->dev_private;
                     ^~~~
drivers/gpu/drm/sm750/smi_drv.c: At top level:
drivers/gpu/drm/sm750/smi_drv.c:276:39: error: ‘DRIVER_IRQ_SHARED’ undeclared here (not in a function); did you mean ‘TIMER_IRQSAFE’?
  .driver_features = DRIVER_HAVE_IRQ | DRIVER_IRQ_SHARED | DRIVER_GEM |
                                       ^~~~~~~~~~~~~~~~~
                                       TIMER_IRQSAFE
drivers/gpu/drm/sm750/smi_drv.c:277:9: error: ‘DRIVER_PRIME’ undeclared here (not in a function); did you mean ‘DRIVER_NAME’?
         DRIVER_PRIME | DRIVER_MODESET,
         ^~~~~~~~~~~~
         DRIVER_NAME
drivers/gpu/drm/sm750/smi_drv.c:309:3: error: ‘struct drm_driver’ has no member named ‘gem_prime_res_obj’; did you mean ‘gem_prime_export’?
  .gem_prime_res_obj = smi_gem_prime_res_obj,
   ^~~~~~~~~~~~~~~~~
   gem_prime_export
drivers/gpu/drm/sm750/smi_drv.c:309:23: error: initialization of ‘struct sg_table * (*)(struct drm_gem_object *)’ from incompatible pointer type ‘struct reservation_object * (*)(struct drm_gem_object *)’ [-Werror=incompatible-pointer-types]
  .gem_prime_res_obj = smi_gem_prime_res_obj,
                       ^~~~~~~~~~~~~~~~~~~~~
drivers/gpu/drm/sm750/smi_drv.c:309:23: note: (near initialization for ‘driver.gem_prime_get_sg_table’)
In file included from drivers/gpu/drm/sm750/smi_fbdev.c:15:
drivers/gpu/drm/sm750/smi_drv.h:178:32: error: field ‘mem_global_ref’ has incomplete type
    struct drm_global_reference mem_global_ref;
                                ^~~~~~~~~~~~~~
drivers/gpu/drm/sm750/smi_drv.h:179:29: error: field ‘bo_global_ref’ has incomplete type
    struct ttm_bo_global_ref bo_global_ref;
                             ^~~~~~~~~~~~~
drivers/gpu/drm/sm750/smi_drv.h:296:50: warning: ‘struct reservation_object’ declared inside parameter list will not be visible outside of this definition or declaration
      uint32_t flags, struct sg_table *sg, struct reservation_object *resv, struct smi_bo **psmibo);
                                                  ^~~~~~~~~~~~~~~~~~
drivers/gpu/drm/sm750/smi_fbdev.c: In function ‘smifb_create_object’:
drivers/gpu/drm/sm750/smi_fbdev.c:48:21: warning: unused variable ‘cdev’ [-Wunused-variable]
  struct smi_device *cdev = dev->dev_private;
                     ^~~~
cc1: some warnings being treated as errors
make[4]: *** [scripts/Makefile.build:266: drivers/gpu/drm/sm750/smi_drv.o] Error 1
make[4]: *** Waiting for unfinished jobs....
drivers/gpu/drm/sm750/smi_fbdev.c: In function ‘smifb_create’:
drivers/gpu/drm/sm750/smi_fbdev.c:161:2: error: implicit declaration of function ‘drm_fb_helper_fill_fix’; did you mean ‘drm_fb_helper_fill_info’? [-Werror=implicit-function-declaration]
  drm_fb_helper_fill_fix(info, fb->pitches[0], fb->format->depth);
  ^~~~~~~~~~~~~~~~~~~~~~
  drm_fb_helper_fill_info
  AR      drivers/i3c/built-in.a
drivers/gpu/drm/sm750/smi_fbdev.c:163:2: error: implicit declaration of function ‘drm_fb_helper_fill_var’; did you mean ‘drm_fb_helper_fill_info’? [-Werror=implicit-function-declaration]
  drm_fb_helper_fill_var(info, &gfbdev->helper, sizes->fb_width,
  ^~~~~~~~~~~~~~~~~~~~~~
  drm_fb_helper_fill_info
drivers/gpu/drm/sm750/smi_fbdev.c:175:58: error: ‘struct ttm_buffer_object’ has no member named ‘vma_node’
  drm_vma_offset_remove(&bo->bo.bdev->vma_manager, &bo->bo.vma_node);
                                                          ^
  CC [M]  drivers/gpu/drm/tiny/ili9341.o
drivers/gpu/drm/sm750/smi_fbdev.c: In function ‘smi_fbdev_destroy’:
drivers/gpu/drm/sm750/smi_fbdev.c:205:4: error: implicit declaration of function ‘drm_gem_object_unreference_unlocked’; did you mean ‘drm_gem_object_put_unlocked’? [-Werror=implicit-function-declaration]
    drm_gem_object_unreference_unlocked(gfb->obj);
    ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    drm_gem_object_put_unlocked
  AR      drivers/idle/built-in.a
cc1: some warnings being treated as errors
make[4]: *** [scripts/Makefile.build:266: drivers/gpu/drm/sm750/smi_fbdev.o] Error 1
make[3]: *** [scripts/Makefile.build:500: drivers/gpu/drm/sm750] Error 2
make[3]: *** Waiting for unfinished jobs....

I think I may have to throw in the towel on this GPU as well.

Why are all the [okay but not great] working drivers only written for Windows?

[69K-ram]

Respect for even getting this far, I wouldn't have slightest idea where to start, and I sure wouldn't have tried that long.

In the LTT video, he mentions it's using a Windows feature called WARP. And the Wikipedia article about WARP says it uses "just-in-time compilation to x86 machine code". Is it possible the driver is dependent on x86 instructions or something like that? I have no idea how the Linux driver for it works though.

[PixlRainbow]

The thing is though, WARP is a DirectX feature, not a driver feature.

[Coreforge]

Could you try disabling fbdev emulation? Since it crashes after udev, it could be the console trying to access the fbdev, which could lock up the whole pi if something doesn't work, like using 64bit writes. You'll have to add drm_kms_helper.fbdev_emulation=0 to cmdline.txt. Most things involving GRUB also just change the kernel command line, so you can do the same by adding the arguments to cmdline.txt

[geerlingguy]

I added drm_kms_helper.fbdev_emulation=0 to /boot/cmdline.txt on the Pi, and am building the latest 5.10.y kernel with the SM750 staging driver... we'll see how this goes.

[geerlingguy]

@Coreforge - Sadly, that seems to have not made any difference:

https://user-images.githubusercontent.com/481677/106538867-9586bb00-64c2-11eb-9c1b-e6e1fcbbf034.mov

[geerlingguy]

Digging through the driver code, I do notice a bunch of memset_io() calls in sm750.c:

    if (info) {
        par = info->par;
        crtc = &par->crtc;
        cursor = &crtc->cursor;
        memset_io(cursor->vstart, 0x0, cursor->size);
        memset_io(crtc->vScreen, 0x0, crtc->vidmem_size);
        lynxfb_ops_set_par(info);
        fb_set_suspend(info, 0);
    }

    info = sm750_dev->fbinfo[1];

    if (info) {
        par = info->par;
        crtc = &par->crtc;
        cursor = &crtc->cursor;
        memset_io(cursor->vstart, 0x0, cursor->size);
        memset_io(crtc->vScreen, 0x0, crtc->vidmem_size);
        lynxfb_ops_set_par(info);
        fb_set_suspend(info, 0);
    }

Also:

    pr_info("crtc->cursor.mmio = %p\n", crtc->cursor.mmio);
    crtc->cursor.maxH = crtc->cursor.maxW = 64;
    crtc->cursor.size = crtc->cursor.maxH * crtc->cursor.maxW * 2 / 8;
    crtc->cursor.vstart = sm750_dev->pvMem + crtc->cursor.offset;

    memset_io(crtc->cursor.vstart, 0, crtc->cursor.size);
    if (!g_hwcursor) {
        lynxfb_ops.fb_cursor = NULL;
        sm750_hw_cursor_disable(&crtc->cursor);
    }

And:

    /* call chip specific mmap routine */
    err = hw_sm750_map(sm750_dev, pdev);
    if (err)
        return err;

    if (!sm750_dev->mtrr_off)
        sm750_dev->mtrr.vram = arch_phys_wc_add(sm750_dev->vidmem_start,
                            sm750_dev->vidmem_size);

    memset_io(sm750_dev->pvMem, 0, sm750_dev->vidmem_size);

    pci_set_drvdata(pdev, sm750_dev);

Judging by experiences in both https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/4#issuecomment-766275193 and https://github.com/geerlingguy/raspberry-pi-pcie-devices/issues/6#issuecomment-728751399, perhaps changing some of this could help.

[geerlingguy]

paul maydaynight on YouTube asks if I've considered the work from the Linaro team on this driver...

I found this listing of patches by gary guo, like one mentioning bar space addressing and I might try to see if I can reach him.

Edit: The emails associated with those patches seem dead.

[Hacker1245]

paul maydaynight on YouTube asks if I've considered the work from the Linaro team on this driver...

I found this listing of patches by gary guo, like one mentioning bar space addressing and I might try to see if I can reach him.

Edit: The emails associated with those patches seem dead.

Not sure if that would help or if it is even the correct guy but maybe you can try contacting him via LinkedIn https://www.linkedin.com/in/heyi-guo-70129288

[geerlingguy]

@Hacker1245 Worth a try, I sent an inmail message :)

Also got a response from ASRock, basically "This thing is meant for Servers and they're mostly Intel, soooo..." :D

[kwinz]

@geerlingguy After reading through your complete journey (and I am really impressed with and deeply thankful for your perseverance) with the compute module 4 I think everything so far has been much, much harder for us because of this:

I think we may be out of luck, in this case, as @pelwell mentions that the spec sheet for the BCM2711 says: Supports accessing external PCIe configuration space and memory space (no support for I/O space). [...] The PCI spec requires it, but not all platforms (especially non-x86 platforms) support that.

I think it's essential to communicate to the Raspberry foundation that we would really love to see that supported in the Pi 5 / Compute Module 5. They should know about our struggle. I mean maybe we could hack around that limitation with certain GPU model drivers, but the only true fix is to have a Broadcom SOC that properly implements the PCIe specification. Is the Pi foundation aware of this problem yet?

[Coreforge]

With newer Cards the IO space shouldn't be the issue as it mostly isn't required. The radeon cards for example typically use the IO BAR for atom IO, but can also do the same over regular bars. I think the bigger issue is that the pcie controller doesn't support 64bit accesses, so you have to use the 32bit OS, which doesn't solve everything either, or replace every relevant 64bit access in the driver with two 32bit ones, which is tedious.

[geerlingguy]

Yeah, I don't think most driver maintainers really care about the IO bar thing—the cards that depend on it are generally 10+ years old, and many (most?) drivers nowadays don't care if it's not available.

The bigger problem I've started hitting (besides little bugs that affect all ARM64 systems like the patch required to get the Intel WiFi driver working) is the memory issues @Coreforge alludes to. I'm pretty sure that's what's hitting us on the GPUs at this point, and it's also what's making it hard to get things like the Google Coral TPU working, and what we had to spend some time back-and-forth with on the Broadcom RAID card I just covered recently.

[kwinz]

Very interresting! Thanks for the update, I was wondering if you are talking with the Raspberry foundation guys, since they are the Broadcom customers. Maybe it could be fixed in the next hardware revision if we talk to each other.

bigger issue is that the pcie controller doesn't support 64bit accesses

@Coreforge Sorry for the naive question: Is that a hardware or a software (Kernel driver) issue? Not to hijack this Github issue, but do you have a link with more info on the pcie controller limitations?

[Coreforge]

I think it's a hardware limitation, but I don't know much about that stuff either. As long as a card works on 32bit systems though, it shouldn't matter too much, it just might be a lot of work to replace all 64bit accesses. It doesn't work to just not use 64bit accesses for the whole kernel though, so they need to be individually replaced in the drivers. I don't have any more information on the controllers limitations either.

[kwinz]

Thanks @Coreforge. Would the maintainers not resist to Raspberry Pi specific changes spread all over the Kernel in individual drivers? It seems to me there should be a better solution. But I don't know enough unfortunately.

[PixlRainbow]

It doesn't work to just not use 64bit accesses for the whole kernel though, so they need to be individually replaced in the drivers

Wait, what do you mean? What happens if you just modify the function that performs 64 bit reads such that it internally performs two 32 bit reads, but retains its original interface and apparent behavior transparently?

[pelwell]

@kwinz The raspberrypi/linux repo is open to hosting modest downstream patches to thirdparty drivers if it allows them to run on BCM2711.

[Coreforge]

Wait, what do you mean? What happens if you just modify the function that performs 64 bit reads such that it internally performs two 32 bit reads, but retains its original interface and apparent behavior transparently?

I thought this caused it to hang up, but that was only if I removed the writeq-stuff in memset_io (doesn't make too much sense why as the arm code doesn't even have anything bigger than writeb in that function and it works there). It doens't lock up if you do that, but it doesn't really help either. It could also be that I just replaced it with the wrong code, I don't really know any assembly, and I just copied the code from the writel function.

[geerlingguy]

Weird coincidence: on YouTube, Marco said he has a Mini PCIe card—Innodisk EMPV-1201, that he may be able to send for testing.

I looked on their Downloads page (search for 'EMPV-1201') and found a driver that includes Linux binaries for RHEL 6, Ubuntu 16, etc. (but no newer OSes...), but also includes the source—for an SM750!

The README included says:

1. Please copy ".so" file to "/usr/lib/xorg/modules/drivers".

2. Use command "lspci" to check "Siliconmotion SM750 Bus ID" and modify "xorg.conf" with the ID in section "Device" of Driver "siliconmotion". 

3. Please copy "xorg.conf" to ¡¨ /etc/X11.

4. Reboot your system.

Might have to see if I can build that source for the M2_VGA and get it working. Probably not, but it looks like the code is at least different than the sm750fb driver in the Linux kernel source...

[piexil]

lo_hi_readq and lo_hi_writeq exsist as helpers for doing 64bit I/o over 32bit buses, the NVMe driver uses this to do all of it's 64bit mmio.

[samveen]

Weird coincidence: on YouTube, Marco said he has a Mini PCIe card—Innodisk EMPV-1201, that he may be able to send for testing.

I looked on their Downloads page (search for 'EMPV-1201') and found a driver that includes Linux binaries for RHEL 6, Ubuntu 16, etc. (but no newer OSes...), but also includes the source—for an SM750!

@geerlingguy In case you haven't already looked at the the driver sources in the package you mentioned, they are NOT kernel drivers, but Xorg drivers. These are not going to help with the kernel driver issues.

[geerlingguy]

@samveen - Well shucks! Thanks for letting me know.

[bobafetthotmail]

A question that is not related to raspberry Pi: I've been trying to understand if this card has UEFI GOP option ROM. I need a card to use in a system where CSM is disabled and I would like to not waste power and space with a real GPU.

Did you dump the firmware of this card (and can send it to me so I can have a look) or have you tried installing it in a PC and then disabled CSM from UEFI settings? If you see something on screen connected to this card with CSM disabled then it has this option ROM, otherwise it does not.

[sfxworks]

Where can one buy this?

[bobafetthotmail]

There are some on Ebay from a german seller

[sfxworks]

Any other sources? I could have sworn I have seen these before for about 14 USD

[geerlingguy]

@sfxworks - I haven't seen them available from any retailers/suppliers so far. It seems like they're meant to be an OEM-style product integrated into server/device builds.

[TobleMiner]

Hey all,

I got hold of a SM750-based mini PCIe graphics card last weekend. Today I had some time to try it out with a CM4 and investigate the issues you are seeing here. I can confirm that the CM4 does indeed hang when trying to insert the sm750fb kernel module.
However there is a simple (partial) fix. After doing a bit of debugging it seems that any writes more than 256 byte in size to memory BARs of the SM750 cause the BCM2711 to hang. Those kinds of writes happen when memset_io is called on a memory area with write combining enabled. The following patch maps the SM750 framebuffer BAR without enabling write combining:

diff --git a/drivers/staging/sm750fb/sm750_hw.c b/drivers/staging/sm750fb/sm750_hw.c
index 7136d751cff5..01e515ed7e9b 100644
--- a/drivers/staging/sm750fb/sm750_hw.c
+++ b/drivers/staging/sm750fb/sm750_hw.c
@@ -78,8 +78,8 @@ int hw_sm750_map(struct sm750_dev *sm750_dev, struct pci_dev *pdev)
        sm750_dev->vidmem_start, sm750_dev->vidmem_size);

    /* reserve the vidmem space of smi adaptor */
-   sm750_dev->pvMem = ioremap_wc(sm750_dev->vidmem_start,
-                     sm750_dev->vidmem_size);
+   sm750_dev->pvMem = ioremap(sm750_dev->vidmem_start,
+                  sm750_dev->vidmem_size);
    if (!sm750_dev->pvMem) {
        pr_err("Map video memory failed\n");
        ret = -EFAULT;

Note however that this patch does not prevent large writes to the frambuffer BAR in general. An external application can still issue large writes that might cause a hang. It is however good enough to get a framebuffer console: You might have to remap the framebuffer console to see anything at all if you have another display attached via HDMI: sudo con2fbmap 1 1

I'm not sure why the CM4 hangs when issuing large writes to the framebuffer BAR. I've not looked into it any further yet but am thinking about attaching a JTAG debugger to my CM4 to see what exactly the CPU is doing when it hangs. I might also do more tests with other random PCIe cards and see how they behave when issuing large writes to them.

In general I think this is a good point to continue debugging from. Maybe someone at Broadcom will find this infomation useful and can give some insight into what is happening.