[GBM/KMS] Could not queue DRM page flip on screen HDMI1 (Device or resource busy)

[SupervisedThinking]

The issue

When I try to start a game the screen stays black but I can hear the sound playing

[i] Executing command: [`/usr/bin/retroarch.start`,`-L`,`/tmp/cores/fbneo_libretro.so`,`/storage/roms/fba/1942.zip`]
[i] Working directory: `/usr/bin`
[i] Process 2224 started
[i] ----------------------------------------
rr-config-script: Kodi service was already stopped
rr-config-script: set refresh rate to 60hz - WIP
rr-config-script: Retroarch force audio driver to PulseAudio & MIDI output to FluidSynth
[i] ----------------------------------------
[i] The external program has finished cleanly
[e] Could not queue DRM page flip on screen HDMI1 (Device or resource busy)
[e] Could not queue DRM page flip on screen HDMI1 (Device or resource busy)
[e] Could not queue DRM page flip on screen HDMI1 (Device or resource busy)

System info

I'm building a LibreELEC fork for emulation purposes, the system is a ROCKPro64 RK3399 SBC which uses linux kernel 5.10.17, mesa 21.1.0-dev & Qt5.15.2 and the whole thing uses GBM / EGL so no x11 or wayland.

https://github.com/SupervisedThinking/LibreELEC-RR/

I guess this could be related to https://github.com/moonlight-stream/moonlight-qt/issues/509#issuecomment-773819855 and basically SDL & Qt "fights" for the plane to render? Might be wrong but maybe a push into the right direction

Pegasus version

https://github.com/mmatyas/pegasus-frontend/releases/tag/weekly_2021w05

[mmatyas]

Hi, have you tried experimenting with the Qt EGLFS env vars? They're mentioned here: https://pegasus-frontend.org/docs/user-guide/platform-linux/#gpu-driver-notes. The Qt documentation also lists several more: https://doc.qt.io/qt-5/embedded-linux.html. In the case of Pegasus, only Qt does rendering, so the DRM should be free, it's just that Qt often needs a little handholding to find it.

[SupervisedThinking]

It's not like I can't start Pegasus or use it but once I start Retroarch or anything else SDL2 based I get a blank screen, sound output and the log spams the error.

These are the options I set before I run Qt stuff:

# Set Qt5 QPA EGLFS environment vars
set_QT_environment_vars() {
  oe_setup_addon service.rr-config-tool

  if [ ! "$1" = "cursor" ]; then
    export QT_QPA_EGLFS_HIDECURSOR=1
  else
    export QT_QPA_EGLFS_HIDECURSOR=0
  fi

  export QT_QPA_EGLFS_DEPTH=32
  export QT_QPA_EGLFS_PHYSICAL_HEIGHT=${RR_QT_QPA_EGLFS_PHYSICAL_HEIGHT}
  export QT_QPA_EGLFS_PHYSICAL_WIDTH=${RR_QT_QPA_EGLFS_PHYSICAL_WIDTH}
  export QT_QPA_PLATFORM=eglfs
  export QT_QPA_EGLFS_INTEGRATION=eglfs_kms
  echo "rr-config-script: set Qt5 environment variables:"
  echo "rr-config-script: hide mouse cursor   : "${QT_QPA_EGLFS_HIDECURSOR}
  echo "rr-config-script: EGLFS color depth   : "${QT_QPA_EGLFS_DEPTH}"bit"
  echo "rr-config-script: physical width      : "${QT_QPA_EGLFS_PHYSICAL_WIDTH}"mm"
  echo "rr-config-script: physical height     : "${QT_QPA_EGLFS_PHYSICAL_HEIGHT}"mm"
  echo "rr-config-script: QPA platform        : "${QT_QPA_PLATFORM}
  echo "rr-config-script: QPA platform plugin : "${QT_QPA_EGLFS_INTEGRATION}

  # Debug Qt plugins
  if [ ! -z ${DEBUG_QT_AGAIN} ]; then
    export QT_DEBUG_PLUGINS=1
    export QT_LOGGING_RULES=qt.qpa.*=true
    echo "rr-config-script: enabled Qt5 plugin debugging"
  fi
}

https://github.com/SupervisedThinking/LibreELEC-RR/blob/master-rr/packages/supervisedthinking/oem/Rockchip/config/etc/profile.d/921-video-settings-rockchip.conf#L50-L79

[mmatyas]

Ah I see! Have you tried applying this patch: https://github.com/mmatyas/pegasus-frontend/blob/master/etc/rpi4/kms_launch_fix.diff? It's a workaround for some fkms issues on the Pi 4, maybe that might help you too?

[SupervisedThinking]

I haven't yet but I'll give it a try 👍🏻

[SupervisedThinking]

Well I tried your patch but it fails because Project ERROR: Library 'libdl' is not defined.

BUILD      pegasus-frontend (target)
    TOOLCHAIN      make
Project MESSAGE: Deployment (`make install`) paths:
Project MESSAGE:   - Binaries: `/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/install_pkg/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/usr/bin`
Project MESSAGE:   - License and Readme: unset, will not install
Project MESSAGE:   - X11: Icon files: `/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/install_pkg/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/usr/share/icons/trash`
Project MESSAGE:   - X11: Desktop file: `/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/install_pkg/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/usr/share/applications`
Project MESSAGE:   - X11: AppStream file: unset, will not install
Project MESSAGE: Using Qt gamepad backend
Project MESSAGE: APNG support disabled
Project MESSAGE: Git revision: 'alpha14-113-g9abf0653-dirty'
Executing (target): make 
cd thirdparty/ && ( test -e Makefile || /home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/toolchain/bin/qmake -o Makefile /home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/build/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/thirdparty/thirdparty.pro INSTALLDIR=/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/install_pkg/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/usr/bin INSTALL_BINDIR=/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/install_pkg/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/usr/bin INSTALL_DATADIR=/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/install_pkg/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/usr/bin INSTALL_ICONDIR=/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/install_pkg/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/usr/share/icons/trash INSTALL_DESKTOPDIR=/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/install_pkg/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/usr/share/applications ) && make -f Makefile 
make[1]: Entering directory '/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/build/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/.armv8a-libreelec-linux-gnueabihf/thirdparty'
make[1]: Nothing to be done for 'first'.
make[1]: Leaving directory '/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/build/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/.armv8a-libreelec-linux-gnueabihf/thirdparty'
cd src/ && ( test -e Makefile || /home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/toolchain/bin/qmake -o Makefile /home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/build/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/src/src.pro INSTALLDIR=/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/install_pkg/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/usr/bin INSTALL_BINDIR=/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/install_pkg/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/usr/bin INSTALL_DATADIR=/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/install_pkg/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/usr/bin INSTALL_ICONDIR=/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/install_pkg/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/usr/share/icons/trash INSTALL_DESKTOPDIR=/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/install_pkg/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/usr/share/applications ) && make -f Makefile 
make[1]: Entering directory '/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/build/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/.armv8a-libreelec-linux-gnueabihf/src'
cd backend/ && ( test -e Makefile || /home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/toolchain/bin/qmake -o Makefile /home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/build/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/src/backend/backend.pro INSTALLDIR=/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/install_pkg/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/usr/bin INSTALL_BINDIR=/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/install_pkg/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/usr/bin INSTALL_DATADIR=/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/install_pkg/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/usr/bin INSTALL_ICONDIR=/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/install_pkg/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/usr/share/icons/trash INSTALL_DESKTOPDIR=/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/install_pkg/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/usr/share/applications ) && make -f Makefile 
Project MESSAGE: Enabled third-party data sources:
Project MESSAGE:   - EmulationStation
Project MESSAGE:   - Logiqx
Project MESSAGE:   - Skraper Assets
Project MESSAGE: This project is using private headers and will therefore be tied to this specific Qt module build version.
Project MESSAGE: Running this project against other versions of the Qt modules may crash at any arbitrary point.
Project MESSAGE: This is not a bug, but a result of using Qt internals. You have been warned!
Project ERROR: Library 'libdl' is not defined.
make[1]: *** [Makefile:75: sub-backend-make_first] Error 3
make[1]: Leaving directory '/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/build/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/.armv8a-libreelec-linux-gnueabihf/src'
make: *** [Makefile:49: sub-src-make_first] Error 2
FAILURE: scripts/build pegasus-frontend during make_target (default)
*********** FAILED COMMAND ***********
make ${PKG_MAKE_OPTS_TARGET}
**************************************

[mmatyas]

Try adding QMAKE_LIBS_LIBDL=-ldl as a QMake parameter (like with the install locations), that fixed a similar issue on the Pi 4 for me.

[SupervisedThinking]

After adding QMAKE_LIBS_LIBDL=-ldl I end up with fatal error: X11/Xlib.h: No such file or directory which should normally fixable by:

export CFLAGS+=" -DMESA_EGL_NO_X11_HEADERS"
export CXXFLAGS+=" -DMESA_EGL_NO_X11_HEADERS"

but no clue how to add those flags as to QMake, I've read this https://stackoverflow.com/questions/17578150/add-cflags-to-qmake-project-without-hard-coding-them-in-the-pro-file but then again I probably have to patch some .pro files.

Log:

/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/toolchain/bin/qmlcachegen --resource=/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/build/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/src/frontend/frontend.qrc --resource=/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/build/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/src/qmlutils/qmlutils.qrc --resource=/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/build/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/src/themes/themes.qrc -o menu_settings_common_ToggleOption_qml.cpp /home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/build/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/src/frontend/menu/settings/common/ToggleOption.qml
In file included from /home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/toolchain/armv8a-libreelec-linux-gnueabihf/sysroot/usr/include/EGL/egl.h:39,
                 from /home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/toolchain/armv8a-libreelec-linux-gnueabihf/sysroot/usr/include/QtEglSupport/5.15.2/QtEglSupport/private/qt_egl_p.h:70,
                 from /home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/toolchain/armv8a-libreelec-linux-gnueabihf/sysroot/usr/include/QtEglFSDeviceIntegration/5.15.2/QtEglFSDeviceIntegration/private/qeglfsglobal_p.h:56,
                 from /home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/toolchain/armv8a-libreelec-linux-gnueabihf/sysroot/usr/include/QtEglFSDeviceIntegration/5.15.2/QtEglFSDeviceIntegration/private/qeglfsintegration_p.h:54,
                 from /home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/build/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/src/backend/ProcessLauncher.cpp:32:
/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/toolchain/armv8a-libreelec-linux-gnueabihf/sysroot/usr/include/EGL/eglplatform.h:134:10: fatal error: X11/Xlib.h: No such file or directory
  134 | #include <X11/Xlib.h>
      |          ^~~~~~~~~~~~
compilation terminated.
make[2]: *** [Makefile:1482: ProcessLauncher.o] Error 1
make[2]: Leaving directory '/home/supervisedthinking/development/LibreELEC-RR/build.LibreELEC-RK3399.arm-10.0-devel/build/pegasus-frontend-9abf065322d0cf89a4ef8e9bd92cfd25a06b8d9c/.armv8a-libreelec-linux-gnueabihf/src/backend'
make[1]: *** [Makefile:75: sub-backend-make_first] Error 2
make[1]: *** Waiting for unfinished jobs....

Have a look at moonlight-qt which compiles fine for example:

EGL config https://github.com/moonlight-stream/moonlight-qt/blob/master/app/app.pro#L261-L270 https://github.com/moonlight-stream/moonlight-qt/blob/master/app/streaming/video/ffmpeg-renderers/renderer.h#L12-L14

Could you add a somewhat similar logic? From what I've read I guess if HAVE_EGL is defined then the according configs were set automagically ^^

[mmatyas]

The include comes from your sysroot's EGL libraries: if you follow through the backtrace, Qt includes the EGL headers here: https://github.com/qt/qtbase/blob/40143c189b7c1bf3c2058b77d00ea5c4e3be8b28/src/platformsupport/eglconvenience/qt_egl_p.h#L70. egl.h includes eglplatform.h which has a following section, copied from the Debian armhf package:

#elif defined(__unix__) || defined(__APPLE__)

#if defined(MESA_EGL_NO_X11_HEADERS)

typedef void            *EGLNativeDisplayType;
typedef khronos_uintptr_t EGLNativePixmapType;
typedef khronos_uintptr_t EGLNativeWindowType;

#else

/* X11 (tentative)  */
#include <X11/Xlib.h>
#include <X11/Xutil.h>

typedef Display *EGLNativeDisplayType;
typedef Pixmap   EGLNativePixmapType;
typedef Window   EGLNativeWindowType;

#endif /* MESA_EGL_NO_X11_HEADERS */

The Qt file linked above should already define MESA_EGL_NO_X11_HEADERS for you if the Qt build was configured to target EGL, in which case QT_EGL_NO_X11 also gets defined by qmake automatically.

This error might indicate some configuration issues with the Qt build, but if you just want a quick workaround, you can add build flags to Pegasus with QMAKE_CXXFLAGS+=-DMESA_EGL_NO_X11_HEADERS as a qmake parameter.

[SupervisedThinking]

I'm aware of the root cause for this problem but since I define this: https://github.com/SupervisedThinking/LibreELEC-RR/blob/master-rr/packages/supervisedthinking/toolkit/qt-everywhere/package.mk#L151-L154 and moonlight-qt as Qt5 program with EGL runs fine.

Both platform plugins use DEFINES += QT_EGL_NO_X11 but I don't see why this should matter for pegasus if pegasus does not define it too because you can define at build time what you want to use. X11 and EGL could both be available afaik, my generic build supports pure KMS/GBM too while also running X11.

https://github.com/qt/qtbase/blob/5.15.2/src/plugins/platforms/eglfs/deviceintegration/eglfs_kms/eglfs_kms.pro#L11-L12 https://github.com/qt/qtbase/blob/5.15.2/src/plugins/platforms/eglfs/deviceintegration/eglfs_kms_egldevice/eglfs_kms_egldevice.pro#L7-L8

Btw. this is most likely not a RPi4 exclusive "bug" it's something you have to consider for all SBCs using KMS/GBM and relying on EGL without X11/wayland iirc. So e.g. Amlogic or Rockchip SoCs supported by lima or panfrost or RPi3/RPi4 etc.

[mmatyas]

If I recall, qmake should apply some EGL-related defines on its own, but I might be indeed incorrect about this/not apply in this case. If the Qt build otherwise works fine then I guess manually adding QT_EGL_NO_X11 would be a usable solution.

Btw. this is most likely not a RPi4 exclusive "bug"

Yes, it's a KMS related issue: because Pegasus launches other graphical programs, it must fully release the EGL handle on the display, otherwise the launched app won't receive input. This requires some private Qt calls, which is why this is a separate patch at the moment. This is still done through Qt however, Pegasus itself depends only on that, and doesn't use EGL or X11 on its own anywhere.

[SupervisedThinking]

@cgutman maybe you can give a hint how you handle EGL detection & QT_EGL_NO_X11 handling?

@mmatyas well I'm not an expert if it comes to qt - when I build my qt package it detects the avaiable eglfs stuff and sets defines, that's correct but afaik you have to set MESA_EGL_NO_X11_HEADERS manually or your qt project probably has to "check" if EGL was defined or something. Well don't worry... it's not the only "troublesome" project... Kodi(xbmc) needs those flags, mupen64plus-nx needs them... well a lot of stuff needs it 😄

In the end it would be cool if you could work in your patch to set these extra defines for builds using EGL automatically or at least a build opt which needs to be passed would also be fine because if you build the same package for different platform it's getting tricky.

Just in case someone else runs into similar trouble, fixed by: https://github.com/SupervisedThinking/LibreELEC-RR/commit/4477fa81c8f5e696affc5c5ffd3baeca6a20bd6a#diff-649825c78e7ba9c44db151c0e5319a0c150e356aef394835a582c8d3918674f6 https://github.com/SupervisedThinking/LibreELEC-RR/commit/e084f855bcdf05ba247a2957c2492ac739bc01db#diff-649825c78e7ba9c44db151c0e5319a0c150e356aef394835a582c8d3918674f6

I've made some tests with a ROCKPro64 board with latest LibreELEC 10.0 master, mainline kernel 5.10.17 & mesa 21.1-dev and Retroarch/DOSBox etc. runs fine & returns clean to pegasus once you exit them,

[cgutman]

I think my situation with Moonlight is simpler because both DRM users (Qt and SDL) are in my process, and I can ensure Qt will not try to render while SDL is master (and vice versa). I didn't want to link to any private symbols in Qt for fear of possible breakage later. A few simple hooks do the job for me.

The same solution would work with multiple processes and LD_PRELOAD, but you'd need to use some IPC mechanism to tell each process when it should drop and set master.

EDIT: Actually if you just want to get out of the way of some future DRM user, you could probably just use the same technique to get Qt's DRM FD and then just call drmDropMaster() on it when you're about to launch the other program, and drmSetMaster() when that program exits (similar to what Moonlight does when we see a close() on SDL's DRM FD).

[SupervisedThinking]

@cgutman thx for your explanantion, even though I probably only understood half of the details 🙈, from my last testruns I can say that the "RPi4 patch" works fine for RK3399/A311D SBC so my primary "intentention/question" was how you detect "HAVE_EGL" at build time. Do you check the Qt5 config to determine if it's set or not? And if so how do you do that. Because @mmatyas might could just copy your approach so that no extra build flags or patch magic is needed 👍🏻