How to ignore a specific driver for QtSql?

[spiderkeys]

When I try to package my app (which does depend on QtSql), ldd will be run on each of the libraries within the ${QT_PATH}/plugins/sqldrivers/ directory, as you might expect to happen. However, now in Qt 6.6.1 (and not in 6.5.2), there is a new SQL driver called libqsqlmimer.so for the Mimer SQL library (https://www.mimer.com/mimer-sql-joins-qt-6-6-enhancing-database-connectivity/). ldd fails to find libmimerapi.so, and the appimage build fails as a result.

From what I can tell, the two obvious paths to resolve this are:

1. Install Mimer SQL
2. Remove the Mimer driver from my Qt install

Well, I have no need for Mimer (and you can't even trivially install it on Ubuntu anyway, because it is a paid product - likely Qt automotive related shenanigans), so I'm currently working around the issue by just outright deleting the driver.

That said, it would be nice if there was a way to tell linuxdeploy/plugin-qt to ignore specific drivers which aren't needed. In this case, I only use the sqlite driver in my app, so there is no need to deploy the others either.

I tried to find such a method, but didn't come across anything.

[TheAssassin]

There is no such feature as of yet. I have not investigated this driver. Is it really part of a standard Qt distribution? Where do you get your Qt from?

In any case, I'm not sure your approach is appropriate. We may want to exclude that driver by default if needed, i.e., make it opt-in.

[spiderkeys]

I install Qt in my CI environment using aqtinstall (https://github.com/miurahr/aqtinstall)

aqt install-qt linux desktop 6.6.1 gcc_64 -O "/opt/qt" -m \
    qt3d qtcharts qtimageformats qtlocation \
    qtmultimedia qtpositioning qtquick3dphysics \
    qtserialport qtspeech qtwebchannel qtwebengine \
    qtwebsockets qtwebview qt5compat qtquick3d qtshadertools

[spiderkeys]

It does show Mimer as one of the supported drivers here: https://doc.qt.io/qt-6/sql-driver.html

I guess it ultimately depends on if you configure the Qt build to build that driver or not, and the default appears to be all.

[spiderkeys]

The exact binary distribution that is pulled by aqt is hosted on an official Qt mirror site (Constant, in my case), as per: https://download.qt.io/static/mirrorlist/

qtbase is the module that includes the Sql plugin, and this is the file I get from my install process: https://qt.mirror.constant.com/online/qtsdkrepository/linux_x64/desktop/qt6_661/qt.qt6.661.gcc_64/6.6.1-0-202311210527qtbase-Linux-RHEL_8_8-GCC-Linux-RHEL_8_8-X86_64.7z

Looking in here, I can confirm that the mimer driver is included.

[TheAssassin]

Could you post a (sanitized) log of a failed attempt so I can get an idea of the problem, please?

[spiderkeys]

Sure, given Qt installed via aqt via:

aqt install-qt linux desktop 6.6.1 gcc_64 -O "/opt/mr/qt" -m \
    qt3d qtcharts qtimageformats qtlocation \
    qtmultimedia qtpositioning qtquick3dphysics \
    qtserialport qtspeech qtwebchannel qtwebengine \
    qtwebsockets qtwebview qt5compat qtquick3d qtshadertools

And given a linuxdeploy invocation of:

/opt/mr/linuxdeploy/linuxdeploy-x86_64.AppImage \
  --appdir "${BUILD_DIR}/AppDir" \
  --plugin qt \
  --output appimage \
  --executable "${BUILD_DIR}/AppDir/usr/bin/myapp" \
  --exclude-library="libqsqlmimer.so" \ # Tried this for fun... didn't work.
  --custom-apprun="${SRC_DIR}/app/myapp/appimage/AppRun"

this is the relevant log output/error message:

linuxdeploy version 1-alpha (git commit ID 6a583d5), GitHub actions build 195 built on 2023-10-26 10:45:18 UTC

-- Creating basic AppDir structure -- 
Creating directory /tmp/AppImageLauncher-build-k2Ualt/AppDir/usr/bin/ 
Creating directory /tmp/AppImageLauncher-build-k2Ualt/AppDir/usr/lib/ 
Creating directory /tmp/AppImageLauncher-build-k2Ualt/AppDir/usr/share/applications/ 
Creating directory /tmp/AppImageLauncher-build-k2Ualt/AppDir/usr/share/icons/hicolor/ 
Creating directory /tmp/AppImageLauncher-build-k2Ualt/AppDir/usr/share/icons/hicolor/16x16/apps/ 
Creating directory /tmp/AppImageLauncher-build-k2Ualt/AppDir/usr/share/icons/hicolor/32x32/apps/ 
Creating directory /tmp/AppImageLauncher-build-k2Ualt/AppDir/usr/share/icons/hicolor/64x64/apps/ 
Creating directory /tmp/AppImageLauncher-build-k2Ualt/AppDir/usr/share/icons/hicolor/128x128/apps/ 
Creating directory /tmp/AppImageLauncher-build-k2Ualt/AppDir/usr/share/icons/hicolor/256x256/apps/ 
Creating directory /tmp/AppImageLauncher-build-k2Ualt/AppDir/usr/share/icons/hicolor/scalable/apps/ 

-- Deploying dependencies for existing files in AppDir -- 
Deploying dependencies for ELF file /tmp/AppImageLauncher-build-k2Ualt/AppDir/usr/bin/myapp 
Deploying shared library /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0
Deploying copyright files for file /lib/x86_64-linux-gnu/libgstreamer-1.0.so.0 
Deploying shared library /lib/x86_64-linux-gnu/libgobject-2.0.so.0
Deploying copyright files for file /lib/x86_64-linux-gnu/libgobject-2.0.so.0 
Deploying shared library /lib/x86_64-linux-gnu/libglib-2.0.so.0
Deploying copyright files for file /lib/x86_64-linux-gnu/libglib-2.0.so.0 
Deploying shared library /lib/x86_64-linux-gnu/libgstapp-1.0.so.0
Deploying copyright files for file /lib/x86_64-linux-gnu/libgstapp-1.0.so.0 
Deploying shared library /opt/mr/qt/6.6.1/gcc_64/lib/libQt6Sql.so.6
WARNING: Could not find copyright files for file /opt/mr/qt/6.6.1/gcc_64/lib/libQt6Sql.so.6 using dpkg-query 
Deploying shared library /opt/mr/qt/6.6.1/gcc_64/lib/libQt6QuickControls2.so.6
WARNING: Could not find copyright files for file /opt/mr/qt/6.6.1/gcc_64/lib/libQt6QuickControls2.so.6 using dpkg-query 
Deploying shared library /opt/mr/qt/6.6.1/gcc_64/lib/libQt6Multimedia.so.6
WARNING: Could not find copyright files for file /opt/mr/qt/6.6.1/gcc_64/lib/libQt6Multimedia.so.6 using dpkg-query 
Deploying shared library /opt/mr/qt/6.6.1/gcc_64/lib/libQt6Qml.so.6
WARNING: Could not find copyright files for file /opt/mr/qt/6.6.1/gcc_64/lib/libQt6Qml.so.6 using dpkg-query 
Deploying shared library /opt/mr/qt/6.6.1/gcc_64/lib/libQt6Network.so.6
WARNING: Could not find copyright files for file /opt/mr/qt/6.6.1/gcc_64/lib/libQt6Network.so.6 using dpkg-query 
Deploying shared library /opt/mr/qt/6.6.1/gcc_64/lib/libQt6Widgets.so.6
WARNING: Could not find copyright files for file /opt/mr/qt/6.6.1/gcc_64/lib/libQt6Widgets.so.6 using dpkg-query 
Deploying shared library /opt/mr/qt/6.6.1/gcc_64/lib/libQt6Gui.so.6
WARNING: Could not find copyright files for file /opt/mr/qt/6.6.1/gcc_64/lib/libQt6Gui.so.6 using dpkg-query 
Deploying shared library /opt/mr/qt/6.6.1/gcc_64/lib/libQt6Core.so.6
...
...
...
[qt/stdout] -- Deploying module: quick -- 
[qt/stdout] 
[qt/stdout] -- Deploying module: quicktemplates2 -- 
[qt/stdout] 
[qt/stdout] -- Deploying module: sql -- 
[qt/stdout] Deploying SQL plugins 
[qt/stdout] Deploying shared library /opt/mr/qt/6.6.1/gcc_64/plugins/sqldrivers/libqsqlpsql.so (destination: /tmp/AppImageLauncher-build-sLqhyX/AppDir/usr/plugins/sqldrivers/)
[qt/stdout] WARNING: Could not find copyright files for file /opt/mr/qt/6.6.1/gcc_64/plugins/sqldrivers/libqsqlpsql.so using dpkg-query 
[qt/stdout] Deploying dependencies for ELF file /opt/mr/qt/6.6.1/gcc_64/plugins/sqldrivers/libqsqlpsql.so 
[qt/stdout] Deploying shared library /lib/x86_64-linux-gnu/libpq.so.5
[qt/stdout] Deploying copyright files for file /lib/x86_64-linux-gnu/libpq.so.5 
[qt/stdout] Deploying shared library /lib/x86_64-linux-gnu/libssl.so.3
[qt/stdout] Deploying copyright files for file /lib/x86_64-linux-gnu/libssl.so.3 
[qt/stdout] Deploying shared library /lib/x86_64-linux-gnu/libcrypto.so.3
[qt/stdout] Deploying copyright files for file /lib/x86_64-linux-gnu/libcrypto.so.3 
[qt/stdout] Deploying shared library /lib/x86_64-linux-gnu/libldap-2.5.so.0
[qt/stdout] Deploying copyright files for file /lib/x86_64-linux-gnu/libldap-2.5.so.0 
[qt/stdout] Deploying shared library /lib/x86_64-linux-gnu/liblber-2.5.so.0
[qt/stdout] Deploying copyright files for file /lib/x86_64-linux-gnu/liblber-2.5.so.0 
[qt/stdout] Deploying shared library /lib/x86_64-linux-gnu/libsasl2.so.2
[qt/stdout] Deploying copyright files for file /lib/x86_64-linux-gnu/libsasl2.so.2 
[qt/stdout] Deploying shared library /lib/x86_64-linux-gnu/libgnutls.so.30
[qt/stdout] Deploying copyright files for file /lib/x86_64-linux-gnu/libgnutls.so.30 
[qt/stdout] Skipping deployment of blacklisted library /lib/x86_64-linux-gnu/libp11-kit.so.0 
[qt/stdout] Deploying shared library /lib/x86_64-linux-gnu/libidn2.so.0
[qt/stdout] Deploying copyright files for file /lib/x86_64-linux-gnu/libidn2.so.0 
[qt/stdout] Deploying shared library /lib/x86_64-linux-gnu/libunistring.so.2
[qt/stdout] Deploying copyright files for file /lib/x86_64-linux-gnu/libunistring.so.2 
[qt/stdout] Deploying shared library /lib/x86_64-linux-gnu/libtasn1.so.6
[qt/stdout] Deploying copyright files for file /lib/x86_64-linux-gnu/libtasn1.so.6 
[qt/stdout] Deploying shared library /lib/x86_64-linux-gnu/libnettle.so.8
[qt/stdout] Deploying copyright files for file /lib/x86_64-linux-gnu/libnettle.so.8 
[qt/stdout] Deploying shared library /lib/x86_64-linux-gnu/libhogweed.so.6
[qt/stdout] Deploying copyright files for file /lib/x86_64-linux-gnu/libhogweed.so.6 
[qt/stdout] Skipping deployment of blacklisted library /lib/x86_64-linux-gnu/libgmp.so.10 
[qt/stdout] Deploying shared library /lib/x86_64-linux-gnu/libffi.so.8
[qt/stdout] Deploying copyright files for file /lib/x86_64-linux-gnu/libffi.so.8 
[qt/stdout] Deploying shared library /opt/mr/qt/6.6.1/gcc_64/plugins/sqldrivers/libqsqlmimer.so (destination: /tmp/AppImageLauncher-build-sLqhyX/AppDir/usr/plugins/sqldrivers/)
[qt/stdout] WARNING: Could not find copyright files for file /opt/mr/qt/6.6.1/gcc_64/plugins/sqldrivers/libqsqlmimer.so using dpkg-query 
[qt/stdout] Deploying dependencies for ELF file /opt/mr/qt/6.6.1/gcc_64/plugins/sqldrivers/libqsqlmimer.so 
[qt/stdout] ERROR: Could not find dependency: libmimerapi.so 
ERROR: Failed to run plugin: qt (exit code: 1) 

[TheAssassin]

To me, this smells like a bug in Qt. How can they ship an incomplete distribution?

I'm not decided on a strategy yet. I'll check their bug tracker.

[spiderkeys]

Well, I think that is a weird thing about Qt in general (just seems to be more egregious in their Sql plugin). There are definitely a bunch of assumptions made around what files exist in a user's environment:

ldd /opt/mr/qt/6.6.1/gcc_64/plugins/sqldrivers/libqsqlmysql.so 
        linux-vdso.so.1 (0x00007ffc3a737000)
        libmysqlclient.so.21 => /lib/x86_64-linux-gnu/libmysqlclient.so.21 (0x00007f9beec03000)
        ...

Like there's really no reason it should be expected that I have libmysqlclient available on my machine either, it just happens to be the case that it is in this case. I think at least with sqlite they compile the sqlite3.c file as part of the plugin.

I've never been able to pinpoint any documentation that specifies exactly what you need within a given Linux distribution Qt to be fully functional, I just keep apt-installing (or conan-installing) packages until everything works...

Interested to learn what they respond with on their tracker!

[TheAssassin]

In a quick search I could not find related bug reports. Feel free to post some links.

I guess the main issue is that most of the time, users of this software use distribution provided (or otherwise well packaged) Qt distributions rather than the upstream Qt. Those usually ensure the installation of all dependencies.

[bjorn]

Like there's really no reason it should be expected that I have libmysqlclient available on my machine either, it just happens to be the case that it is in this case.

If it wasn't the case, sqldrivers/libqsqlmysql.so would just fail to load and the Qt application would not be able to talk to MySQL through QtSql. But this is only a problem for those Qt applications that need to connect to MySQL. Similarly, Qt can ship libqsqlmimer.so but there is no need to also ship a libmimerapi.so. Only those who distribute applications depending on Mimer SQL will need to ship this library. For other applications, the Mimer SQL plugin can fail to load without causing any issues.

So this isn't a bug in Qt. Instead, linuxdeploy-plugin-qt should provide a way to exclude plugins (since it can be useful to not ship unused plugins in general) and possibly, a missing dependency of a plugin should not be a fatal error like this:

[qt/stdout] Deploying dependencies for ELF file /opt/mr/qt/6.6.1/gcc_64/plugins/sqldrivers/libqsqlmimer.so 
[qt/stdout] ERROR: Could not find dependency: libmimerapi.so 

Maybe it could just be a warning instead.

[7ymekk]

the issue is not only with mimersql. We use SQLite only in our project, but all SQL plugins are being deployed:

113K libqsqlmimer.so 113K libqsqlmysql.so 141K libqsqlodbc.so 117K libqsqlpsql.so

332K .AppDir/usr/lib/libpq.so.5 6,6M .AppDir/usr/lib/libmysqlclient.so.21 433K .AppDir/usr/lib/libodbc.so.2 444K .AppDir/usr/lib/libmimerapi.so

not mentioning dependencies of libpq and others. it makes our AppImage >10MB too big. Is there any option to fix that other than removing files in Qt installation?

[TheAssassin]

You can remove the --output option and run linuxdeploy with the Qt plugin once, then clean up the files in your AppDir, then run linuxdeploy without the Qt plugin but with --output as a workaround.

[7ymekk]

@TheAssassin cheers for the tip, it works

[dantti]

@TheAssassin just FYI this happens with the upstream Qt packages, aqtinstall uses that, sqlmimer and mysqlclient libraries are not bundled into the distributed binaries as the user is expected to have them if they want to use it (plus obvious legal issues and increased distribution size), this is not a problem outside linuxdeploy Qt plugin because only this tool will do an "ldd" scan to see if plugins have deps met, so yes, --exclude-plugin would be best as this happens with most plugins Qt ship.

[dantti]

@TheAssassin so I started adding EXCLUDE_QT_PLUGINS and just realized that App dir class already has a way to exclude things, but as @spiderkeys also mentioned it does not work, this cmd line option is not available on the linuxdeploy-plugin-qt stand alone version, so the actual bug here is that --exclude-library libqsqlmimer.so isn't actually passed to the qt plugin.

Adding this option to the linuxdeploy-plugin-qt stand alone command fixed the issue, which of course would be nice, maybe there should be a way to be able to add all linuxdeploy args, usually that's done with --, so ./linuxdeploy-plugin-qt ... -- --main-linuxdeploy-flags.

But adding the option on the plugin by my understanding of the code, will not receive the needed args by default, it seems to me that each AppDir option should be stored into LINUXDEPLOY_ env vars that get's automatically read by AppDir plugins instances.

Do you think that's a good approach? I don't want to spend time coding in the wrong direction...

[nuttyartist]

I remember I had the same issue, but I fixed it somehow. Now I got the same issue again (and even previous commits have this issue). @spiderkeys, how did you solve this in your CI?

[spiderkeys]

@nuttyartist we build and deploy Qt using conan now, so this is no longer a problem for us.

EDIT: Disregard the below as it wouldn't solve the problem - I was misremembering how we solved a different issue.

That said, our original solution was to:
1. Build the appimage once using linuxdeploy
2. Extract that appimage 
3. Delete the undesired drivers/files from that directory
4. Resquash the directory with AppImageTool

I think the only way we got around this issue in the past was deleting the offending shared libs before running linuxdeploy

[dantti]

@nuttyartist you can also use my linuxdeploy appimages fork where the issue is fixed, hopefully @TheAssassin will merge my fixes soon... https://github.com/dantti/linuxdeploy/releases/tag/continuous https://github.com/dantti/linuxdeploy-plugin-qt/releases/tag/continuous

[jrozner]

Is there any timeline on addressing this with an official solution?