Investigate usage of LibreSSL in AppImage

[mariokamenjak]

Hi, when opening the beta AppImage I get this error message: SSL support is not available or incomplete. Some websites may work incorrectly or do not work at all.

And they really don't work. OS is Debian Testing.

[mariokamenjak]

I also got this error on the error of the web page: error:140A90C4:SSL routines:func(169):reason(196) Perhaps this can help: https://confluence.atlassian.com/hipchatkb/hipchat4-debian-client-doesn-t-connect-859526103.html

[Emdek]

@mariokamenjak, do you have OpenSSL installed? I could bundle it, but I'm not sure if this is a good idea.

[mariokamenjak]

openssl is already the newest version (1.1.0c-2).

[Emdek]

@mariokamenjak, apparently Qt looks for 1.0 only. And it seems that simply bundling OpenSSL libraries will require adding license exception.

[Emdek]

I'll try to bundle LibreSSL instead.

[Emdek]

I've tried using it as drop-in replacement but runtime check used by Qt prevents it from being used, I'll try another approach.

[Frenzie]

Ok, so in light of the existing Gentoo qtnetwork patches it's easiest to use Qt 5.6.2 with LibreSSL 2.4.5 or Qt 5.7.x with LibreSSL 2.5.1.

Here's what I did for LibreSSL on Debian Stretch:

wget https://ftp.openbsd.org/pub/OpenBSD/LibreSSL/libressl-2.4.5.tar.gz
tar -zxvf libressl-2.4.5.tar.gz
cd libressl-2.4.5
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX:PATH="${HOME}/opt/libressl" ..
make
make install

For Qt (I'm sure you know, but I made notes so I might as well include it, cf. Qt docs)

git clone https://code.qt.io/qt/qt5.git
cd qt5
git checkout v5.6.2
git submodule update --init
#or just git submodule update if you've already done that before

Apply patches

cd qtbase
wget https://github.com/gentoo/libressl/raw/master/dev-qt/qtnetwork/files/0001-Fix-compilation-with-libressl.patch
wget https://github.com/gentoo/libressl/raw/master/dev-qt/qtnetwork/files/qtnetwork-5.6.2-libressl-warn.patch

patch < 0001-Fix-compilation-with-libressl.patch src/network/ssl/qsslcontext_openssl.cpp
patch < qtnetwork-5.6.2-libressl-warn.patch config.tests/unix/openssl/openssl.cpp

#build-deps for qt, may not include everything for Qt 5.6 on Jessie
sudo apt-get build-dep qt5-default

#to include the libs properly during build, of course you shouldn't need -I if you installed it to the system
OPENSSL_LIBS="-L${HOME}/opt/libressl/lib -lssl -lcrypto -ltls" ./configure -openssl-linked -I"${HOME}/opt/libressl/include"

sudo make install

Then finally to build something with it

# where relevant
/usr/local/Qt-5.6.2/bin/qmake
LD_LIBRARY_PATH="${HOME}/opt/libressl/lib" make
LD_LIBRARY_PATH="${HOME}/opt/libressl/lib" ./libressl-test

PS The libressl-test source:

libressl-test.pro

QT += core network
QT -= gui

CONFIG += c++11

TARGET = libressl-test
CONFIG += console
CONFIG -= app_bundle

TEMPLATE = app

SOURCES += main.cpp

main.cpp

#include <QCoreApplication>
#include <QSslSocket>
#include <QNetworkAccessManager>
#include <QNetworkRequest>

int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);

    qDebug() << QSslSocket::sslLibraryVersionString();
    qDebug() << QSslSocket::sslLibraryBuildVersionString();

    QNetworkAccessManager manager;
    manager.get(QNetworkRequest(QUrl("https://otter-browser.org")));

    return a.exec();
}

[Emdek]

@Frenzie, thanks for testing, I'll try to build entire framework with QtWebKit when I'll have some free time. It seems that Debian Jessie would be the best choice for new host, since Wheezy has outdated toolchain.

[Emdek]

AppImage for weekly 191 uses algorithm that tries to match and load compatible OpenSSL version, hopefully that will be enough to solve this for now. Hopefully OpenSSL domination will end sooner than later. ;-)

[beastie1]

Long live LibreSSL!

Hopefully OpenSSL domination will end sooner than later. ;-)

Agreed.

[Emdek]

It turns out that OpenSSL is messed up even more than I thought, they broke ABI also in one of patch releases, which means that we need different approach. :-/

[Emdek]

Another idea, do not bundle Qt at all (except of QtWebkit), build against Qt 5.4 (minimum required version) and warn users if dependencies are missing on startup. As a bonus it will make package smaller.

[Frenzie]

QtWebKit doesn't directly depend on OpenSSL some SSL library?

[Emdek]

@Frenzie, nope, just QtNetwork. I was experimenting with variants of that library linking OpenSSL 1.0.1 and another for 1.0.2g (they broke ABI even in patch release!) but no luck so far, I'm getting errors about missing symbols...

[Frenzie]

Huh, I guess bundling only QtWebKit would be a really interesting idea, though on the surface more so for a Debuntu package than for an AppImage.

[crass]

It looks like this is still an issue in the latest AppImage. I've run into this problem on Ubuntu 20.04 and the fix in #1558 won't work because there is no openssl1.0 package. There are 1.0 libs that can be installed via the package libssl1.0, but I already had those installed and still ran in to this issue.

I was able to work around this because I run the AppImage under firejail. The libssl1.0 package was already installed so I had the needed libraries. With firejail I blacklisted the libssl version 1.1 system libraries, so they were skipped and the 1.0 system libraries were used. This allowed me to visit SSL sites with the otter AppImage.

I wanted to see if I could figure out more of a root cause, so I looked further. After some debugging, I found that the AppImage (otter-browser-0.9.99.3-rc12-x86_64.AppImage) actually already has the ssl libraries bundled with it. And it appears to be LibreSSL 2.5.1. So changing to LibreSSL has not fixed the issue (or created another issue with the same symptom).

I'm running the AppImage with firejail, which is why some of the path components might look funny. Initially, I was inspecting the execution with strace, but then I discovered setting the environment variable QT_DEBUG_PLUGINS=1 gives great debugging info as to why loading of particular library paths fail. Here is the relevant output (I deny access to the system ssl libs via firejail):

Cannot load library crypto: (crypto: cannot open shared object file: No such file or directory)
loaded library "crypto"
Cannot load library ssl: (/usr/lib/ssl: cannot read file data: Is a directory)
QLibraryPrivate::unload succeeded on "crypto"
loaded library "/run/firejail/appimage/.appimage-1694041/lib/libcrypto.so.1.0.0"
Cannot load library /run/firejail/appimage/.appimage-1694041/lib/libssl.so.1.0.0: (/run/firejail/appimage/.appimage-1694041/lib/libssl.so.1.0.0: undefined symbol: SHA512_Transform)
Cannot load library /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0: (/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0: cannot open shared object file: Permission denied)
QLibraryPrivate::unload succeeded on "/run/firejail/appimage/.appimage-1694041/lib/libcrypto.so.1.0.0"
loaded library "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1"
Cannot load library /usr/lib/x86_64-linux-gnu/libssl.so.1.1: (/usr/lib/x86_64-linux-gnu/libssl.so.1.1: cannot open shared object file: Permission denied)
QLibraryPrivate::unload succeeded on "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1"
loaded library "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0"
Cannot load library /run/firejail/appimage/.appimage-1694041/lib/libssl.so.1.0.0: (/run/firejail/appimage/.appimage-1694041/lib/libssl.so.1.0.0: undefined symbol: SHA512_Transform)
Cannot load library /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0: (/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0: cannot open shared object file: Permission denied)
QLibraryPrivate::unload succeeded on "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0"
loaded library "/usr/lib/x86_64-linux-gnu/libcrypto.so"
Cannot load library /usr/lib/x86_64-linux-gnu/libssl.so: (/usr/lib/x86_64-linux-gnu/libssl.so: cannot open shared object file: Permission denied)
QLibraryPrivate::unload succeeded on "/usr/lib/x86_64-linux-gnu/libcrypto.so"
Cannot load library /usr/lib/x86_64-linux-gnu/libcrypto.a: (/usr/lib/x86_64-linux-gnu/libcrypto.a: invalid ELF header)

You can see that the bundled libssl is being found, but is failing to load with the error undefined symbol: SHA512_Transform. Running nm -D on the library indeed shows the undefined symbol. What at first seems odd is that the symbol is defined in the bundled libcrypto which appears to load successfully just before. However, running ldd on the libssl gives the following output:

ldd lib/libssl.so.1.0.0
        linux-vdso.so.1 (0x00007ffc3bcb4000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb32e6da000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fb32eb4d000)

There's no libcrypto.so.1.0.0 showing up as a dependency. Even though libcrypto is loaded, the linker must not know to resolve undefined symbols in libssl against it. So first problem is that the libssl library needs to be built with a shared library dependency on libcrypto.

This issue can be worked around by use LD_PRELOAD to force load the libcrypto library at process start. That fails again, but with a different issue:

Cannot load library crypto: (crypto: cannot open shared object file: No such file or directory)
loaded library "crypto"
Cannot load library ssl: (/usr/lib/ssl: cannot read file data: Is a directory)
QLibraryPrivate::unload succeeded on "crypto" 
loaded library "/run/firejail/appimage/.appimage-1694041/lib/libcrypto.so.1.0.0"
loaded library "/run/firejail/appimage/.appimage-1694041/lib/libssl.so.1.0.0"
qt.network.ssl: QSslSocket: cannot resolve SSLv2_client_method
qt.network.ssl: QSslSocket: cannot resolve SSLv3_client_method
qt.network.ssl: QSslSocket: cannot resolve SSLv2_server_method
qt.network.ssl: QSslSocket: cannot resolve SSLv3_server_method
qt.network.ssl: Incompatible version of OpenSSL

Now both of the bundled libssl and libcrypto are loading without error, but four symbols can not be resolved. Indeed those symbols are not to be found in either library. Here's some similar methods:

bash-5.0$ nm -D lib/libssl.so.1.0.0 |grep _client_method
00000000000165c4 T DTLSv1_client_method
0000000000043d52 T SSLv23_client_method
0000000000043d5d T TLS_client_method
0000000000043d77 T TLSv1_1_client_method
0000000000043d84 T TLSv1_2_client_method
0000000000043d6a T TLSv1_client_method
bash-5.0$ nm -D lib/libcrypto.so.1.0.0 |grep _client_method

It looks like LibreSSL combines SSLv2_client_method and SSLv3_client_method into one method named SSLv23_client_method, and the same for the server variant. So this suggests that we can't use LibreSSL after all, unless you want to build a patched version which provides the missing method named (pretty trivial) or rebuild libQt5Network.so to not use those symbols (looks like you want after this commit).

I hope this provides a path forward in resolving this long standing issue.

[baryluk]

Hi. I usually compile otter from source on Debian, but I had a computer with Fedora 34, and tried appimage (which I like very much).

But it does not include openssl (hard to say if it is good or bad decision, there are pros and cons of including / not including it - if the otter's appimage is updated frequently, then it is fine to bundle it, otherwise, it is not a great idea - people can run for long time with possibly vulnerable ssl version).

And it has issues:

~/Downloads$ ./otter-browser-0.9.99.3-rc12-x86_64.AppImage 
qt.network.ssl: QSslSocket: cannot resolve CRYPTO_num_locks
qt.network.ssl: QSslSocket: cannot resolve CRYPTO_set_id_callback
qt.network.ssl: QSslSocket: cannot resolve CRYPTO_set_locking_callback
qt.network.ssl: QSslSocket: cannot resolve ERR_free_strings
qt.network.ssl: QSslSocket: cannot resolve EVP_CIPHER_CTX_cleanup
qt.network.ssl: QSslSocket: cannot resolve EVP_CIPHER_CTX_init
qt.network.ssl: QSslSocket: cannot resolve sk_new_null
qt.network.ssl: QSslSocket: cannot resolve sk_push
qt.network.ssl: QSslSocket: cannot resolve sk_free
qt.network.ssl: QSslSocket: cannot resolve sk_num
qt.network.ssl: QSslSocket: cannot resolve sk_pop_free
qt.network.ssl: QSslSocket: cannot resolve sk_value
qt.network.ssl: QSslSocket: cannot resolve SSL_library_init
qt.network.ssl: QSslSocket: cannot resolve SSL_load_error_strings
qt.network.ssl: QSslSocket: cannot resolve SSL_get_ex_new_index
qt.network.ssl: QSslSocket: cannot resolve SSLv2_client_method
qt.network.ssl: QSslSocket: cannot resolve SSLv23_client_method
qt.network.ssl: QSslSocket: cannot resolve SSLv2_server_method
qt.network.ssl: QSslSocket: cannot resolve SSLv23_server_method
qt.network.ssl: QSslSocket: cannot resolve X509_STORE_CTX_get_chain
qt.network.ssl: QSslSocket: cannot resolve OPENSSL_add_all_algorithms_noconf
qt.network.ssl: QSslSocket: cannot resolve OPENSSL_add_all_algorithms_conf
qt.network.ssl: QSslSocket: cannot resolve SSLeay
qt.network.ssl: Incompatible version of OpenSSL
Empty filename passed to function
...

openssl 1.1.1l-2 is installed on the Fedora 34.