search

status-im / status-desktop

Status Desktop client made in Nim & QML
https://status.app
Mozilla Public License 2.0
292 stars 78 forks source link

Setup Linux CI for Qt5.15.8+KDAB using minimum glibc version compatibility #9350

Closed alexjba closed 3 months ago

alexjba commented 1 year ago

Description

Prebuilt binaries of Qt5.15.8 are compiled using glibc 2.28 and migrating to this version will breaking compatibility with older linux versions.

The purpose of this task is to setup the Linux CI to use latest Qt5.15.8 + KDAB fixes and keep the glibc dependency to a lower version (Ideally 2.23 or lower). Also, we will have this issue when moving to Qt6 and solving this will also help long term with newer qt versions.

There are multiple options that could be considered:

  1. Finding an out of the box setup - precompiled Qt binaries compatible with older linux versions
  2. Compiling Qt5.15.8 with KDAB patches + Status and link with glibc 2.23 (probably the easiest way to do it is on a linux distribution that uses this version)
  3. Try this hack https://github.com/wheybags/glibc_version_header and enable Nix environment on CI
  4. https://github.com/phusion/holy-build-box

On the CI side there are two approaches that were considered when migrating to Qt5.15: Docker and Nix. Nix is preferred, but it has the same limitation - will break compatibility with older linux dostros. We're using Docker for now.

See: #9222, #9232

anastasiyaig commented 1 year ago

@alexjba @jakubgs we need that for all platforms (windows and mac inclusive). windows now is 5.15.2 and arm mac is 5.15.8 when intel is 5.15.5

jakubgs commented 1 year ago

Actually, we could combine using Nix together with use of https://github.com/wheybags/glibc_version_header. And that might be the best option for us, since it would provide locked Qt for both Linux and MacOS, and possibly even Windows if we managed to use WSL for CI. Though that might require extra work.

yakimant commented 1 year ago

Hi guys, I started to research on this issue.

Here is a first finding - there is no binary for Qt 5.15.3+ available.

❯ aqt list-qt linux desktop
5.9.0 5.9.1 5.9.2 5.9.3 5.9.4 5.9.5 5.9.6 5.9.7 5.9.8 5.9.9
5.10.0 5.10.1
5.11.0 5.11.1 5.11.2 5.11.3
5.12.0 5.12.1 5.12.2 5.12.3 5.12.4 5.12.5 5.12.6 5.12.7 5.12.8 5.12.9 5.12.10 5.12.11 5.12.12
5.13.0 5.13.1 5.13.2
5.14.0 5.14.1 5.14.2
5.15.0 5.15.1 5.15.2
6.0.0 6.0.1 6.0.2 6.0.3 6.0.4
6.1.0 6.1.1 6.1.2 6.1.3
6.2.0 6.2.1 6.2.2 6.2.3 6.2.4
6.3.0 6.3.1 6.3.2
6.4.0 6.4.1 6.4.2 6.4.3
6.5.0 6.5.1 6.5.2 6.5.3
6.6.0

which taked binaries from https://download.qt.io/online/qtsdkrepository/linux_x64/desktop/

See https://github.com/miurahr/aqtinstall/pull/572

Do I miss smth here and there are other "Prebuilt binaries of Qt5.15.8" for Linux?

yakimant commented 1 year ago

Homebrew maintained binaries: https://github.com/Homebrew/homebrew-core/pkgs/container/core%2Fqt%2F5/versions?filters%5Bversion_type%5D=tagged

For example 5.15.8_3-1: https://github.com/Homebrew/homebrew-core/pkgs/container/core%2Fqt%2F5/89485747?tag=5.15.8_3-1 (includes Linux one)

yakimant commented 1 year ago

Ubuntu packages:

https://packages.ubuntu.com/search?keywords=qtbase5-dev&searchon=names&suite=all&section=all

yakimant commented 1 year ago

Compilation doc: https://doc.qt.io/qt-5/linux.html#building-qt-5-from-source

caybro commented 1 year ago

23.04 apparently has 5.15.8

alexjba commented 1 year ago

Homebrew maintained binaries: https://github.com/Homebrew/homebrew-core/pkgs/container/core%2Fqt%2F5/versions?filters%5Bversion_type%5D=tagged

For example 5.15.8_3-1: https://github.com/Homebrew/homebrew-core/pkgs/container/core%2Fqt%2F5/89485747?tag=5.15.8_3-1 (includes Linux one)

Yes, at that time we've found the binaries in Homebrew, nix packages and the Ubuntu distro @caybro is using.

@caybro Can you please share the url for the Ubuntu distro you're using? IDK if it helps, but for the record 😄

caybro commented 1 year ago

Homebrew maintained binaries: https://github.com/Homebrew/homebrew-core/pkgs/container/core%2Fqt%2F5/versions?filters%5Bversion_type%5D=tagged For example 5.15.8_3-1: https://github.com/Homebrew/homebrew-core/pkgs/container/core%2Fqt%2F5/89485747?tag=5.15.8_3-1 (includes Linux one)

Yes, at that time we've found the binaries in Homebrew, nix packages and the Ubuntu distro @caybro is using.

@caybro Can you please share the url for the Ubuntu distro you're using? IDK if it helps, but for the record 😄

Not using Ubuntu but Fedora but here you go :laughing: https://packages.fedoraproject.org/search?query=qt5-*

alexjba commented 1 year ago

@yakimant The downside of using Homebrew packages is that the Docker image will increase in size. Around 8gb if I remember correctly. Probably it doesn't really matter if we're downloading the image only once per machine and this doesn't affect the CI jobs duration.

yakimant commented 1 year ago

That's interesting, nix uses KDE patched sources for Qt5: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/qt-5/5.15/srcs-generated.json

But original for Qt6: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/qt-6/srcs.nix

caybro commented 1 year ago

That's interesting, nix uses KDE patched sources for Qt5: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/qt-5/5.15/srcs-generated.json

But original for Qt6: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/qt-6/srcs.nix

Because for Qt 6, there's no difference between publicly available OSS and commercial versions

yakimant commented 1 year ago

Thanks for you comments!

How can I reproduce the issue?

No issues for Windows and macOS, right? (Only finding/building the binaries)

alexjba commented 1 year ago

To reproduce this you'll need to download Qt5.15.8+ binaries and use them to compile status desktop on the Docker image we're currently using.

yakimant commented 1 year ago

Cool! Is it a build time or runtime issue?

yakimant commented 1 year ago

Also, can you please share, which 5.15.8+ binaries did you use, that are compiled with glibc 2.28? And why it is a problem, if Ubuntu 20.04 has glibc 2.31, which is newer: https://packages.ubuntu.com/search?suite=all&section=all&arch=any&keywords=libc-dev-bin&searchon=names

alexjba commented 1 year ago

Also, can you please share, which 5.15.8+ binaries did you use, that are compiled with glibc 2.28? And why it is a problem, if Ubuntu 20.04 has glibc 2.31, which is newer: https://packages.ubuntu.com/search?suite=all&section=all&arch=any&keywords=libc-dev-bin&searchon=names

I've been using the homebew binaries. Most probably these ones: https://github.com/Homebrew/homebrew-core/pkgs/container/core%2Fqt%2F5/62123271?tag=5.15.8. It was most probably a link error. Sorry for not documenting it better and I'm writing now from memory 👎 But it's easy to get to the error. All it's needed is to configure the docker image used on CI to use a newer QT and compile status-desktop using this Qt version.

It's a problem because we need to support latest non-LTS Ubuntu version and compiling the app with a more recent glibc will break the compatibility. But now the latest non-LTS Ubuntu version is questionable to me.

@John-44 I see here the latest non-LTS Ubuntu version is now 18.04 https://www.geekersdigest.com/updated-list-of-linux-distributions-with-long-term-support-lts/. Does this mean we must now support 18.04 and the 16.04 support is a best effort?

John-44 commented 1 year ago

Also, can you please share, which 5.15.8+ binaries did you use, that are compiled with glibc 2.28? And why it is a problem, if Ubuntu 20.04 has glibc 2.31, which is newer: https://packages.ubuntu.com/search?suite=all&section=all&arch=any&keywords=libc-dev-bin&searchon=names

I've been using the homebew binaries. Most probably these ones: https://github.com/Homebrew/homebrew-core/pkgs/container/core%2Fqt%2F5/62123271?tag=5.15.8. It was most probably a link error. Sorry for not documenting it better and I'm writing now from memory 👎 But it's easy to get to the error. All it's needed is to configure the docker image used on CI to use a newer QT and compile status-desktop using this Qt version.

It's a problem because we need to support latest non-LTS Ubuntu version and compiling the app with a more recent glibc will break the compatibility. But now the latest non-LTS Ubuntu version is questionable to me.

@John-44 I see here the latest non-LTS Ubuntu version is now 18.04 https://www.geekersdigest.com/updated-list-of-linux-distributions-with-long-term-support-lts/. Does this mean we must now support 18.04 and the 16.04 support is a best effort?

@alexjba 18.04 support ended in April 2023, so I think we only need to support 20.04 and later Ubuntu LTS releases, and support for 18.04 and 16.04 can be relegated to 'best effort'.

It's fine is we stop supporting 18.04 and 16.04. We shouldn't be expending resources on supporting these versions of Ubuntu given that even Canonical isn't supporting them (without a paid subscription anyhow) atm.

alexjba commented 1 year ago

Great news! Glad we're free to upgrade the docker image to Ubuntu 20.04. It helps a lot with glibc compatibility issues.

yakimant commented 1 year ago

glibc_version_header might have an issue later: https://news.ycombinator.com/item?id=32479263 It's a warning message, need to check.

alexjba commented 1 year ago

glibc_version_header might have an issue later: https://news.ycombinator.com/item?id=32479263 It's a warning message, need to check.

@yakimant I don't think we need any workarounds anymore. The simplest solution I see is to drop the support for Ubuntu 16.04 and go straight to 20.04. Ubuntu 20.04 seems to be compatible with the prebuilt QT binaries. This would mean to update the Docker image used on CI to use Ubuntu 20.04 + latest Qt binaries.

caybro commented 1 year ago

glibc_version_header might have an issue later: https://news.ycombinator.com/item?id=32479263 It's a warning message, need to check.

@yakimant I don't think we need any workarounds anymore. The simplest solution I see is to drop the support for Ubuntu 16.04 and go straight to 20.04. Ubuntu 20.04 seems to be compatible with the prebuilt QT binaries. This would mean to update the Docker image used on CI to use Ubuntu 20.04 + latest Qt binaries.

Yes!

yakimant commented 1 year ago

Yes, I'm leaning to this direction too. Especially with Qt6 official binaries available.

BTW, we use Ubuntu 20.04 for build already: https://github.com/status-im/status-desktop/pull/11390

alexjba commented 1 year ago

Yes, I'm leaning to this direction too. Especially with Qt6 official binaries available.

BTW, we use Ubuntu 20.04 for build already: #11390

Awesome! 👍 Had no idea!

yakimant commented 1 year ago

Looks like Qt 5.15.10 from Hombrew is built againd glibc 2.35.

I have the following errors in build log:

/usr/bin/ld: /home/linuxbrew/.linuxbrew/Cellar/qt@5/5.15.10_1/lib/libQt5Quick.so: undefined reference to `hypot@GLIBC_2.35'

Also objdump shows:

$ objdump -T /home/linuxbrew/.linuxbrew/Cellar/qt@5/5.15.10_1/lib/libQt5Quick.so | grep -v GLIBCXX | grep GLIBC | sed 's/.*GLIBC_\([.0-9]*\).*/\1/g' | sort -uV
2.32
2.33
2.34
2.35

I will think tomorrow, what does it mean. Most probably it was built on Ubuntu 22.04 LTS, which has glibc 2.35 So it cannont be used on anything earlier.

micieslak commented 1 year ago

Hi guys, just one another idea - what about just building qt from sources? 5.15.11 is officially, publicly available here: https://download.qt.io/archive/qt/5.15/5.15.11/single/

jakubgs commented 1 year ago

@micieslak that's why I'm suggesting using Nix, since they already build Qt from source: https://github.com/NixOS/nixpkgs/tree/master/pkgs/development/libraries/qt-5

The problem with using Nix is that it uses recent Glibc, but that could also be remedied in theory: https://discourse.nixos.org/t/compiling-with-old-glibc/12054

yakimant commented 1 year ago

Yes, that's one of the options. I was trying Homebrew binaries first as a quick fix

It doesn't appear to be working though.

so it's a dead end for us, unless we want to stick to 5.15.5 before Qt6.

yakimant commented 1 year ago

For brew packages it works, because they also deploy their own glibc 2.35 as a dependency.

As I understand AppImage packing tools (go-appimage, appimage-builder) can bundle glibc too (+10MB).

I've seen some comments, that flatpak, snap allow this too.

Another finding, Steam supports Ubuntu 12.04 LTS

yakimant commented 1 year ago

Another option is to statically link glibc, but a lot of people don't recommend it.

So we are left with these 2 main directions now:

  1. Bundle new glibc (2.35) into AppImage. Use brew Qt binary packages (5.15.10). Can be done on any Ubuntu, including 22.04.
  2. Build everything with old glibc (at least 2.31, maybe even earlier). Using Nix. On any Ubuntu, including 22.04.
yakimant commented 1 year ago

@jakubgs, @alexjba, @caybro, can you please share your thoughts on "bundling new glibc" vs "building with old glibc"?

I think both are possible, both have some trade-offs.

  1. Probably easier. Certain size overhead. Probably some issues with kernel version dependency.
  2. Full control on build environment. Need to rebuild everything, but it will be cached. Potentially benefit Windows and macOS builds.
jakubgs commented 1 year ago

You know my opinion, I think we should use Nix. But to elaborate more on the topic:

  1. Bundling new glibc into AppImage:
    • Is the simplest and quickest solution.
    • I hate Homebrew, and using it on linux is a travesty, but it would simplify things in theory.
    • Does that actually resolve the issue of AppImage itself using new glibc?
    • Does not address Windows platform.
  2. Building Qt with Nix and possibly using old glibc:
    • Would give us more control over how we build Qt.
    • Would possibly include having to resolve difficult Qt build issues.
    • Using older glibc might result in other things breaking at build or runtime.
    • Potentially does address Windows platform using WSL.

Clearly option nr 1 will require less work than option nr 2.

yakimant commented 12 months ago

Small update on the project status.

I was able to build and pack appimage inside of the pure nix shell. Build is available for testing.

Environment used some nixpkgs snapshot with following:

I had some workarounds, so here is the list of issues will need to be solved later:

Right now I'm working on the following approach: modern nix packages with old glibc: https://discourse.nixos.org/t/compiling-with-old-glibc/12054/4

Other options to keep at the back of the head are:

Interesting tools to explore:

yakimant commented 11 months ago

Good news, I was able to build the app in pure nix shell with overriden glibc.

I've pinned the build here: https://ci.status.im/job/status-desktop/job/prs/job/linux/job/x86_64/job/package/job/PR-9232/30/ I'm on macOS, need to run a Linux VM to test it, so any help is very welcome. It will allow me to fix possible issues earlier.

What I did since my last update:

  1. Override nixpkgs glibc with 2.31 using the method descibed in nix community. It is - using recent glibc derivation with old source code. The other options left behind: a) copying old derivation and use new nixpkgs b) using old nixpkgs and copy new Qt derivation
  2. There were several issues, some were not solved, but workarounded and I need to return to them to later.
  3. Packed linuxdeployqt using 2 historical derivations by @jakubgs and @pedropombeiro
  4. Wrapped ldd to use nix bash, cause it had /bin/sh in shebang. This issue I belive is only relevant to nix-shell, but not nix-build ran in sandbox (chroot) environment. See https://github.com/NixOS/nix/issues/903

Next steps:

yakimant commented 10 months ago

Update on Nix cache for status-desktop dependencies rebuilt with glibc 2.31

TL;DR

nix-cache.status.im is populated. Builds on https://github.com/status-im/status-desktop/pull/9232 are running as usual, taking the same (~16m) time as before.

Saving nix packages for nix-shell from Nix GC

The problem was, that Nix deps were cleaned up, so I could not copy them to the remote cache.

Another problem is, that conventional way of pinning didn't work for nix-shell, because build dependencies are not kept from GC by default (see keep-outputs option and rather long, but detailed desctiption here

So the solution here (one of several possible) was to:

nix-build shell.nix -A inputDerivation -o .nix-shell-inputs

This created a proper GC root, which is keeping the dependencies.

Disk space usage

Initial analysis of Grafana Disk Usage graphs didn't show any spikes of usage during the dependencies build time (both as a Jenkins job and manually).

Later I've got the precises size of shell dependencies:

nix path-info --recursive --closure-size --human-readable /nix/store/...
...
/nix/store/...-status-desktop-build-shell                     2.4G

Showed 2.4G space used. It's not much. We have more than 1T left on build agents and 500G on cache. This will use developers disk though.

Copying to the remote Nix store (cache)

Looks like nix copy is a newer version of nix-copy-closure:

nix copy --to ssh://cache-01.do-ams3.nix.ci.statusim.net -v /nix/store/...-status-desktop-build-shell

Small problem with ssh environment

First is showed

zsh:1: command not found: nix-store
error: cannot connect to 'cache-01.do-ams3.nix.ci.statusim.net'

ssh command failed indeed:

$ ssh cache-01.do-ams3.nix.ci.statusim.net nix-env -h
zsh:1: command not found: nix-env

Fixed by

$ cat ~/.zshenv
if [ -e '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh' ]; then
        . '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'
fi

Check

nix path-info --recursive --closure-size --human-readable /nix/store/...

showed the same on cache host.

Jenkins job succeeded in sane time.

Substituter

As cache worked for status-desktop job worked - system level nix.conf is enough with no additional settings:

$ nix show-config substituters
https://nix-cache.status.im/ https://cache.nixos.org/

Build time

The only difference to privious setup is additional 3m for copying nix dependencies. Probably we will need to pin them, so they are not GCed.

jakubgs commented 10 months ago

The way we avoid losing core dependencies in status-mobile is using the nix/scripts/gcroots.sh script:

# Save derivation from being garbage collected
"${GIT_ROOT}/nix/scripts/gcroots.sh" "shells.${TARGET}"

https://github.com/status-im/status-mobile/blob/a2856cf1c39fdd23e18f33af71bae480500f5c3a/nix/scripts/shell.sh#L64-L65 https://github.com/status-im/status-mobile/blob/a2856cf1c39fdd23e18f33af71bae480500f5c3a/nix/scripts/build.sh#L63-L64

The same happens in build.sh. This script creates files in _NIX_GCROOTS:

# Location of symlinks to derivations that should not be garbage collected
export _NIX_GCROOTS = /nix/var/nix/gcroots/per-user/$(USER)/status-mobile

https://github.com/status-im/status-mobile/blob/bfa23c182fad6c5f7ab595926f1552e74405a19d/Makefile#L46-L47

Which results in this:

 > ls -l /nix/var/nix/gcroots/per-user/${USER}/status-mobile  
total 6
lrwxrwxrwx 1 jakubgs jakubgs 67 Nov  3 15:21 shells.android -> /nix/store/sv178z2sk55rx0mkzd1f8p8rkgb4gsi9-status-mobile-shell.drv
lrwxrwxrwx 1 jakubgs jakubgs 67 Jun  7 17:20 shells.android-sdk -> /nix/store/piwz48yis0plnw77j2yrh7x1s4znnyzx-status-mobile-shell.drv
lrwxrwxrwx 1 jakubgs jakubgs 67 Oct 18 22:21 shells.clojure -> /nix/store/yaiflgbv5d0020zmgvnapyy1nka56nz3-status-mobile-shell.drv
lrwxrwxrwx 1 jakubgs jakubgs 67 Oct 26 19:43 shells.default -> /nix/store/jl9wl4z4108xhlcq7f9vqmrsrbi9h1a5-status-mobile-shell.drv
lrwxrwxrwx 1 jakubgs jakubgs 67 Oct 17 10:52 shells.fastlane -> /nix/store/f2xm1v95nlcpq2jylhazx7k66v45cdbd-status-mobile-shell.drv
lrwxrwxrwx 1 jakubgs jakubgs 67 Oct 19 13:52 shells.gradle -> /nix/store/isjy6j4y2ngb1mn4a7rb8bz2z6v0nb7p-status-mobile-shell.drv
lrwxrwxrwx 1 jakubgs jakubgs 67 Jun  6 12:58 shells.keytool -> /nix/store/gl35knm0bn5d39fk6r18nasa12ywi0va-status-mobile-shell.drv
lrwxrwxrwx 1 jakubgs jakubgs 67 Aug 21 13:25 shells.status-go -> /nix/store/4rfmf1s4wmbc9yz6si1maa316hyc6i14-status-mobile-shell.drv
lrwxrwxrwx 1 jakubgs jakubgs 78 Oct 19 14:00 targets.mobile.android.release -> /nix/store/xnw1av47zwns8gyy5y7qc91278w8kkfm-status-mobile-build-pr-android.drv
lrwxrwxrwx 1 jakubgs jakubgs 70 Nov 15 12:01 targets.mobile.jsbundle -> /nix/store/91x65f4an9gm5hfs8wn2cqhhvjv2hpfc-status-mobile-jsbundle.drv
lrwxrwxrwx 1 jakubgs jakubgs 73 Oct 18 22:17 targets.status-go.library -> /nix/store/z10l2pqrm0npwmym7nvbxzgmfgwh5pn0-status-go-0.171.1-3326362.drv
lrwxrwxrwx 1 jakubgs jakubgs 81 Aug 21 13:23 targets.status-go.mobile.android -> /nix/store/ssjannkisj8pjsf6w0hdrjh14ygvfq6j-status-go-0.163.9-f2f599f-android.drv

These symlinks prevent these derivations and their dependencies from being removed by nix-collect-garbage.

It's not pretty but it works quite well, since it only symlinks the most recent built shells and targets, so old dependencies and artifacts can be removed effectively with garbage collection.

jakubgs commented 10 months ago

As for Nix cache, we do provide the cache via configuration in /etc/nix/nix.conf: 

admin@linux-01.he-eu-hel1.ci.devel:~ % cat /etc/nix/nix.conf 
build-users-group = nixbld
trusted-users = jenkins @admin
experimental-features = flakes nix-command
substituters = https://nix-cache.status.im/ https://cache.nixos.org/
trusted-substituters = https://nix-cache.status.im/ https://cache.nixos.org/
trusted-public-keys = nix-cache.status.im-1:x/93lOfLU+duPplwMSBR+OlY4+mo+dCN7n0mr4oPwgY= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=

Because if we didn't then the config from nix.conf in the repo would be ignored due to multi-user installation that includes a nix-daemon service would not trust it, and hence would not trust the cache public key either.

This doesn't mean the config should not be added to the repo, since it still does take effect for single-user installations if NIX_USER_CONF_FILES environment variable is set properly in the Makefile.

yakimant commented 8 months ago

@caybro, @alexjba, @micieslak,

I have progress on building Status Desktop for Linux in Nix shell. I've used Nix packages with glibc 2.31 and upgraded Qt to 5.15.8.

There were some issues & workarounds on the way, so solution is not perfect so far. But it is built and ran successfully.

Now I need some help from developers to debug a login issue, which I don't have enough invormation to work on: https://github.com/status-im/status-desktop/pull/9232#issuecomment-1910463309

Please help.

I will be doing some check on my side too.

alexjba commented 8 months ago

@caybro, @alexjba, @micieslak,

I have progress on building Status Desktop for Linux in Nix shell. I've used Nix packages with glibc 2.31 and upgraded Qt to 5.15.8.

There were some issues & workarounds on the way, so solution is not perfect so far. But it is built and ran successfully.

Now I need some help from developers to debug a login issue, which I don't have enough invormation to work on: #9232 (comment)

Please help.

I will be doing some check on my side too.

We're currently quite focused 0.16 release. Would be ok for you if I create a separate task and we'll pick it up once 0.16 is wrapped up?

alexjba commented 8 months ago

@yakimant I've created this task https://github.com/status-im/status-desktop/issues/13311. Could you please add some instructions on how to compile the app using nix? I'm afraid it's not just the 5.15.8 upgrade alone because some linux users are already using newer qt versions for the debug builds.

caybro commented 8 months ago

Yeah we can definitely lend our hands; that said, there is already a much more recent Qt OSS release out, namely 5.15.12 available at https://download.qt.io/official_releases/qt/5.15/5.15.12/ . It would be nice to target that

yakimant commented 8 months ago

Jakub asked me to post updates on research here instead of https://github.com/status-im/status-desktop/pull/9232

So far I've fixed some issues by

This didn't help with https://github.com/status-im/status-desktop/issues/13311

But I've found a line, which gives a hint:

Fatal: Could not find QtWebEngineProcess (:0, )

Will research on this issue.

yakimant commented 8 months ago

Instead of workarounging linuxdeployqt issue with Nix, trying to setup env vars, so it does the job: https://github.com/probonopd/linuxdeployqt/blob/2b38449ca9e9c68ad53e28531c017910ead6ebc4/tools/linuxdeployqt/shared.cpp#L1561-L1563

There will be a problem with nss still, possible solutions:

https://github.com/probonopd/linuxdeployqt/issues/35

yakimant commented 8 months ago

QT_INSTALL_LIBEXECS,.. vars didn't help, need to check again later. Continue with current approach of manually copying file.

Running strace of file access show the following:

$ sudo strace --signal='!SIGURG' --trace=access -f status.AppImage
...
[pid 123932] access("/nix/store/ajsb74nncikzrl71pn6la99j3rz0acr7-qtwebengine-5.15.12/libexec/QtWebEngineProcess", F_OK) = -1 ENOENT (No such file or directory)
[pid 123932] access("/tmp/.mount_StatusmaSRVj/usr/bin/QtWebEngineProcess", F_OK) = -1 ENOENT (No such file or directory)
Fatal: Could not find QtWebEngineProcess (:0, )
...
yakimant commented 8 months ago

Have progress, new portion of straces:

[pid 124072] access("/tmp/.mount_StatusdeD4yV/usr/libexec/QtWebEngineProcess", F_OK) = 0
[pid 124072] access("/nix/store/ajsb74nncikzrl71pn6la99j3rz0acr7-qtwebengine-5.15.12/resources/qtwebengine_resources.pak", F_OK) = -1 ENOENT (No such file or directory)
Warning: Qt WebEngine resources not found at /nix/store/ajsb74nncikzrl71pn6la99j3rz0acr7-qtwebengine-5.15.12/resources. Trying parent directory... (:0, )
[pid 124072] access("/nix/store/ajsb74nncikzrl71pn6la99j3rz0acr7-qtwebengine-5.15.12/qtwebengine_resources.pak", F_OK) = -1 ENOENT (No such file or directory)
Warning: Qt WebEngine resources not found at /nix/store/ajsb74nncikzrl71pn6la99j3rz0acr7-qtwebengine-5.15.12. Trying application directory... (:0, )
[pid 124072] access("/tmp/.mount_StatusdeD4yV/usr/bin/qtwebengine_resources.pak", F_OK) = -1 ENOENT (No such file or directory)
Warning: Qt WebEngine resources not found at /tmp/.mount_StatusdeD4yV/usr/bin. Trying fallback directory... The application MAY NOT work. (:0, )
Warning: Path override failed for key base::DIR_QT_LIBRARY_DATA and path '/root/.Status Desktop' (:0, )
[pid 124072] access("/nix/store/ajsb74nncikzrl71pn6la99j3rz0acr7-qtwebengine-5.15.12/translations/qtwebengine_locales", F_OK) = -1 ENOENT (No such file or directory)
Warning: Installed Qt WebEngine locales directory not found at location /nix/store/ajsb74nncikzrl71pn6la99j3rz0acr7-qtwebengine-5.15.12/translations/qtwebengine_locales. Trying application directory... (:0, )
[pid 124072] access("/tmp/.mount_StatusdeD4yV/usr/bin/qtwebengine_locales", F_OK) = -1 ENOENT (No such file or directory)
Warning: Qt WebEngine locales directory not found at location /tmp/.mount_StatusdeD4yV/usr/bin/qtwebengine_locales. Trying fallback directory... Translations MAY NOT not be correct. (:0, )
Warning: Path override failed for key ui::DIR_LOCALES and path '/root/.Status Desktop' (:0, )
[pid 124072] access("/tmp/.mount_StatusdeD4yV/usr/bin/qtwebengine_dictionaries", F_OK) = -1 ENOENT (No such file or directory)
[pid 124072] access("/nix/store/ajsb74nncikzrl71pn6la99j3rz0acr7-qtwebengine-5.15.12/qtwebengine_dictionaries", F_OK) = -1 ENOENT (No such file or directory)
yakimant commented 8 months ago

Also can be set by env vars:

See https://github.com/qt/qtwebengine/blob/5.15.8/src/core/web_engine_library_info.cpp https://doc.qt.io/qt-5/qtwebengine-deploying.html

Looks like need to put them to usr/bin

yakimant commented 8 months ago

After copying all qtwebengine dependencies, app is still crashing, but there is not obvious issue in the logs.

I tried to play with different QT webeinge options:

QTWEBENGINE_DISABLE_SANDBOX=1
QTWEBENGINE_CHROMIUM_FLAGS=--enable-logging --log-level=0 --v=3 --single-process --no-sandbox --disable-gpu

But it didn't help.

Some Chrome flags desctiption pages:

yakimant commented 8 months ago

@alexjba, I'm quite stuck in crush debugging. Let's have a look together when you have time.

Or we could have a small pairing session before. Maybe you could give me some hints, so I could move forward.

yakimant commented 8 months ago

@micieslak helped to disable Qt WebEngine, which we expected to cause a crash: https://github.com/status-im/status-desktop/issues/13540

The app is running and login is succesfull, no crash anymore.

@stefandunca, can you please help to understand, how much do we need / use it? @anastasiyaig, can you please continue smoke testing?

Thank you guys, great help with the project!

stefandunca commented 8 months ago

... @stefandunca, can you please help to understand, how much do we need / use it? @anastasiyaig, can you please continue smoke testing? ...

As discussed privately, we need this for Wallet Connect. There is a workaround using a WebView, but it would require more work to re-implement. Also, the outdated web browser feature for dapp integration, which is now disabled, requires it.