Open johnny-mayo opened 5 months ago
FIXED...?
I was beat and ready to give up, then I thought I would give Gittyup one last shot.
I cloned the repository and compiled from source, AND IT WORKED.
I get a GUI, and I don't get "Indexer Crashed". Gittyup works perfectly. I can create a new repository. I can open my existing project.
I compiled from master, not from the most recent development branch. I've tweaked the build script for my purposes. WARNING - this will delete the Gittyup directory WARNING - I don't know what I'm doing NOTE - I am not using "git checkout deps", but the result works fine WARNING - did I mention that I don't know what I'm doing?
With that said, the following is a script that "works for me" on my up-to-date Debian system. I figure it should work for Ubuntu and similar, based on apt.
===============================================
sudo apt install build-essential libgl1-mesa-dev sudo apt install cmake sudo apt install libgit2-dev sudo apt install cmark sudo apt install git sudo apt install libssh2-1-dev sudo apt install openssl sudo apt install qtbase5-dev qtchooser qt5-qmake qtbase5-dev-tools sudo apt install qttools5-dev sudo apt install ninja-build
rm -rf Gittyup git clone https://github.com/Murmele/Gittyup.git cd Gittyup
git checkout master git pull origin master
git submodule init git submodule update
cd dep/openssl/openssl/ ./config -fPIC make
cd ../../.. mkdir -vp build/release cd build/release cmake -G Ninja -DCMAKE_BUILD_TYPE=Release ../.. ninja
===============================================
This was after I tried the following (which failed):
flatpak uninstall com.github.Murmele.Gittyup rm -rf ~/.var/app/com.github.Murmele.Gittyup rm -rf ~/.local/share/flatpak/Gittyup rm -rf ~/.local/share/Gittyup rm -rf ~/.config/gittyup.github.com rm -rf ~/python/step4
Then, I start the AppImage version of Gittyup and create a new project at ~/python/step4, and I get the same "Indexer Crashed" error.
Then I "flatpak install com.github.Murmele.Gittyup" and run the flatpak version, and still, no gui.
I really didn't want to reinstall my system from scratch, only for something unknown to change, and have Gittyup stop working again.
Can you build the flatpak package using the DEBUG flag for gittyup?
just pass -DCMAKE_BUILD_TYPE=Debug
to the cmake command inside the manifest
So the gui showed up with version Gittyup 1.3.0 but not with 1.4.0? Because recently we released 1.4.0 maybe this makes problems for you?
KDE Application Platform org.kde.Platform 5.15-23.08 flathub system
you probably have the newest platform version installed?
Can you try a flatpak update
?
I've created a flatpak build on my Debian box with the -DCMAKE_BUILD_TYPE=Debug flag in the manifest.
It still does not open a GUI window. It crashes at exactly the same point in the strace logs. I'm also attaching the merged log file from strace and bpftrace that my script produced. The openat /dev/dri/renderD128 is at 22:29:56.246221 I did the "flatpak build-bundle" to send you the result in flatpak format.
Comments:I noticed that the com.github.Murmele.Gittyup.yaml in the 1.4 source tree is for version 1.3, so I fixed that. I used the install.sh, build.sh, and createBundle.sh you pointed me to. I modified install.sh a little: flatpak remote-delete --user GittyupRepo flatpak remote-add --user GittyupRepo $(pwd)/GittyupRepo --no-gpg-verify --if-not-exists flatpak install --user GittyupRepo com.github.Murmele.Gittyup If you want me to run that again differently, tell me.
I was not sure what environment I should be using to create this build, but maybe that is abstracted away by the flatpak-builder process.
I have an idea about a compiler option I want to try.
Did you get a more readable stacktrace?
@johnny-mayo did you try running flatpak update
? Flatpak comes with its own graphics libraries, which don't automatically get updated when your OS gets a graphics driver update. Running flatpak update
normally installs the newly needed graphics libraries.
I finally got back to this issue and figured it out...well, because another flatpak app was touching /dev/dri/renderD128 and dying a miserable death the same as Gittyup, according to strace.
The TL;DR of it is that something in flatpak is trying to do something in the nouveau driver that the nouveau driver does not support very well, and "pop goes the weasel" (dmesg below).
Solutions: 1) Use the Nvidia binaries (don't like this idea) 2) Maybe play with https://github.com/NVIDIA/open-gpu-kernel-modules 3) Blacklist nouveau from loading and lose accel in programs that run fine with it 4) Maybe disable the X11 Composite extension (but probably not) 5) flatpak override --user --env=LIBGL_ALWAYS_SOFTWARE=1 com.github.Murmele.Gittyup
A little more on that fifth option. It disables hardware rendering for the specified flatpak app. See here:
(base) user@erying:~$ flatpak info --show-permissions com.github.Murmele.Gittyup [Context] shared=network;ipc; sockets=x11;wayland;fallback-x11;ssh-auth; devices=dri; filesystems=home;/tmp;xdg-config/kdeglobals:ro;xdg-run/keyring;
[Session Bus Policy] com.canonical.AppMenu.Registrar=talk org.kde.kconfig.notify=talk org.kde.KGlobalSettings=talk org.freedesktop.Flatpak=talk org.freedesktop.Notifications=talk org.freedesktop.secrets=talk
[Environment] LIBGL_ALWAYS_SOFTWARE=1
...so, it sets the LIBGL_ALWAYS_SOFTWARE as a default env var for the app, so then you can just run it normally:
flatpak run com.github.Murmele.Gittyup
...and it just runs...without hardware acceleration.
Since Gittyup is not something that benefits from GPU acceleration, this works for me, and I don't have to muck with 3rd party binaries or...well...I don't know what the current stability is of Nvidia's open source driver, but it was alpha in 2022...almost daily changes, though...
I would suggest adding the following line to the "finish-args:" section of: https://github.com/Murmele/Gittyup/blob/master/com.github.Murmele.Gittyup.yml
--env=LIBGL_ALWAYS_SOFTWARE=1
Notes: I am a little surprised "flatpak update" did not fix it. Nor did: sudo flatpak override --device=all com.github.Murmele.Gittyup Nor did making sure the following were installed and updated: vulkan-utils mesa-vulkan-drivers mesa-utils libgl1-mesa-glx libx11-xcb1 libxcb-dri3-0 libxcb-present0 libpciaccess0 libdrm2 libxshmfence1 libxxf86vm1 libbsd0 Nor did these help: --log-level=DEBUG, LIBGL_DEBUG=verbose, sudo systemctl stop apparmor, --filesystem=home, --repair
I did not try (but will if someone gives reason to):
flatpak install flathub org.freedesktop.Platform.GL.default
flatpak install flathub org.freedesktop.Platform.GL.nvidia-
As an aside, I would really like some input on https://github.com/Murmele/Gittyup/discussions/789 I've done quite a bit of googling...but is it a non-problem because of a common solution? I'd like to know from someone that knows. This portable container stuff is interesting, and I like that AppImage does not require any app be installed to run it (like flatpak) ...and, golly gee, it seems to work with nouveau just fine...(why? hmm)
Anyway, the dmesg logs, as promised:
[22474.995230] BUG: kernel NULL pointer dereference, address: 0000000000000000
[22474.995234] #PF: supervisor read access in kernel mode
[22474.995235] #PF: error_code(0x0000) - not-present page
[22474.995236] PGD 0 P4D 0
[22474.995237] Oops: 0000 [#16] PREEMPT SMP NOPTI
[22474.995239] CPU: 1 PID: 50522 Comm: gittyup Tainted: G D 6.1.0-22-amd64 #1 Debian 6.1.94-1
[22474.995241] Hardware name: INTEL HM570/HM570, BIOS THM570106 06/14/2022
[22474.995242] RIP: 0010:nvkm_gr_units+0x5/0x20 [nouveau]
[22474.995293] Code: 00 48 85 ff 74 12 48 8b 07 48 8b 40 58 48 85 c0 74 06 ff e0 cc 66 90 cc 31 c0 c3 cc cc cc cc 66 0f 1f 44 00 00 0f 1f 44 00 00 <48> 8b 07 48 8b 40 48 48 85 c0 74 06 ff e0 cc 66 90 cc 31 c0 c3 cc
[22474.995294] RSP: 0018:ffffbd34c3187d48 EFLAGS: 00010202
[22474.995295] RAX: ffff9c4c12056000 RBX: ffffbd34c3187dd8 RCX: ffff9c4c01cb10d0
[22474.995296] RDX: ffff9c4e24f37c00 RSI: 0000000000000000 RDI: 0000000000000000
[22474.995297] RBP: ffff9c4c323a4000 R08: 000000000000000d R09: 000000000000e280
[22474.995297] R10: 0000000000000010 R11: 0000000000000000 R12: ffffffffc0a83c10
[22474.995298] R13: ffffbd34c3187dd8 R14: ffff9c512b865400 R15: 0000000000000010
[22474.995299] FS: 00007f5246087880(0000) GS:ffff9c538fa40000(0000) knlGS:0000000000000000
[22474.995300] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[22474.995301] CR2: 0000000000000000 CR3: 0000000611c9c001 CR4: 0000000000770ee0
[22474.995302] PKRU: 55555554
[22474.995302] Call Trace:
[22474.995304]
Thank you,
johnny-mayo
Yes, adding --env=LIBGL_ALWAYS_SOFTWARE=1 to the finish-args: section of https://github.com/Murmele/Gittyup/blob/master/com.github.Murmele.Gittyup.yml produces a Gittyup flatpak bundle file that, when installed, properly sets the environment variable and allows the Gittyup flatpak to produce a GUI when run with a normal "flatpak run com.github.Murmele.Gittyup" by the user without having to do anything special because hardware video acceleration is disabled by the environment variable, and the the software emulation works just fine.
I hope to do the pull request tomorrow.
In the attached issue783.zip file are all the scripts and Dockerfile needed to create an archlinux:latest docker image (like in the github CI) with everything installed that is necessary to build Gittyup.flatpak.
Start with README
Toodles,
johnny-mayo
But switching to Software Rendering is not the solution. It works for a lot of other distributions and it makes the software slower
I agree, completely. My solution is a hacky kluge work-around.
The problem is not with Gittyup. This used to work, then stopped, with the same 1.4 version of Gittyup. It also affects Anki flatpak in exactly the same way, according to strace logs, which is the reason I came back to this problem.
The factors involved, I think, are the video driver, video driver version, and flatpak (with its video shared objects and other support stuff) and possibly other library version issues in the system.
My few google searches gave me the impression this is not something of a priority on flatpak's side. They seem to point to something (anything?) else in the system and close the situation. "Not my problem" syndrome.
I don't have the resources to investigate this non-Gittyup problem further.
I would like Gittyup to work for more people.
Thank you,
johnny-mayo
P.S. I might not be able to do the pull request until tomorrow, if you still want to go this direction.
This seems to me like it's most likely a nouveau problem. Until recently, nouveau was so unstable and unfinished that even 2d desktop usage was problematic. You being on Debian Stable suggests that you are still using a nouveau version that is in such a state. From the dmesg you shared, it looks like this is a kernel bug and should be fixed there. Most likely it has already been fixed a while ago and Debian Stable is just missing said fix. On Debian Testing, nouveau works surprisingly well nowadays, even if I would still not recommend it because some basic features are still missing.
Thank you very much for the input.
From my limited googling, I got the impression that even the proprietary Nvidia binary blobs were causing issues with flatpak, but I had no idea that Nouveau was in such a sad state of affairs on Debian stable. Maybe I should try out the open source Nvidia option?...they seem to be acting like it is important to put energies into its development.
Both the compiled from source and the AppImage versions of Gittyup work just fine with Debian stable Nouveau. It seems to be more of a "flatpak thing" with Nouveau (and/or the kernel, you say?), which is ironic, considering one of the goals of flatpak is to be able to run apps on any Linux based OS, but instead the app runs anywhere but on flatpak. Again, the same problem affects the Anki flatpak, with the same "workaround" solution.
I am curious, though, since my assumptions may have been incorrect...how does Gittyup benefit from hardware acceleration?
Whatever the root cause(s) might be, the decision must be made as to whether it is better to completely avoid the possibility of the issue (at a performance cost?), or if it is better to try to make sure the users become aware of this workaround or be told they need a different video driver if they have problems with the flatpak version, or to not use the flatpak version. I suppose a deciding factor would be the frequency of occurrence, if that information is available.
In that respect, I have no input. I'm here to help. I'm out of resources, though.
Thank you,
johnny-mayo
I would think that on a normal system, performance would be similar. It just renders stuff on the CPU instead of the GPU, which takes processing power away from something like a compiler running alongside Gittyup. Where it might make a bigger difference are less powerful machines, like single board computers.
I think the main issue is likely just feeling wasteful rendering on the CPU, when there is a GPU available for the task, and also it feeling like we are fixing an issue in the wrong place.
One thing I am wondering though: As far as I know, LIBGL_ALWAYS_SOFTWARE
is a feature of Mesa; Would that even work on systems not using Mesa (like proprietary Nvidia, proprietary AMD(?), and others)?
I offer my reflections, but in no way wish to steer opinions. This is just a thought experiment for me,
at this point.
One little thing...Gittyup is built from an ubuntu-latest vm in the CI, and from that, the AppImage is built,
which works just fine, but the flatpak version is built on an Arch docker image/container, and it doesn't
work (if my very limited CI yml interpretation is correct)...not sure it would prove much if it actually
worked with ubuntu, though.
I did not think to consider single board or other low-end computers. Two things come to mind. First,
I hope it is no longer the case that hardware acceleration support on SBCs is generally lacking.
Second, if the image Gittyup renders is static because the user is not manipulating the program
during compile (or some other CPU intensive task (on an SBC?)), would they notice any performance
difference from GPU acceleration? I don't think their expectations should be particularly high, anyway.
From what I gather, LIBGL_ALWAYS_SOFTWARE is only used if the Mesa library is used for OpenGL
rendering. Maybe this is a good thing, as it would not disable hardware acceleration, and, thus, be a
performance detriment, except in the case of Mesa?
Here's a disgusting and terrible idea: In the Gittyup flatpak, point the manifest to run a wrapper script that
sets LIBGL_ALWAYS_SOFTWARE only if the specific criteria leading to failure is met, so, something like:
#!/bin/bash
if lspci -k | grep -E "Kernel driver in use: nouveau"; then
if (do something here to figure out if the nouveau version is less than x.x); then
export LIBGL_ALWAYS_SOFTWARE=1
fi
fi
exec /path/to/your/application "$@"
I have no idea how to tell, across distros, using programs that are always installed, not requiring root,
and not just looking at the symlinks, the version of nouveau in use.
I have no idea what other specific factors should be tested for, either. I do not see a lot of questions being
asked when "program no workie" in flatpak discussions or issues. More googling could be done.
I did say it was a bad idea.
Anyway, thought experiment. Infinite rabbit holes. I've got to leave this issue to others. I do, I do.
johnny-mayo
After using it for a month or so, Gittyup (Debian, xfce (X11, not Wayland), flatpak) now seems to start, but does not show the graphical window.
If I start it with the command line with "flatpak run -v --branch=stable --arch=x86_64 --command=gittyup com.github.Murmele.Gittyup", the process starts, shows some messages/errors/warnings, and hangs.
One message is: 'Failed to load module "xapp-gtk3-module"', which, after a lot of googling, is something to do with flatpak, but not something that would keep a flatpak app from running. The next message (which further tells me that the last message was not a terminal error) is "Qt: Session management error", which is an ignorable message that can be eliminated by "unset SESSION_MANAGER" before starting the app. Not that it matters, but "sudo apt install xapp" and sudo apt install xapp-gtk3-module" do not find anything by those names in apt, but that was the extent to which I unnecessarily(?) tried to fix that problem.
If I follow the instructions here: https://docs.flatpak.org/en/latest/debugging.html, and run Gittyup in gdb, I get:
(gdb) run Starting program: /app/bin/gittyup [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x7ffff27786c0 (LWP 24)] [New Thread 0x7ffff1f776c0 (LWP 25)] Gtk-Message: 14:49:16.097: Failed to load module "xapp-gtk3-module" Qt: Session management error: Authentication Rejected, reason : None of the authentication protocols specified are supported and host-based authentication failed [New Thread 0x7fffebfff6c0 (LWP 26)] [New Thread 0x7fffeaa786c0 (LWP 27)] [New Thread 0x7fffea2776c0 (LWP 28)] [New Thread 0x7fffe9a766c0 (LWP 29)] [New Thread 0x7fffe92756c0 (LWP 30)] [Thread 0x7ffff65e3880 (LWP 21) exited]
...then, after a 10 seconds or so:
[Thread 0x7fffe92756c0 (LWP 30) exited] [Thread 0x7fffe9a766c0 (LWP 29) exited] [Thread 0x7fffea2776c0 (LWP 28) exited] [Thread 0x7fffeaa786c0 (LWP 27) exited]
...then there is nothing more in the logs, and no gui.
If I start it from xfce, I see this process tree: xfce4-panel─┬─bwrap───bwrap───gittyup───3*[{gittyup}]
When I used strace with "strace flatpak run -v --branch=stable --arch=x86_64 --command=gittyup com.github.Murmele.Gittyup |& tee gittyup.strace.log" and grepped through the output file, I saw that it was referencing many config files, so I tried uninstalling all flatpak apps, deleting ~/.var/app/com.github.Murmele.Gittyup, apt uninstalling flatpak, and reinstalling everything from scratch, and I see that it is no longer referencing the .git/gittyup files in my project directories (which, yes, I tried deleting previously), but still, no gui.
I even tried "sudo chmod u+s /usr/bin/bwrap" for giggles, but no dice.
The only file in the ~/.local/share/flatpak/ tree is config, which contains:
[core] repo_version=1 mode=bare-user-only min-free-space-size=500MB
The Appimage version of Gittyup works just fine. I wonder if it has anything to do with the environment or config file differences between the Appimage and the flatpak versions.
I have VSCodium installed in flatpak, and it works just fine. If I "strace flatpak run com.vscodium.codium |& tee vscodium" and use meld to diff the output with that of Gittyup, I see that it does the same miniconda3 bwrap stuff, but, besides the architectural differences, I don't see anything that stands out.
Everything was working fine, until I did an update on apt and flatpak (and maybe a "sudo shutdown -r now" while everything was running), so I tried installing and fully updating in a Debian VM, and it works fine. In the VM, however, I do not have miniconda installed, for one.
In the VM, where the Gittyup flatpak runs just fine (from xfce or terminal), grep returns no mention of "xapp-gtk3-module", but there is the "Qt: Session management error" line, followed by the screenshot I have included.
Regarding the apt and flatpak updates, here is what was updated in apt at that time:
Start-Date: 2024-06-17 14:55:31 Commandline: apt upgrade -y Requested-By: user (1000) Upgrade: containerd.io:amd64 (1.6.32-1, 1.6.33-1), docker-compose-plugin:amd64 (2.27.0-1debian.12bookworm, 2.27.1-1debian.12bookworm), docker-ce-cli:amd64 (5:26.1.3-1debian.12bookworm, 5:26.1.4-1debian.12bookworm), gstreamer1.0-gl:amd64 (1.22.0-3+deb12u1, 1.22.0-3+deb12u2), libarchive13:amd64 (3.6.2-1, 3.6.2-1+deb12u1), libavdevice59:amd64 (7:5.1.4-0+deb12u1, 7:5.1.5-0+deb12u1), libgstreamer-gl1.0-0:amd64 (1.22.0-3+deb12u1, 1.22.0-3+deb12u2), ffmpeg:amd64 (7:5.1.4-0+deb12u1, 7:5.1.5-0+deb12u1), libarchive-tools:amd64 (3.6.2-1, 3.6.2-1+deb12u1), gstreamer1.0-alsa:amd64 (1.22.0-3+deb12u1, 1.22.0-3+deb12u2), gir1.2-gst-plugins-base-1.0:amd64 (1.22.0-3+deb12u1, 1.22.0-3+deb12u2), libpostproc56:amd64 (7:5.1.4-0+deb12u1, 7:5.1.5-0+deb12u1), docker-buildx-plugin:amd64 (0.14.0-1debian.12bookworm, 0.14.1-1debian.12bookworm), docker-ce:amd64 (5:26.1.3-1debian.12bookworm, 5:26.1.4-1debian.12bookworm), libndp0:amd64 (1.8-1, 1.8-1+deb12u1), gstreamer1.0-x:amd64 (1.22.0-3+deb12u1, 1.22.0-3+deb12u2), libavcodec59:amd64 (7:5.1.4-0+deb12u1, 7:5.1.5-0+deb12u1), gstreamer1.0-plugins-base:amd64 (1.22.0-3+deb12u1, 1.22.0-3+deb12u2), docker-ce-rootless-extras:amd64 (5:26.1.3-1debian.12bookworm, 5:26.1.4-1debian.12bookworm), python3-pil.imagetk:amd64 (9.4.0-1.1+b1, 9.4.0-1.1+deb12u1), libavutil57:amd64 (7:5.1.4-0+deb12u1, 7:5.1.5-0+deb12u1), libswscale6:amd64 (7:5.1.4-0+deb12u1, 7:5.1.5-0+deb12u1), python3-pil:amd64 (9.4.0-1.1+b1, 9.4.0-1.1+deb12u1), libswresample4:amd64 (7:5.1.4-0+deb12u1, 7:5.1.5-0+deb12u1), libavformat59:amd64 (7:5.1.4-0+deb12u1, 7:5.1.5-0+deb12u1), libgstreamer-plugins-base1.0-0:amd64 (1.22.0-3+deb12u1, 1.22.0-3+deb12u2), firefox-esr:amd64 (115.11.0esr-1deb12u1, 115.12.0esr-1deb12u1), libavfilter8:amd64 (7:5.1.4-0+deb12u1, 7:5.1.5-0+deb12u1) End-Date: 2024-06-17 14:55:53
I'm not sure how to see what versions of Gittyup dependencies were in flatpak before I did the flatpak app update. I think the versions of the dependencies are what the Gittyup flatpak app says it needs, though, and the Gittyup flatpak version hasn't changed in some time...should not the dependency versions installed in flatpak remain the same?
I have to point out that I've installed updating apt and flatpak apps in the VM works fine, and with xfce, so I don't think it is a version problem. In my hardware box with the problem, I've unintstalled and wiped out every config file I could find and resinstalled from scratch, which did not fix the problem, so I don't think it was a config file corruption problem. I do not have a standardized Debian config, as I install tools when I find them as solutions to problems I have, so it would be near impossible to recreate the environment I have.
For me, I'm fine with running the Appimage version. Maybe this is really a flatpak issue.
For the purposes of helping the community, I'm willing to try suggestions to resolve this issue. gittyup.strace.log