Intermittent Crash (=No Longer Running)

[metal450]

Environment and versions

• Syncthing tray: 1.1.0
• Qt version: 5.6.2
• Operating system: Kubuntu 20.04

Bug description Occasionally (and unfortunately, not consistently reproducible), I'll notice that neither SyncThing Tray nor SyncThing are running on my system. I haven't been able to discern exactly when it happens - i.e. I may be using my system normally for days or weeks, with syncthing working, then at some point I'll notice that it's suddenly no longer running. Once I manually restart it, it runs fine. I'd say on average this happens once every 2 weeks or so.

Here are the full & exact steps I used when setting up both SyncThing & SyncThing Tray (including all settings that were changed from default):

curl -s https://syncthing.net/release-key.txt | sudo apt-key add -
echo "deb https://apt.syncthing.net/ syncthing stable" | sudo tee /etc/apt/sources.list.d/syncthing.list
sudo apt update
sudo apt install syncthing
syncthing (<-1st run will make a config file, start itself, & open the web UI)
Actions->Settings->Connections->Enable Relaying=Uncheck
Actions->Settings->Connections->Enable NAT traversal=Uncheck
Actions->Settings->Connections->Sync protocol listen address=tcp://0.0.0.0:22007
Add Devices & shared folders
Run SyncThing Tray AppImage
Tray->Connection->Insert values from local Syncthing configuration
Tray->Connection->Apply connection settings and try to reconnect
Tray->Connection->Connect automatically on startup
Tray->Connection->Reconnect = 10000 ms
Startup->Autostart->Start the tray icon when the desktop environment launches=ON
Startup->Syncthing launcher->Launch Syncthing when starting the tray icon=ON
Startup->Systemd->Enable & Start

That's it - my complete setup, resulting in the periodic "disappearing" of the running SyncThing Tray.

[Martchus]

Maybe Syncthing Tray crashes, indeed. However, without further information there's really nothing I can help with.

Did you check using a process monitor whether the processes are really gone (and have not just crashed)? In case of a crash I need a stack trace preferably of a build with debugging symbols.

I'm confused about your last 2 setup steps. Are you starting Syncthing using the internal launcher or using systemd? On a system with systemd I recommend the latter. Maybe there's also some information about the process of Syncthing itself available in the journal then.

Note that I provide the AppImage mainly for trying it out. I would not recommend to use it for a permanent setup and rather create some decent packages for my system. Especially the old Qt version which comes within the AppImage might be problematic. (A Qt bug might be the reason for the crashes.)

[metal450]

Did you check using a process monitor whether the processes are really gone

Yup, I checked the process monitor & there was no process.

In case of a crash I need a stack trace preferably of a build with debugging symbols.

Instructions for how I could locate/provide this, next time I find it not to be running?

I'm confused about your last 2 setup steps.

I actually added the systemd step later, once I had experienced this behavior numerous times, just as a possible afterthought that that maybe it would i.e. restart it after it died or something. The behavior was the same without the systemd step.

Maybe there's also some information about the process of Syncthing itself available in the journal then.

Can you give specific steps of what I should do, look for, & report back?

Note that I provide the AppImage mainly for trying it out. I would not recommend to use it for a permanent setup

Oh, I totally agree - I actually spent some time looking for i.e. a proper repository I could install from rather than using the AppImage (so that updates could come from my package manager), but wasn't able to find one. Can you provide a link/instructions? It'd be great to use a repo rather than AppImage, for reasons even beyond this issue :D

[Martchus]

I actually added the systemd step later

Note that you should decide whether you want to use the internal launcher *xor the systemd integration (unless you really want to start 2 instances of Syncthing using different methods).

just as a possible afterthought that that maybe it would i.e. restart it after it died or something.

Systemd can indeed restart the process for you. Checkout the systemd documentation for that. Syncthing Tray's systemd integration has no business with that and it entirely depends on your unit file.

Instructions for how I could locate/provide this, next time I find it not to be running?

Checkout the README for build instructions. For debugging symbols, just add -DCMAKE_BUILD_TYPE=Debug to the cmake arguments. Syncthing Tray is not special in this regard from any other C/C++ application. If you're using systemd on your system you can also use its coredumpctl to access stack traces of crashed processes (regardless whether the processes have been started using a separate unit).

Can you give specific steps of what I should do, look for, & report back?

If you've been starting Syncthing via systemd you can use journalctl -u <name_of_unit>. If you've been using a user unit you also need to use --user with journalctl. For more information please read systemd's documentation. For user the Arch Wiki is also useful to read.

Can you provide a link/instructions?

I don't provide Ubuntu/Debian builds. Looks like they also don't provide any packages themselves yet: https://repology.org/project/syncthingtray/versions

There's already an issue for that but I don't care about Debian/Ubuntu myself so don't expect anything from my side.

[metal450]

Note that you should decide whether you want to use the internal launcher *xor the systemd integration (unless you really want to start 2 instances of Syncthing using different methods).

Ah, that makes sense - so it's not really helping to restart syncthing tray when it crashes, only syncthing itself, which seems like it's not the issue here. Makes sense that that didn't solve it then.

Systemd can indeed restart the process for you. Checkout the systemd documentation for that. Syncthing Tray's systemd integration has no business with that and it entirely depends on your unit file.

Never used systemd nor do I know what a unit file is, but if I understand correctly, it sounds like this may be a solution (i.e. externally setup systemd to relaunch syncthing tray when it crashes) - so I guess this is where I'll start.

Checkout the README for build instructions. For debugging symbols,...

Ah, so I guess that's not possible with the appimage - I have to recompile from source. I guess I'll look into doing that if I can't solve it with systemd.

I don't provide Ubuntu/Debian builds

Bummer, but understood.

Thanks again!

[Martchus]

if I can't solve it with systemd.

Note that building an application and starting/monitoring/restarting an application are different tasks. My hints regarding build instructions where (obviously) about the building part and my hints about systemd about the starting/monitoring part. So both hints could be combined and used together.

Never used systemd

Maybe you're already using it without knowing it. If systemctl status shows all services running on your system (and not just something like "command not found") you're using systemd. Of course this doesn't mean that Syncthing is started as its own systemd unit. However, for getting a stack trace via coredumpctl that's also not necessary. Without debugging symbols that stack trace will not be that useful, though. (On some distributions coredumpctl needs to be installed separately.)

Note that you likely don't want to start Syncthing Tray itself using systemd but only Syncthing itself. That's because Syncthing Tray requires the graphical stack (e.g. X server or Wayland compositor) to be up and running. So it makes more sense to let the graphical shell launch it.

[metal450]

Note that building an application and starting/monitoring/restarting an application are different tasks. My hints regarding build instructions where (obviously) about the building part and my hints about systemd about the starting/monitoring part

Of course :)

Maybe you're already using it without knowing it.

Err, yeah, I meant I hadn't explicitly/manually used it. I knew it was in use, just not how to manually configure.

unless you really want to start 2 instances of Syncthing using different methods

Crap, you're right - I do see 2 separate instances of SyncThing in the task manager. Glad you pointed that out! So I guess I should select either Startup->Syncthing launcher->Launch Syncthing when starting the tray icon or Startup->Systemd->Enable & Start (but not both), right?

Note that you likely don't want to start Syncthing Tray itself using systemd but only Syncthing itself.

Oh ok, then actually what I'd had in mind is not a valid approach. Since it's Syncthing Tray that's "disapearing" (probably crashing), it seems like systemd wouldn't really help.

I guess my next step from here should be to just build from source rather than using AppImage, & I'll leave it like that for awhile & see if it happens again. If not, we know the issue is with the appimage. If so, then can look at doing a debug build.

[metal450]

....Wait, correction again: even with the systemd options disabled, after a reboot, I'm still seeing syncthing twice in the task manager.

If I click "stop launched instance" under Startup->Syncthing launcher, both instances go away. If I "apply and launch now" again, it again starts x2. Is that normal?

[Martchus]

right?

Right. However, the systemd integration might not have worked at all in your case. (It is easy to end up with a setup where it doesn't work, see https://github.com/Martchus/syncthingtray#configuring-systemd-integration.)

Is that normal?

Yes (if one of the processes is the child of the other).

[metal450]

Gotcha.

So I spent some time trying to get it to build, but keep getting tripped up. Since this is only a very intermittent issue, think I'll put it on the backburner for the time being, in hopes of someone coming forward with a deb - or at the very least, now I have a thread to keep track of any future observations if/when it occurs again in the future. Thanks for all the thoughts so far tho :)

[metal450]

Alright, so it definitely seems to be crashing pretty frequently - I'm back on trying to figure this out.

In the build instructions, it says "Install [...] the required Qt modules," which include "core network dbus gui widgets svg webenginewidgets/webkitwidgets." I couldn't find any clear instructions in the readme on how to do this, but after some googling around, I believe this refers to: sudo apt install libqt5core5a libqt5gui5 libqt5network5 libqt5dbus5 libqt5widgets5 libqt5svg5 libqt5webenginewidgets5

However, with that installed, I'm getting a build error:

CMake Error at ../../qtutilities/cmake/modules/QtLinkage.cmake:67 (find_package):
  By not providing "FindQt5Gui.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "Qt5Gui", but
  CMake did not find one.

  Could not find a package configuration file provided by "Qt5Gui" (requested
  version 5.6) with any of the following names:

    Qt5GuiConfig.cmake
    qt5gui-config.cmake

  Add the installation prefix of "Qt5Gui" to CMAKE_PREFIX_PATH or set
  "Qt5Gui_DIR" to a directory containing one of the above files.  If "Qt5Gui"
  provides a separate development package or SDK, be sure it has been
  installed.
Call Stack (most recent call first):
  ../../qtutilities/cmake/modules/QtConfig.cmake:80 (use_qt_module)
  ../../qtutilities/CMakeLists.txt:186 (include)

-- Configuring incomplete, errors occurred!
See also "/home/metal450/CMakeFiles/CMakeOutput.log".

...So my guess is that's not the correct way to "Install [...] the required Qt modules" (on Ubuntu?)

Thanks again~

[Martchus]

I couldn't find any clear instructions in the readme on how to do this

Because it depends on your platform. You'll obviously have to look at their manual or check for what the Qt project provides themselves.

Judging by the error message I'd say the packages you've been installing only contain the Qt libraries but not further development files required at build time. Likely there are some corresponding -devel packages provided by your distribution. Again, you'll have to check their documentation. Maybe they also don't have a concrete documentation - so just have a look how other Qt-based projects are packaged.

[metal450]

Ok, got it. It was my first experience building without 'complete' start-to-finish instructions, but after some trial & error and Googling, the necessary installs were:

sudo apt install cmake build-essential extra-cmake-modules qtbase5-dev libqt5svg5-dev libqt5webkit5-dev libqt5core5a libqt5gui5 libqt5network5 libqt5dbus5 libqt5widgets5 libqt5svg5 libqt5webenginewidgets5 qttools5-dev qtwebengine5-dev libkf5plasma-dev libkf5kio-dev

Also, the make install command required sudo - so my full & complete steps were:

sudo apt install cmake build-essential extra-cmake-modules qtbase5-dev libqt5svg5-dev libqt5webkit5-dev libqt5core5a libqt5gui5 libqt5network5 libqt5dbus5 libqt5widgets5 libqt5svg5 libqt5webenginewidgets5 qttools5-dev qtwebengine5-dev libkf5plasma-dev libkf5kio-dev
mkdir syncthing-tray
export SOURCES=$HOME/syncthing-tray
cd $SOURCES
git clone https://github.com/Martchus/cpp-utilities.git c++utilities
git clone https://github.com/Martchus/qtutilities.git
git clone https://github.com/Martchus/syncthingtray.git
git clone https://github.com/Martchus/subdirs.git
cd $BUILD_DIR
cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX="/install/prefix" $SOURCES/subdirs/syncthingtray
sudo make install -j$(nproc)
sudo mv syncthingtray/tray /opt/syncthingtray

Finally I added it to AutoStart. So now it's running this version instead of AppImage - time will tell if the crashes remain :)

[Martchus]

To anybody who finds this: It is generally not a good idea to build as root, don't copy these commands. See the explanation below.

---

libqt5webkit5-dev … qtwebengine5-dev

You only need to install one of them and the web engine is preferable (see https://github.com/Martchus/syncthingtray#select-qt-module-for-web-view-and-javascript).

libqt5core5a libqt5gui5 libqt5network5 libqt5dbus5 libqt5widgets5 libqt5svg5 libqt5webenginewidgets5

Assuming the distribution packages work like on openSUSE you don't need to install these packages explicitly because the devel packages should already pull in the required libraries.

Also, the make install command required sudo

It requires of course only sudo if the install destination you're configuring is not writable by your current user. I hope you didn't take /install/prefix to literally. Of course one should substitute that part with e.g. some place within one's home directory. You maybe have to make sure that the applications can find the libraries if you install such a location, e.g. by setting LD_LIBRARY_PATH (see https://gitlab.kitware.com/cmake/community/-/wikis/doc/cmake/RPATH-handling). Note that if you really want to install to a destination not writable by your current user you should better specify a writable DESTDIR on the make invocation and move the files into the desired prefix afterwards.

sudo mv syncthingtray/tray /opt/syncthingtray

It looks like you're simply moving the build-tree here. This works only because CMake uses absolute paths for the rpath here. This means your setup will cease to work if you remove the rest of the build directory because the binary in /opt/… is using the libraries from $BUILD_DIR. Note that the build tree is generally not moveable. Either conduct the build where you want to use it or install stuff to the desired location via the install target.

[metal450]

Assuming the distribution packages work like on openSUSE you don't need to install these packages explicitly because the devel packages should already pull in the required libraries.

Yeah, I was basically just struggling/trial-and-erroring to figure out what was missing, & I had installed those non-devel ones first, so it's possible they would've been implicitly included by -devel, which were installed later.

It requires of course only sudo if the install destination you're configuring is not writable by your current user. I hope you didn't take /install/prefix to literally. Of course one should substitute that part with e.g. some place within one's home directory.

...So....I actually didn't even notice that was there & needed to be replaced, I just copied the build command given in the instructions. Might be a good idea to emphasize it above that command block ("Build and install everything in one step (replacing /install/prefix with your desired location)"), or at least make the placeholder path bold or all caps to draw attention to it a bit.

sudo mv syncthingtray/tray /opt/syncthingtray

It looks like you're simply moving the build-tree here. This works only because CMake uses absolute paths for the rpath here. This means your setup will cease to work if you remove the rest of the build directory because the binary in /opt/… is using the libraries from $BUILD_DIR. Note that the build tree is generally not moveable.

Fwiw, it actually did continue to work even after deleting everything in the builddir (I guess I omitted that as my 'final step' - after moving, I deleted everything else that it had left in my homedir).

Note that if you really want to install to a destination not writable by your current user you should better specify a writable DESTDIR on the make invocation and move the files into the desired prefix afterwards.

Yeah - changing /install/prefix to /opt/syncthingtray is also not user-writeable, so although using "sudo make install" did seem to work in this case (including even after removing the build files in my homedir), I guess you're saying this is a bad idea (to install with sudo). If so, I'm still not really sure how I should "move the files into the desired prefix afterwards,' as per above, you said just using my move command shouldn't work (but it seemed to?).

It would be really helpful if you could provide complete build steps that would work for /opt/syncthingtray (a non-user-writeable path), so I don't keep messing stuff up by trying to guess the appropriate commands... :/

Thanks

[Martchus]

Fwiw, it actually did continue to work even after deleting everything in the builddir (I guess I omitted that as my 'final step' - after moving, I deleted everything else that it had left in my homedir).

What I've said only applies when building shared libraries (which one would typically do when creating a GNU/Linux package). However, CMake actually defaults to static libraries so what I've said doesn't apply to your build. Then you can indeed just copy the executable to any place you want and delete the rest.

Might be a good idea to

Yes, I suppose I should overhaul these instructions.

I guess you're saying this is a bad idea (to install with sudo).

The way you've built it does not only install with sudo but actually conducts the whole build as root. The step make install achieves an installation which obviously requires doing the build before so that's done automatically as part of this step. The step cmake … only configures the build but does not do the actual build.

It would be really helpful if you could provide complete build steps …

All steps before the cmake command were fine, that's how I'd do the rest (not tested):

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX="/opt/syncthingtray" "$SOURCES/subdirs/syncthingtray"
make DESTDIR="$PWD/installation" install -j$(nproc) # setting "DESTDIR" is the key
sudo rm -r "/opt/syncthingtray" # avoid keeping old files around
sudo mv "$PWD/installation" "/opt/syncthingtray"

[metal450]

Perfect, thanks - that works, with just one change: the mv command should be

sudo mv "$PWD/installation/opt/syncthingtray" /opt/syncthingtray

:)

[metal450]

It hasn't crashed since I started running the compiled version, so I think it's pretty safe to say that the issue is with the AppImage specifically.

[Martchus]

I'm closing the issue then.

[metal450]

I didn't close the issue because the problem with the AppImage still remains (aka I wanted to report that the crash had been narrowed down, but not solved).

[Martchus]

I've closed the issue because I'm not planning to take any effort in improving the AppImage further. Of course anybody is free to provide better distributions of this project (e.g. a better AppImage or a Debian package) but I don't think we need to keep this ticked any longer open for that.