release 3.0.0 alpha0 (appimage)

[Elv13]

Alpha0 will be released soon. Last minute testing showed an upstream regression make the daemon crash during some necessary account import steps. I am holding the release until this is resolved. It is currently unclear if it's exploitable, but it does leak the private key into the core dump and main memory during the crash, so I consider this too serious to ask "the world" to test this version until a proper mitigation has been found. Hopefully it wont require to disable account import for Beta1, but this is a possibility.

[star-buck]

@Elv13 : To allow easy testing, please start building an appimage.

[Elv13]

Just an update to confirm I am still working on the appimage. Scarlett spent a lot of time a few months ago and it never fully worked. So far this is the same here. The previous work is kind of a dead end. First of all, what that system did was very complex. The Dockerfile to build Scarlett appimage is huge and doesn't scale well (edit: and can't work because of the new -fPIC requirement). It requires tons of work for each new appimage and is a bit unsustainable from a maintenance POV. It's also no longer compatible with neither the current daemon or the new Ring-KDE QML rewrite. After discussion with other devs at Randa, we agreed on making the whole process better and more scalable.

To do this, I am switching to static builds instead of dynamic ones. This reduces the number of components that have to be configured in the appimage by ~90%. However both Qt and KF5 (https://bugreports.qt.io/browse/QTBUG-38913) need to improve support for this. There isn't a million way forward. It's either that or having people working full time on maintaining this forever. The problems Scarlett faced with all the appimages are real and need to be solved instead of using brute force (and tons of time) to mitigate.

Better self contained app support also helps a lot with sandboxing like Snap and Flatpack by reducing the image complexity by the already mentioned 90%. It's the same technique that has over the last couple years taken over the Docker world. There is something called Alpine Linux and Musl LibC that are now the most popular container target. This is the same thing as they did. It fixes the same issue they had and we currently have. Static builds create smaller package, load faster, are more secure and are cheaper to maintain and deploy.

The downside of this (and the whole "have an appimage for beta 1") milestone is that none of this is simple, none of this is tested and none of this is quick to get to work. I am done with getting rid of Ring-Daemon and embedding LibRing into ring-kde. I am mostly done having the static Qt fixed. I also removed some dependencies from ring-kde to simplify sandboxing and appimages (and Android, as a free side effect...). What remains is having an environment where all KF5 framework behave correctly. At this speed, it's about a week off if everything goes well, but there is remaining unknowns.

Getting rid of the daemon and embedding the code also solves the problem where the ring-daemon+ring-lrc+ring-kde topology would have been incompatible with every systems where the Gnome client is installed as the version would have conflicted. This issue would have potentially caused data loss and could not have been shipped. This system was already used for the official Ring.cx Windows build, but hasn't been fully tested/supported on Linux yet, let alone sandboxed. That's solved.

One of the limitation I am currently facing that I will have to ship 3 appimages instead of one. One for Alsa, one for PulseAudio and one for JACK. Not doing that would break audio on systems using either Alsa (without Pulse) or JACK (a bit rare and limited to media professionals). However this doesn't require any additional development time. It's as simple as changing the apt-get command in the appimage builder to include or exclude these systems. There may or may not be the need for a build with video acceleration and one without. I am not certain yet if this is required or not. I fear that it will crash or fail to work when video starts if the hardware doesn't support accel correctly (like the nouveau video driver or vesa). I believe this is the issue that prevented sgclark from getting it to fully work, but I am not certain. I will investigate once I get the appimage working.

[star-buck]

Hi Emmanuel,

please note that Scarlett has been reassigned to not work on any appimages.

You would have to do the appimage yourself and look around for help, like on github.com/appimage, #irc appimage etc.

There are many appimages out there like vidcutter, etc. so ring-kde should be doable.

Just the automation scarlett was attempting isnt feasable hence her reassignment.

Building an appimage of ring-kde successfully also demonstrates that things arent overly complex as they used to be for the ring-related parts required (not talking about alsa, jack, etc., as i never heard of any music player appimage do that either).

[Elv13]

Scarlett has been reassigned to not work on any appimages.

I know. What I was referring to is that I started using her work as a starting point to investigate the system and identify the issues that caused the effort to fail. I didn't want to go in full "not invented here" mode until I was knowledgeable about the existing body of work that has been invested into appimage.

You have to do the appimage yourself

I am very well aware of that and have been working on it for the last 4 days (excluding Saturday, because I was traveling away from Randa). Ever since I was done with the actual beta development. By the way, it's less or more ready to be released, all that's missing is the appimage.

so ring-kde should be doable.

Doable: yes, but an order of magnitude more complex than a single process app. Applications using a single executable seems quite easy to package. However the default Ring topology is client/server and is spread across multiple singleton processes. None of those process can be supplied or even exists on the host system without causing the whole stack to fail. This is why the previous message discusses combining them into a static binary. Then, the daemon has an huge number of dependencies. I am done with the daemon part and now working on client related issues.

Just the automation scarlett was attempting isnt feasable hence her reassignment.

This is correct. I don't think the way the automation system did things could scale. I am attempting to standardize and enable automation on a very small subset of the system to avoid doing repetitive tasks on the worst parts of the process. I am not attempting to automate a generic system for every KDE apps. This would take months and it's not my job.

As detailed in the previous message, it revolves around improving the ability of KF5 to bootstrap its frameworks into the application using static linkage. The goal is to reduce the number of Qt5/KF5 libraries installed into the appimage fstree by 90%. This will in turn reduce the maintenance (monetary) cost. After discussing this with some other KF5 and KDAB devs, I am quite convinced this is the best (technical) way to go for Ring-KDE (and eventually other KDE applications). As previously mentioned, this feature is also favorable for FlatPack and Snap to improve security and maintainability. I also discussed this with dvratil, who originally authored FlatPack while he was working at RedHat (he has since moved to KDAB).

Building an appimage of ring-kde successfully also demonstrates that things arent overly complex as they used to be for the ring-related parts required (not talking about alsa, jack, etc., as i never heard of any music player appimage do that either).

It's a different area of complexity. Building Ring-KDE when you already have a daemon is now trivial as it fetches its dependencies itself and self-automate the setup. Before you hired me to work on it, the setup required compiling 3 different applications with their own set of challenges.

However, that improvement is totally parallel to the issues faced with the appimage. The appimage problem are due to the 200+ dependencies induced by the whole stack and the already mentioned multi-process topology. It will work and will be fixed, but the challenges here are not faced by your average appimage.

As for the little sound system non-issue, this is very minor and is not really a problem. It could be solved on the software side too, but 3 images is simpler for now. Anyway, it affects like 1% of the potential userbase. Back in the SFLPhone days, indie radio stations were using sflphone-kde because it was the only free/libre software that allowed to route calls to their equipment correctly. However it is a microscopic market not worth spending any amount of time on. The biggest "problem" is that it will crash on Arch and Gentoo systems (or Neon/Maui/NetRunner users who removed Pulse) where it cannot load the pulse .so. Therefor having a separate image makes more sense that writing software code to detect the current sound system.

[star-buck]

On Sep 19, 2017 10:17, "Emmanuel Lepage Vallée" notifications@github.com wrote:

Scarlett has been reassigned to not work on any appimages.

I know. What I was referring to is that I started using her work as a starting point to investigate the system and identify the issues that caused the effort to fail. I didn't want to go in full "not invented here" mode until I was knowledgeable about the existing body of work that has been invested into appimage.

Okay great, wasnt aware of that.

Building an appimage of ring-kde successfully also demonstrates that things arent overly complex as they used to be for the ring-related parts required (not talking about alsa, jack, etc., as i never heard of any music player appimage do that either).

It's a different area of complexity. Building Ring-KDE when you already have a daemon is now trivial as it fetches its dependencies itself and self-automate the setup. Before you hired me to work on it, the setup required compiling 3 different applications with their own set of challenges.

However, that improvement is totally parallel to the issues faced with the appimage. The appimage problem are due to the 200+ dependencies induced by the whole stack and the already mentioned multi-process topology. It will work and will be fixed, but the challenges here are not faced by your average appimage.

As for the little sound system non-issue, this is very minor and is not really a problem. It could be solved on the software side too, but 3 images is simpler for now. Anyway, it affects like 1% of the potential userbase. Back in the SFLPhone days, indie radio stations were using sflphone-kde because it was the only free/libre software that allowed to route calls to their equipment correctly. However it is a microscopic market not worth spending any amount of time on. The biggest "problem" is that it will crash on Arch and Gentoo systems (or Neon/Maui/NetRunner users who removed Pulse) where it cannot load the pulse .so. Therefor having a separate image makes more sense that writing software code to detect the current sound system.

Sounds feasable. I have also asked some people from appimage over here to have a look and possibly help.

[TheAssassin]

The AppImage team can provide support, yes. It will be best to do it interactively, so I'd invite you to join our IRC channel on Freenode: #AppImage. We should be able to build an AppImage that pleases everyone and comply your requirements.

I am not 100% sure what your problems are (and I can't check out the source code), but it looks like a Qt dependency issue. Have you seen linuxdeployqt? This project (which is also developed by the AppImage team) makes bundling Qt plugins and libraries for AppImages really easy. It prepares the AppDir, bundles non-Qt libraries (which you need for the AppImage anyway), and can even build the final AppImage in a single step.

By the way, you will want to script the AppImage creation right from the beginning. I have a set of sample scripts which we might copy-paste together to e.g., do it in a Docker container.

So, as said before, join #AppImage and we'll get it done I guess.

[star-buck]

alpha0 done. will add new ticket for alpha1