Consider packaging Audio Proj. Manager/Transcriber as a flatpak or snap

[n8marti]

I'm one of the main Wasta developers/maintainers and a general Linux supporter, and I've been looking at various SIL apps and how they're packaged/distributed. I think distributing via either snap or flatpak is a good long-term strategy for both reaching the most number of users and for simplifying the building and maintaining of desktop software. And on the Wasta side we are supporting both formats as fully as possible (Software Store integration, offline installs and updates, etc.).

If you are looking for more feedback, I'm happy to share about what I've packaged so far using these formats and some of the pros and cons I've seen. I'd also be willing to contribute to make it happen. Also, the Bloom and FLEx teams have begun packaging using flatpaks, so they might have even better feedback there. I'm more familiar with snaps, but I've now published packages in both formats:

• https://snapcraft.io/uniscape (simple PyTk app)
• https://snapcraft.io/lameta-unofficial (Electron/yarn "sister" app to SayMore)
• https://snapcraft.io/squeeze-vid (Python/ffmpeg CLI app)
• https://flathub.org/apps/details/io.github.sil_car.SqueezeVid (same CLI app from above, but as a flatpak)
• https://snapcraft.io/logos10-unofficial (Logos 10 via WINE)

[gtryus]

We looked at snap and flatpak earlier and didn't use these formats because Audio project manager often functions as a conduit. On Windows, users can edit their audio with Audacity if they want a more powerful audio editor. Package with either of these formats puts the program into a sandbox and blocks integration with other applications installed on the OS.

That being said, I am open to hearing counter arguments. Up until now, deb packages also install on all the flavors of Linux including Wasta. (I had read the LTS release notes and it talked of software in any of the three formats.)

[n8marti]

Can you describe the integration more specifically? The sandboxing is rather flexible with flatpaks, and still kind of flexible with snaps. One insurmountable obstacle might be if APM wants to communicate directly with other running processes.

Access from APM to other apps

• Files: If APM needs access to files in the user's home folder (by maintaining a "transcriber" folder under /home/user like the debian package does, or by having read/write access to the Paratect project folder), that's easy to allow for in both snap and flatpak formats, as long as these files are not found in a hidden folder (e.g. can't be in /home/user/.config/, /home/user/.local, etc., but they can be in /home/user, /home/user/Documents, etc.).
• Libraries: If APM needs access to some library installed within Paratext (e.g. somewhere in /usr/lib/Paratext9), then that's not going to be allowed (well, it might be possible with flatpak, maybe).
• Executables: If APM just needs to execute a command already found in the system PATH, then that's no problem.

Access from other apps to APM

• Files: If another app needs access to APM's files, it's the same solution as the first point above.
• Libraries: If another app needs access to some library provided by APM, then it would maybe have to be adapted to look in a different location than with a debian package.
• Executables: If another app needs to execute a command provided by APM, then that executable would need to be exported to PATH when installed. This is fairly straightforward with snaps. I'm not sure how it would work with flatpaks, though.

For me, though, there are some strong advantages to these packaging formats over debian packages:

1. The package doesn't have to be maintained for potentially multiple releases of multiple debian-based distributions (maybe APM only supports LTS versions of Ubuntu/Wasta anyway, but this leads to the 2nd advantage).
2. The app can be packaged once and run the same way on any distribution (debian-based or not) that provides the infrastructure for either flatpak or snap. This makes it possible to support a broader range of users.
3. The app is made available publicly through the flatpak/snap store without the user having to add a special source archive or going to the download page on software.sil.org/transcriber. This also facilitates updates (can be automatic, or can be user-controlled), which with flatpaks/snaps are still handled by the system instead of the user having to check the website periodically for a new installer.

[gtryus]

Audio Project Manager manages a large corpus of audio files. It provides a recording end editing tool for recordings. Some users come to Audio Project Manager having used Audacity to try to manage a large corpus of files. Audacity is an audio editor that can do much more sophisticated editing than Audio Project Manager but it works with a single audio file. So those using Audacity must have a rigorous naming scheme and stick to it religiously. Audio Project Manager saves the team from the details of the file management.

Audio Project Manager allows those who are using the desktop application to integrate with Audacity and use it as their audio editor. The way this works is that when a user first indicates they would like to use Audacity, Audio Project Manager creates a copy of an empty Audacity project and renames it. Then it modifies the configuration file of Audacity so that when Audacity is opened, it opens that particular project and the folder and import and export locations are set to use folders related to the passage they are working on in Audio Project Manager. Thus if they import or export content, they don't have to name or navigate to folders. This allows the user to basically use Audacity as their audio editor with Audio Project Manager without having to have a rigorous naming scheme. They don't end up naming files at all but just using defaults provided by Audio Project Manager.

(Some users have asked for similar functionality with ELAN. This tool again works with a single file but provides ability to have transcriptions attached to different time stamp sequences the way Audio Project Manager does for back translations. This level of integration with ELAN is not completely implemented -- at least not yet.)

In any case, the issue is not with where Audio project manager stores its audio files. That part should work well with flatpack or snap (although I haven't played with it a lot). The issue is that Audio Project Manager needs to be able to reach into the Audacity sandbox and modify its files for this level of integration to work.

Earlier in the Audio Project Manager work, we created both snap and deb packages. It is actually fairly trivial to do and we could in fact create all three kinds of packages. All this is supported by electron-builder. The problem is the confusion this creates for the users. I could create both a deb version and a snap version. If Wasta preferred the installation of the snap version, it would work for all users who didn't care to use Audacity as their audio editor. (We have users who use the web interface and those who use the desktop interface who feel the recorder and audio editor provided by Audio Project Manager is more than adequate.) But then if a Wasta user wanted to get this feature working, they have to uninstall the snap version and install the deb version. This is not at all intuitive.

In terms of letter users know. The desktop software itself lets users know there is an updated version and provides them a link to go get it. So to this point we have allowed desktop users to be in control and informed concerning which version they are using.

I won't go into the horror stories told by one developer who maintains a flatpack version for another product and how much time he has spent trying to prepare this additional package and get it approved for posting on the flatpack site.

We aren't sure whether we need the program to be discoverable by a wider community of Linux users. It does cost us for cloud storage although the cost isn't large. We currently have quite a large user base without posting on flatpack or snap stores.

[rik-shaw]

I think there is vested interest in sorting out how to best have a flatpak integrate with other flatpaks (even to the point of reading / even editing a separate flatpak). But how to do this is something I am not sure if possible. (I'll get to hosting on Flathub or other later).

As another example, I know that @gtryus (THANK YOU!) was /is the key developer behind Pathway, that no longer works (easily) with Linux, and even though I do understand there isn't capacity to keep maintaining Pathway now, it seems that a Pathway flatpak would best be able to remain usable for newer systems and give interaction with the Fieldworks flatpak. But it would take "poking (appropriate) holes" in the default flatpak security model. side note: I would be interested in trying to get Pathway to work in this fashion, maybe @n8marti could assist me in knowing where to start (flatpak noob here :-) I read even yesterday a thread where someone was requesting Pathway for Linux.

Back to APM / Audacity integration I could see how it would be a challenge to get into Flathub as it would be given r/w permission to Audacity config files (? I think I am understanding this correctly ?). My suggestion is that we may face this (or other blockers) with other SIL apps as well and it may be time to consider hosting our own flatpak repository. Flatpak by design can use multiple repositories, so the "de facto" Flathub AND a SIL repo could both be pre-included and configured in Wasta, for example. Of course non-Wasta users could (relatively trivially) add an additional flatpak repo to their system as well.

Snap doesn't allow multiple repositories to be (simultaneously) used, so this is one advantage of flatpak. Certainly traditional .debs are more "native", less problematic in that the sandboxing doesn't exist, and I for one am quite familiar with (simple) debian packaging, but as @n8marti points out it means with each new Linux release the libraries / runtimes, etc. / all need to be rev'd and even doing that is a burden that often leaves us stumbling each release to maintain functionality / avoid regressions .... not even considering "feature upgrades". I think with Bloom and Fieldworks going flatpak, and ParatextLite going snap the "ship has sailed" that SIL looking ahead will move away from .debs (even though yes I prefer them :-)

Now, if APM does keep active with their .deb builds then that still works of course. Getting Audacity from PPA can be done (find the link here): https://ubuntuhandbook.org/index.php/2023/04/ubuntu-ppa-for-audacity/ in order to get to 3.3.3 for 22.04 (or 3.0.3 for 20.04). I could even copy these packages to the "Wasta Apps PPA" so Wasta users would get it automatically.

[n8marti]

Earlier in the Audio Project Manager work, we created both snap and deb packages. It is actually fairly trivial to do and we could in fact create all three kinds of packages. All this is supported by electron-builder. The problem is the confusion this creates for the users. I could create both a deb version and a snap version. If Wasta preferred the installation of the snap version, it would work for all users who didn't care to use Audacity as their audio editor. (We have users who use the web interface and those who use the desktop interface who feel the recorder and audio editor provided by Audio Project Manager is more than adequate.) But then if a Wasta user wanted to get this feature working, they have to uninstall the snap version and install the deb version. This is not at all intuitive.

I would still vote for APM to eventually be released as either a snap or a flatpak in addition to a deb, with the package's description noting that the interoperability feature with Audacity (and any others) wouldn't function, at least not without manual intervention by the user. I think desktop apps on Linux-based systems are moving more and more this direction because of better security and broader compatibility (across OS releases and across OS distributions).

I won't go into the horror stories told by one developer who maintains a flatpack version for another product and how much time he has spent trying to prepare this additional package and get it approved for posting on the flatpack site.

Yes, approval in the flathub repository was not at all easy for the one package I have gotten hosted there, including being directed in some specific ways how it should be compiled during the build process. But, as @rik-shaw says, the flatpak format seems to hold the most promise for possible future inter-operability between packages over the snap format, especially if the decision is made to create an SIL-maintained flatpak repository, which would reduce a lot of package approval friction in exchange for narrower public visibility, just like we have with the SIL deb package repositories.

Taking the Audacity config file as an example, it looks like adding the sandbox permission --filesystem=~/audacity-data/audacity.cfg could accomplish r/w access to it, assuming that would also be approved by the repository gatekeepers. There is no facility for a similar override in snap packaging, because even if --classic confinement were considered, it seems unlikely that it would be approved just so that an APM snap could modify Audacity config. (Side note: I have the Audacity flatpak installed because the deb version is too old and missing a feature I need, and it looks like the flatpak is still using the ~/.audacity-data/audacity.cfg file for its config thanks to having the --filesystem=host permission.)