Flatpak Redistributable

[o-zig]

This issue is for my own documentation while working on a flatpak redistributable for the upstream issue#175.

[o-zig]

Based on Flatpak's basic concepts documentation, the steps needed are:

• Find a runtime that hopefully bundles all necessary libraries (only openjdk-jre I would think)
• Configure the sandbox with necessary exports (e.g. Icon and .desktop file)
• Configure the sandbox with necessary portals (probably anything for X11/Wayland and exporting/importing backups)
• Setup a repository from which new versions could be distributed and or export the package onto Flathub

[o-zig]

Basic identifier structure is documented as: name/architecture/branch

Not sure, but the name of the identifier should probably be: ua.syt0r.kanji or ua.syt0r.kanji-dojo Archticture depends a bit on the runtimes, as the JRE is probably the only thing architecture specific. Probably x86-64 and maybe some ARM-archtictures we should support. Looking at previous releases, we probably only need a stable branch.

[o-zig]

Managed to make a build manifest that manages to build a Flatpak. The outstanding issues:

• Permissions for the sandbox must be configured
• Make the paths for the Kanji Dojo jar version and architecture independent
• Actually test that the application runs as expected

Currently the manifest also assumes the build jar exists already rather than building the repository from source. Not sure if this will be an issue for Flathub, but I also have yet to read their policies in detail.

[o-zig]

Also was trying to look up another Flatpak-Package using Java, and found the manifest for org.freeplane.App useful.

[o-zig]

To build the Flatpak: flatpak-builder --verbose --force-clean --user --install-deps-from=flathub --repo=repo --install builddir ua.syt0r.kanji-dojo.yml

To run the built Flatpak: flatpak run ua.syt0r.kanji-dojo

[o-zig]

Currently running the Flatpak causes an initial stacktrace and no databases seem to function, hence making the app practically useless. The stacktrace:

$ flatpak run ua.syt0r.kanji-dojo

Note that the directories 

'/var/lib/flatpak/exports/share'
'/home/anon/.local/share/flatpak/exports/share'

are not in the search path set by the XDG_DATA_DIRS environment variable, so
applications installed by Flatpak may not appear on your desktop until the
session is restarted.

_amdgpu_device_initialize: amdgpu_get_auth (1) failed (-1)
amdgpu: amdgpu_device_initialize failed.
glx: failed to create dri3 screen
failed to load driver: radeonsi
_amdgpu_device_initialize: amdgpu_get_auth (1) failed (-1)
amdgpu: amdgpu_device_initialize failed.
glx: failed to create dri3 screen
failed to load driver: radeonsi
[SKIKO] warn: Fallback to next API
org.jetbrains.skiko.RenderException: Cannot create Linux GL context
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Analytics Screen: general_dashboard
Exception in thread "DefaultDispatcher-worker-3" java.sql.SQLException: Error opening connection
    at org.sqlite.SQLiteConnection.open(SQLiteConnection.java:283)
    at org.sqlite.SQLiteConnection.<init>(SQLiteConnection.java:67)
    at org.sqlite.jdbc3.JDBC3Connection.<init>(JDBC3Connection.java:28)
    at org.sqlite.jdbc4.JDBC4Connection.<init>(JDBC4Connection.java:19)
    at org.sqlite.JDBC.createConnection(JDBC.java:106)
    at org.sqlite.JDBC.connect(JDBC.java:79)
    at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:681)
    at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:190)
    at app.cash.sqldelight.driver.jdbc.sqlite.ThreadedConnectionManager.getConnection(JdbcSqliteDriver.kt:122)
    at app.cash.sqldelight.driver.jdbc.sqlite.JdbcSqliteDriver.getConnection(JdbcSqliteDriver.kt)
    at app.cash.sqldelight.driver.jdbc.JdbcDriver.connectionAndClose(JdbcDriver.kt:119)
    at app.cash.sqldelight.driver.jdbc.JdbcDriver.execute(JdbcDriver.kt:130)
    at app.cash.sqldelight.db.SqlDriver$DefaultImpls.execute$default(SqlDriver.kt:63)
    at ua.syt0r.kanji.core.user_data.db.core.UserDataDatabaseImpl$Schema.create-0iQ1-z0(UserDataDatabaseImpl.kt:30)
    at ua.syt0r.kanji.core.user_data.db.core.UserDataDatabaseImpl$Schema.create(UserDataDatabaseImpl.kt:25)
    at ua.syt0r.kanji.core.user_data.JvmUserDataDatabaseManager.createDatabaseConnection(JvmUserDataDatabaseManager.kt:26)
    at ua.syt0r.kanji.core.user_data.practice.db.BaseUserDataDatabaseManager$createDeferredDatabaseConnection$1.invokeSuspend(UserDataDatabaseManager.kt:108)
    at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
    at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:111)
    at kotlinx.coroutines.scheduling.TaskImpl.run(Tasks.kt:99)
    at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:585)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:802)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:706)
    at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:693)
    Suppressed: kotlinx.coroutines.internal.DiagnosticCoroutineContextException: [StandaloneCoroutine{Cancelling}@6f7c48ad, Dispatchers.Unconfined]
Caused by: org.sqlite.NativeLibraryNotFoundException: No native library found for os.name=Linux, os.arch=x86_64, paths=[/org/sqlite/native/Linux/x86_64:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib]
    at org.sqlite.SQLiteJDBCLoader.loadSQLiteNativeLibrary(SQLiteJDBCLoader.java:361)
    at org.sqlite.SQLiteJDBCLoader.initialize(SQLiteJDBCLoader.java:75)
    at org.sqlite.core.NativeDB.load(NativeDB.java:69)
    at org.sqlite.SQLiteConnection.open(SQLiteConnection.java:280)
    ... 24 more

[o-zig]

Just of note, Wayland is not supported, trying to use fallback-x11 fails (at least in a Wayland desktop).

[o-zig]

Two extremely useful things for debugging:

1. Open a shell in the installed Flatpak: flatpak run --command=bash --devel --filesystem=$(pwd) ua.syt0r.kanji-dojo
2. Open a shell in the non-installed Flatpak: flatpak-builder --run builddir ua.syt0r.kanji-dojo.yml bash

The above mentioned stack-trace only occurs in the installed version and reading the exception a bit closer there seem to be libraries missing which are available in the development Flatpak.

[o-zig]

Something is odd, according to the flathub forum a symlink is needed for libsqlite3.so.0. However the source path where it links to both exists and according to the stacktrace is searched for the necessary libs, making the symlink unnecessary.

Maybe it wants a different version or expects a different link. Maybe it is looking for a differently named library alltogether.

[o-zig]

Little difficult to build and debug kanji-dojo in such a specific environment (at least I don't know how to quickly). Hence looking at the source code where the exception is thrown.

[o-zig]

Found the likely cause: the sqlitejdbc library does not seem to be available. Possibly the fastest way to include it would be to grab the jar from the latest release.

For publishing to flathub the jar of kanji-dojo should probably also be pulled from the latest release, but that's something for after the flatpak works.

[o-zig]

So including sqlite-jdbc via java -classpath ".:${PATH_TO_JAR}/sqlite-jdbc-3.46.1.3.jar" did not work. Similarly, trying to place the library in places it claims to be searching doesn't help. The SQLite JDBC Driver README makes me thing that a native sqlitejdbc.so needs to be packaged, but it doesn't exist on my normal system or in the non-installed Flatpak.

EDIT: The sqlitejdbc.so and other binaries should be delivered within the jar file.

[o-zig]

A lot of frustrating debugging later, some directory in /tmp must be writable in order for the application to function. I found this by building an extremely bwrap sandbox which showed the same issues and adding permissions till it functions. To my knowledge, flatpaks also utilized Bubblewrap for sandboxing.

[o-zig]

I found the two files are generated when starting the application:

• sqlite-3.44.1.0-b4556498-0bd4-4339-98e3-f9f04206654a-libsqlitejdbc.so
• sqlite-3.44.1.0-b4556498-0bd4-4339-98e3-f9f04206654a-libsqlitejdbc.so.lck

The integrated hash seems to also change with each run, so I guess write permissions on /tmp are the solution. This is deferring the permissions to the base system.

[o-zig]

Probably my local config does not set up xdg-desktop-portal properly, hence no (sensible) directories are available to export/import backups to/from.

Otherwise, it's a little questionable what portals should be offered. desktop seems sensible, but downloads or documents? The whole home directory is, I assume, unlikely to be accepted on Flathub.

[o-zig]

Things I'd still do before opening a PR:

• [x] Within the Flatpak manifest either build the jar, or fetch it from the latest Github releases
• [x] Set sensible permissions to allow importing/exporting backups
• [x] Copy, correct and integrate the .desktop file from the Snap
• [ ] Make the Flatpaks /tmp writable rather than mounting and using the hosts /tmp (Does not seem possible)
• [x] Read Flathub's policies, adjust manifest as necessary
• [ ] Integrate building the Flatpak into Github actions

[o-zig]

A summary of all the useful commands assuming you are in ./Kanji-Dojo/flatpak:

# Build and install Flatpak:
flatpak-builder --verbose --force-clean --user --install-deps-from=flathub --repo=repo --install builddir io.github.syt0r.kanji-dojo.yml
# Run Flatpak
flatpak run io.github.syt0r.kanji-dojo
# Run a shell within the Flatpak
flatpak run --command=bash --devel --filesystem=$(pwd) io.github.syt0r.kanji-dojo
# Lint the manifest (necessary for Flathub submission)
flatpak run --command=flatpak-builder-lint org.flatpak.Builder manifest io.github.syt0r.kanji-dojo
# Lint the generated Flatpak
flatpak run --command=flatpak-builder-lint org.flatpak.Builder repo repo

[o-zig]

One of the thing necessary is a OARS rating. Trying to answer the questions to generate a rating I did my best. I think most of the questions relate to graphic depictions which I would not consider kanji to be. However, seeing as the app does touch vocabulary and kanji with vulgar, violent, extreme and sexual meanings, I at least flagged mild amounts of profanity and discrimination.

This is something that should be critically considered in the pull-request. Another refrence I've been using is Kiten where they do not flag any content, though I honestly don't know how much thought/intention was put behind that as both Kiten and Kanji Dojo are intended as educational tools which teach a language that is capable of expressing such themes.

[o-zig]

Things for a pull-request:

• Integrate building the Flatpak into Github actions
• Test the application thoroughly (maybe also define tests for the future)
• Discuss OARS rating

Things for the future (suggested priorities):

1. Capture proper desktop screenshots (See Flathub quality guidelines)
2. Maybe update current screenshots to reflect SRS changes and the addition of vocabulary lists
3. Maybe update descriptions to also mention vocabulary lists
4. Submit pull-request on Flathub
5. Unify Linux-redistributables as much as is possible/sensible
6. Maybe script making a new Flatpak release