Flatpak package and Flathub publication

[k8ieone]

Really liking the app so far, it would be nice if the project was available as a Flatpak.

[taylorhadden]

Sounds like a good idea to me! I'm not really a linux user myself. Is this something you're knowledgeable on / would mind helping out with?

[k8ieone]

No, infortunately I haven't done anything like this before. But I'll try looking into it, my guess is that it shouldn't be too difficult.

[k8ieone]

Alright, I did some initial research and I'm trying to throw a manifest together. I have a question tho. Would you like to publish the finished application on Flathub yourself?

Flathub's policy is that users are allowed to submit applications that they're not the developer of. https://docs.flathub.org/docs/for-app-authors/submission#theres-an-app-that-id-like-to-see-on-flathub-but-im-not-the-developer

But it seems that having the developer publish the application is preferred. https://docs.flathub.org/docs/for-app-authors/submission#someone-else-has-put-my-app-on-flathubwhat-do-i-do

This is still in the future, the manifest is nowhere near ready yet. I just wanted to shoot this question out beforehand.

I'm hoping this might not even matter because once a repository in https://github.com/flathub has been created the publisher gets write permissions and I hope more people can be invited to also have write access. This way we could both manage the application on Flathub. https://docs.flathub.org/docs/for-app-authors/submission/#after-submission

[taylorhadden]

I have no strong preferences about being the primary publisher. If you or anybody else want to help out here, I'd be quite pleased! That's why Tangent is open source now.

[k8ieone]

I might have spoken too soon :sweat_smile:

Tangent being an Electron app using npm for builds complicates things because Flathub doesn't allow network access from the sandbox where the application is built. The workaround for this is to pull the dependencies before the build process starts and then use npm in offline mode for the build. They have tooling and a guide for this, but the Typewriter fork being a local module seems to break their script for dependency fetching. I'm afraid I'm not enough of an npm guru for this :sweat_smile:

In case anyone wants to pick this up, feel free to do so.

I'll post an update if I manage to get any further, but my hopes aren't so high anymore :pensive:

This is what I have so far:

id: io.github.suchnsuch.Tangent
runtime: org.freedesktop.Platform
runtime-version: '23.08'
sdk: org.freedesktop.Sdk
base: org.electronjs.Electron2.BaseApp
base-version: '23.08'
sdk-extensions:
  - org.freedesktop.Sdk.Extension.node18
# Electron doesn't use a traditional locale format
separate-locales: false
writable-sdk: true
command: run.sh
finish-args:
  - --device=dri
  - --share=ipc
  - --socket=x11
  - --socket=pulseaudio
  # Probably not needed
  #- --share=network
  - --filesystem=home
modules:
  - name: tangent
    buildsystem: simple
    build-options:
      append-path: /usr/lib/sdk/node18/bin
      env:
        NPM_CONFIG_LOGLEVEL: info
        XDG_CACHE_HOME: /run/build/tangent/flatpak-node/cache
        npm_config_cache: /run/build/tangent/flatpak-node/npm-cache
        npm_config_nodedir: /usr/lib/sdk/node18
        npm_config_offline: 'true'
    subdir: main
    sources:
      - type: git
        url: https://github.com/suchnsuch/Tangent
        tag: tangent-v0.7.6
      - type: script
        dest-filename: run.sh
        commands:
          - zypak-wrapper.sh /app/main/electron-sample-app "$@"
      #- type: file
      #  path: io.github.suchnsuch.Tangent.desktop
      #- type: file
      #  path: io.github.suchnsuch.Tangent.svg
      #- type: file
      #  path: io.github.suchnsuch.Tangent.metainfo.xml
    build-commands:
      # build linux
      - npm install --offline
      # install
      #- install -Dm644 -t /app/share/icons/hicolor/scalable/apps/ io.github.suchnsuch.Tangent.svg
      #- install -Dm644 -t /app/share/applications io.github.suchnsuch.Tangent.desktop
      #- install -Dm644 -t /app/share/metainfo io.github.suchnsuch.Tangent.metainfo.xml
      #- install -Dm755 -t /app/bin/ run.sh

[taylorhadden]

Does recursive mode help at all? What kind of errors are you getting?

[k8ieone]

Initially I had an issue with the Typewriter fork, but that was my fault. I didn't realize it was a submodule and that it isn't cloned by default.

Now I cloned the Tangent repo at version 0.7.6 including the submodules (git submodule update --init --recursive) and ran flatpak-node-generator npm package-lock.json.

I'm getting

Reading packages from lockfiles...
Traceback (most recent call last):
  File "/home/k8ie/.local/bin/flatpak-node-generator", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/main.py", line 277, in main
    asyncio.run(_async_main())
  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/main.py", line 198, in _async_main
    packages.update(lockfile_provider.process_lockfile(lockfile))
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/providers/npm.py", line 166, in process_lockfile
    yield from self._process_packages_v2(lockfile, data)
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/providers/npm.py", line 138, in _process_packages_v2
    raise NotImplementedError(
NotImplementedError: Don't know how to handle package packages/tangent-test-workspace-generator in package-lock.json

[k8ieone]

I think this could be a forgotten package in the package-lock.json file. I removed the section mentioning packages/tangent-test-workspace-generator and got a bit further now:

Reading packages from lockfiles...
1070 packages read.
Generating packages [1069/1070] app-builder-bin @ 4.0.0                                                                                                                                                                                       
Traceback (most recent call last):
  File "/home/k8ie/.local/bin/flatpak-node-generator", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/main.py", line 277, in main
    asyncio.run(_async_main())
  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/main.py", line 229, in _async_main
    await progress.run()
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/progress.py", line 78, in run
    await coro
  File "/usr/lib/python3.12/asyncio/tasks.py", line 631, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/progress.py", line 68, in _generate
    await self.module_provider.generate_package(package)
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/providers/npm.py", line 365, in generate_package
    await self.special_source_provider.generate_special_sources(package)
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/providers/special.py", line 482, in generate_special_sources
    await self._handle_playwright(package)
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/providers/special.py", line 373, in _handle_playwright
    metadata = await RemoteUrlMetadata.get(dl_url, cachable=True)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/url_metadata.py", line 41, in get
    async for part in Requests.instance.read_parts(url, cachable=False):
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/requests.py", line 57, in read_parts
    async for part in self._read_parts(url, size):
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/requests.py", line 30, in _read_parts
    async with self._open_stream(url) as stream:
  File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/requests.py", line 24, in _open_stream
    async with session.get(url) as response:
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/aiohttp/client.py", line 1197, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/aiohttp/client.py", line 696, in _request
    resp.raise_for_status()
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/aiohttp/client_reqrep.py", line 1070, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 404, message='The specified blob does not exist.', url=URL('https://playwright.azureedge.net/builds/firefox/1440/firefox-ubuntu-18.04.zip')

[k8ieone]

I tried looking around, but I don't know where that URL comes from.

[taylorhadden]

That looks like it's trying to pull binaries for the development dependencies. Playwright is used for automated testing.

tangent-test-workspace-generator is just a different tool in the mono-repo. Hopefully you should be able to ignore it?

After doing some poking around, you might be hitting this issue?

[k8ieone]

Yes, that seems to be it exactly. Nice find.

Regarding tangent-test-workspace-generator, I think it was moved at some point to a different directory but not updated in the package-lock.json file. If you take a look at that error message, it was looking for it in packages/tangent-test-workspace-generator, but in the repo it's located in the apps folder.

[taylorhadden]

Ah! I should probably do a package update to refresh package-lock.json.

[k8ieone]

If you do so, could you please backport it to the tangent-v0.7.6 tag? Or perhaps create a new one for that like tangent-v0.7.7.

[taylorhadden]

Hmm.. would it be kosher to move ahead with the v0.8.x version as a test case and release on Flathub when v0.8 is ready?

[k8ieone]

They only have a policy that says no unstable releases, so I'd say yes. We can work with 0.8.x and release the package along with 0.8.

[k8ieone]

I was kinda hoping they would have fixed that error in flatpak-node-generator by now, but they didn't. So I'm running a modified copy that includes the fix.

After manually removing the missing packages/tangent-test-workspace-generator package, the command passed and generated the file needed for the Flatpak builder.

[k8ieone]

Alright, I've run into another thing.

Every time I try to build the Flatpak, I get to bwrap: Can't chdir to /run/build/tangent/main: No such file or directory after checking out the submodule.

I tried building https://github.com/flathub/electron-sample-app as well, but I get the same result.

I've also tried to build the Flatpak in a privileged Docker container to make sure this isn't an issue specific to Arch or my system install.

The only related issue I was able to find is https://github.com/flathub/com.visualstudio.code-oss/issues/78, but that's pretty old and was solved by using a newer flatpak-builder version. I'm already using the latest one. I've also tried using the flatpak-builder Flatpak instead of the Arch package, but the result is the same.

[k8ieone]

Nevermind, that issue was caused by the subdir: main statement in the Flatpak manifest. Removing it gets me a little further.

[k8ieone]

I have great news, I was finally able to build and run Tangent in a Flatpak!

Now it's just a matter of going through the Flathub guidelines and making sure it passes them. I'll post a link to the Flatpak packaging repo here once it's at least somewhat ready.

[k8ieone]

I was also able to test Tangent on a mobile phone. I couldn't before because it runs Alpine and Alpine doesn't exactly see eye-to-eye with AppImages. Likely because it uses musl.

Now with the Flatpak it works flawlessly. Tangent even scales down pretty nicely. The note takes up a lot of horizontal space, forcing the user to scroll. Works great in landscape mode tho. Just thought I'd share.

[taylorhadden]

This is awesome news! Thank you for all your hard work!

Using it with a phone is pretty interesting. Try turning down the "min panel width" in the settings. That should help!

[k8ieone]

I did have to make some changes in this repo as well, I'll open a PR for those.

[k8ieone]

@taylorhadden Here are the Flatpak sources in their current state, I recommend looking at the Metainfo file, it contains most of the metadata that will be visible on Flathub and in stores.

I wasn't able to find a .desktop file in the main repo, so I made one as well.

The .desktop file and Metainfo are the two main things where I could have made a mistake, I encourage you to check both and let me know it you don't like something or if you'd like something added.

https://github.com/k8ieone/flathub/tree/new-pr

[k8ieone]

Here's a preview of how the metadata looks when rendered.

[taylorhadden]

This is super cool! I'll take a look at the description.

[k8ieone]

@taylorhadden I have one issue to report and that's that the spell-checking doesn't seem to work.

Does Tangent use some kind of system-wide dictionary that could be missing in the Flatpak? I know some programs use the files in /usr/share/dict for example.

[taylorhadden]

Huh, that's odd. It just uses the default Chromium spellcheck system.

[k8ieone]

Alright, it turns out the spell-checking system automatically downloads the language files needed. This means it needs internet access.

When I initially put the manifest together I disabled networking because I thought an editor probably doesn't need it. Well, I was wrong :sweat_smile:

I'll update the manifest to include network access. After doing this spell-checking works again.