suchnsuch / Tangent

The publicly-available modules of the Tangent project.
141 stars 8 forks source link

Flatpak package and Flathub publication #86

Open k8ieone opened 4 months ago

k8ieone commented 4 months ago

Really liking the app so far, it would be nice if the project was available as a Flatpak.

taylorhadden commented 4 months ago

Sounds like a good idea to me! I'm not really a linux user myself. Is this something you're knowledgeable on / would mind helping out with?

k8ieone commented 4 months ago

No, infortunately I haven't done anything like this before. But I'll try looking into it, my guess is that it shouldn't be too difficult.

k8ieone commented 4 months ago

Alright, I did some initial research and I'm trying to throw a manifest together. I have a question tho. Would you like to publish the finished application on Flathub yourself?

Flathub's policy is that users are allowed to submit applications that they're not the developer of. https://docs.flathub.org/docs/for-app-authors/submission#theres-an-app-that-id-like-to-see-on-flathub-but-im-not-the-developer

But it seems that having the developer publish the application is preferred. https://docs.flathub.org/docs/for-app-authors/submission#someone-else-has-put-my-app-on-flathubwhat-do-i-do

This is still in the future, the manifest is nowhere near ready yet. I just wanted to shoot this question out beforehand.

I'm hoping this might not even matter because once a repository in https://github.com/flathub has been created the publisher gets write permissions and I hope more people can be invited to also have write access. This way we could both manage the application on Flathub. https://docs.flathub.org/docs/for-app-authors/submission/#after-submission

taylorhadden commented 4 months ago

I have no strong preferences about being the primary publisher. If you or anybody else want to help out here, I'd be quite pleased! That's why Tangent is open source now.

k8ieone commented 4 months ago

I might have spoken too soon :sweat_smile:

Tangent being an Electron app using npm for builds complicates things because Flathub doesn't allow network access from the sandbox where the application is built. The workaround for this is to pull the dependencies before the build process starts and then use npm in offline mode for the build. They have tooling and a guide for this, but the Typewriter fork being a local module seems to break their script for dependency fetching. I'm afraid I'm not enough of an npm guru for this :sweat_smile:

In case anyone wants to pick this up, feel free to do so.

I'll post an update if I manage to get any further, but my hopes aren't so high anymore :pensive:

This is what I have so far:

id: io.github.suchnsuch.Tangent
runtime: org.freedesktop.Platform
runtime-version: '23.08'
sdk: org.freedesktop.Sdk
base: org.electronjs.Electron2.BaseApp
base-version: '23.08'
sdk-extensions:
  - org.freedesktop.Sdk.Extension.node18
# Electron doesn't use a traditional locale format
separate-locales: false
writable-sdk: true
command: run.sh
finish-args:
  - --device=dri
  - --share=ipc
  - --socket=x11
  - --socket=pulseaudio
  # Probably not needed
  #- --share=network
  - --filesystem=home
modules:
  - name: tangent
    buildsystem: simple
    build-options:
      append-path: /usr/lib/sdk/node18/bin
      env:
        NPM_CONFIG_LOGLEVEL: info
        XDG_CACHE_HOME: /run/build/tangent/flatpak-node/cache
        npm_config_cache: /run/build/tangent/flatpak-node/npm-cache
        npm_config_nodedir: /usr/lib/sdk/node18
        npm_config_offline: 'true'
    subdir: main
    sources:
      - type: git
        url: https://github.com/suchnsuch/Tangent
        tag: tangent-v0.7.6
      - type: script
        dest-filename: run.sh
        commands:
          - zypak-wrapper.sh /app/main/electron-sample-app "$@"
      #- type: file
      #  path: io.github.suchnsuch.Tangent.desktop
      #- type: file
      #  path: io.github.suchnsuch.Tangent.svg
      #- type: file
      #  path: io.github.suchnsuch.Tangent.metainfo.xml
    build-commands:
      # build linux
      - npm install --offline
      # install
      #- install -Dm644 -t /app/share/icons/hicolor/scalable/apps/ io.github.suchnsuch.Tangent.svg
      #- install -Dm644 -t /app/share/applications io.github.suchnsuch.Tangent.desktop
      #- install -Dm644 -t /app/share/metainfo io.github.suchnsuch.Tangent.metainfo.xml
      #- install -Dm755 -t /app/bin/ run.sh
taylorhadden commented 4 months ago

Does recursive mode help at all? What kind of errors are you getting?

k8ieone commented 3 months ago

Initially I had an issue with the Typewriter fork, but that was my fault. I didn't realize it was a submodule and that it isn't cloned by default.

Now I cloned the Tangent repo at version 0.7.6 including the submodules (git submodule update --init --recursive) and ran flatpak-node-generator npm package-lock.json.

I'm getting

Reading packages from lockfiles...
Traceback (most recent call last):
  File "/home/k8ie/.local/bin/flatpak-node-generator", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/main.py", line 277, in main
    asyncio.run(_async_main())
  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/main.py", line 198, in _async_main
    packages.update(lockfile_provider.process_lockfile(lockfile))
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/providers/npm.py", line 166, in process_lockfile
    yield from self._process_packages_v2(lockfile, data)
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/providers/npm.py", line 138, in _process_packages_v2
    raise NotImplementedError(
NotImplementedError: Don't know how to handle package packages/tangent-test-workspace-generator in package-lock.json
k8ieone commented 3 months ago

I think this could be a forgotten package in the package-lock.json file. I removed the section mentioning packages/tangent-test-workspace-generator and got a bit further now:

Reading packages from lockfiles...
1070 packages read.
Generating packages [1069/1070] app-builder-bin @ 4.0.0                                                                                                                                                                                       
Traceback (most recent call last):
  File "/home/k8ie/.local/bin/flatpak-node-generator", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/main.py", line 277, in main
    asyncio.run(_async_main())
  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/main.py", line 229, in _async_main
    await progress.run()
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/progress.py", line 78, in run
    await coro
  File "/usr/lib/python3.12/asyncio/tasks.py", line 631, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/progress.py", line 68, in _generate
    await self.module_provider.generate_package(package)
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/providers/npm.py", line 365, in generate_package
    await self.special_source_provider.generate_special_sources(package)
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/providers/special.py", line 482, in generate_special_sources
    await self._handle_playwright(package)
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/providers/special.py", line 373, in _handle_playwright
    metadata = await RemoteUrlMetadata.get(dl_url, cachable=True)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/url_metadata.py", line 41, in get
    async for part in Requests.instance.read_parts(url, cachable=False):
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/requests.py", line 57, in read_parts
    async for part in self._read_parts(url, size):
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/requests.py", line 30, in _read_parts
    async with self._open_stream(url) as stream:
  File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/flatpak_node_generator/requests.py", line 24, in _open_stream
    async with session.get(url) as response:
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/aiohttp/client.py", line 1197, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/aiohttp/client.py", line 696, in _request
    resp.raise_for_status()
  File "/home/k8ie/.local/share/pipx/venvs/flatpak-node-generator/lib/python3.12/site-packages/aiohttp/client_reqrep.py", line 1070, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 404, message='The specified blob does not exist.', url=URL('https://playwright.azureedge.net/builds/firefox/1440/firefox-ubuntu-18.04.zip')
k8ieone commented 3 months ago

I tried looking around, but I don't know where that URL comes from.

taylorhadden commented 3 months ago

That looks like it's trying to pull binaries for the development dependencies. Playwright is used for automated testing.

tangent-test-workspace-generator is just a different tool in the mono-repo. Hopefully you should be able to ignore it?

After doing some poking around, you might be hitting this issue?

k8ieone commented 3 months ago

Yes, that seems to be it exactly. Nice find.

Regarding tangent-test-workspace-generator, I think it was moved at some point to a different directory but not updated in the package-lock.json file. If you take a look at that error message, it was looking for it in packages/tangent-test-workspace-generator, but in the repo it's located in the apps folder.

taylorhadden commented 3 months ago

Ah! I should probably do a package update to refresh package-lock.json.

k8ieone commented 3 months ago

If you do so, could you please backport it to the tangent-v0.7.6 tag? Or perhaps create a new one for that like tangent-v0.7.7.

taylorhadden commented 3 months ago

Hmm.. would it be kosher to move ahead with the v0.8.x version as a test case and release on Flathub when v0.8 is ready?

k8ieone commented 3 months ago

They only have a policy that says no unstable releases, so I'd say yes. We can work with 0.8.x and release the package along with 0.8.

k8ieone commented 3 months ago

I was kinda hoping they would have fixed that error in flatpak-node-generator by now, but they didn't. So I'm running a modified copy that includes the fix.

After manually removing the missing packages/tangent-test-workspace-generator package, the command passed and generated the file needed for the Flatpak builder.

k8ieone commented 3 months ago

Alright, I've run into another thing.

Every time I try to build the Flatpak, I get to bwrap: Can't chdir to /run/build/tangent/main: No such file or directory after checking out the submodule.

I tried building https://github.com/flathub/electron-sample-app as well, but I get the same result.

I've also tried to build the Flatpak in a privileged Docker container to make sure this isn't an issue specific to Arch or my system install.

The only related issue I was able to find is https://github.com/flathub/com.visualstudio.code-oss/issues/78, but that's pretty old and was solved by using a newer flatpak-builder version. I'm already using the latest one. I've also tried using the flatpak-builder Flatpak instead of the Arch package, but the result is the same.

k8ieone commented 3 months ago

Nevermind, that issue was caused by the subdir: main statement in the Flatpak manifest. Removing it gets me a little further.

k8ieone commented 3 months ago

I have great news, I was finally able to build and run Tangent in a Flatpak!

Now it's just a matter of going through the Flathub guidelines and making sure it passes them. I'll post a link to the Flatpak packaging repo here once it's at least somewhat ready.

k8ieone commented 3 months ago

I was also able to test Tangent on a mobile phone. I couldn't before because it runs Alpine and Alpine doesn't exactly see eye-to-eye with AppImages. Likely because it uses musl.

Now with the Flatpak it works flawlessly. Tangent even scales down pretty nicely. The note takes up a lot of horizontal space, forcing the user to scroll. Works great in landscape mode tho. Just thought I'd share.

taylorhadden commented 3 months ago

This is awesome news! Thank you for all your hard work!

Using it with a phone is pretty interesting. Try turning down the "min panel width" in the settings. That should help!

k8ieone commented 3 months ago

I did have to make some changes in this repo as well, I'll open a PR for those.

k8ieone commented 3 months ago

@taylorhadden Here are the Flatpak sources in their current state, I recommend looking at the Metainfo file, it contains most of the metadata that will be visible on Flathub and in stores.

I wasn't able to find a .desktop file in the main repo, so I made one as well.

The .desktop file and Metainfo are the two main things where I could have made a mistake, I encourage you to check both and let me know it you don't like something or if you'd like something added.

https://github.com/k8ieone/flathub/tree/new-pr

k8ieone commented 3 months ago

Here's a preview of how the metadata looks when rendered.

image

image

taylorhadden commented 2 months ago

This is super cool! I'll take a look at the description.

k8ieone commented 2 months ago

@taylorhadden I have one issue to report and that's that the spell-checking doesn't seem to work.

Does Tangent use some kind of system-wide dictionary that could be missing in the Flatpak? I know some programs use the files in /usr/share/dict for example.

taylorhadden commented 2 months ago

Huh, that's odd. It just uses the default Chromium spellcheck system.

k8ieone commented 2 months ago

Alright, it turns out the spell-checking system automatically downloads the language files needed. This means it needs internet access.

When I initially put the manifest together I disabled networking because I thought an editor probably doesn't need it. Well, I was wrong :sweat_smile:

I'll update the manifest to include network access. After doing this spell-checking works again.

taylorhadden commented 2 months ago

Hmm, we probably also want some flag to disable the built-in auto-update behavior, right? Are you seeing any issues with that?

k8ieone commented 2 months ago

I thought that this behavior was controlled by the APPIMAGE env variable.

I see this during startup:

APPIMAGE env is not defined, current application is not an AppImage

I'm not sure how the app behaves when it tries to update itself. I haven't seen that happen yet.

But yes, it would make sense to disable auto-updates when running in Flatpak.

k8ieone commented 1 month ago

@taylorhadden Do you have a plan for the next stable release? If you have a date in mind, keep me posted please. It's possible some new issue may pop up with the new release and I'd like to be ready for it.

k8ieone commented 1 month ago

Nevermind, I just checked the releases page :smile:

I'll try to get to finishing up the publication as soon as I can.

k8ieone commented 3 weeks ago

Flathub PR open! :arrow_up:

k8ieone commented 3 weeks ago

@taylorhadden Quick update in case you're not following the PR:

They had an issue with the screenshots being in the flatpak repo and not somewhere upstream. I'll fix that by using the screenshots from the website.

Second more serious issue is that they would like the .desktop and metainfo files to also be upstream. I'll try getting a PR ready when I get to it. Is there a good place for these files in the project? I don't want to break your existing packaging by adding them. We'll also need to do another release after these are added, or we can wait until you release 0.9.0. I'll leave that up to you, although I'm a bit afraid if the changes take too long the PR will fall into a pile of TODO PRs. I was a bit shocked when I saw the number of open PRs on Flathub.

taylorhadden commented 3 weeks ago

Thanks for pushing this through! Those files can live in the "build" folder of Tangent. Maybe in "build/flatpak"?

There are some bug fixes that want to get into a 0.8.2, so no worries there.

k8ieone commented 2 weeks ago

Thanks for the merge. Do you think you could re-tag the last commit? Either that or create another tag. I feel like we're really close here :crossed_fingers:

taylorhadden commented 2 weeks ago

Done!

k8ieone commented 1 week ago

Seems like we're blocked until Electron is able to fix this https://github.com/electron/electron/issues/43819

Either we wait for this to get resolved or we use an older version.