Flatpak version - Githubissues

Anx450z commented 3 months ago

What feature or new tool do you think should be added to DevToys?

Flatpak version of the app.

Why do you think this is needed?

Easy installation on Linux distros with more compatibility.

Solution/Idea

No response

Comments

No response

dylanmtaylor commented 3 months ago

I'd love to see this as well. There are distros out there like Fedora Silverblue where Flatpaks are the preferred method of app delivery.

JoelNichols commented 3 months ago

Agreed. Alternatively, an AppImage would be fine too.

0verEngineer commented 3 months ago

This way all Linux distros would be able to easily use this software

badcel commented 3 months ago

Flatpak is not a bad idea as the GirCore binaries are build to match the flatpak binaries. This avoids missing dependencies and libraries which are named differently than the flatpak ones.

veler commented 2 months ago

Hi there, Is anyone familiar with how to build a flatpak manifest? I'm reading the documentation and I'm getting quite confused by a few things. For your information, I'm quite a beginner at Linux. I learn it slowly while developing DevToys for it. I'm more of a Windows person.

If I base myself on the DEB package, I have a devtoys.desktop file like this:

[Desktop Entry]
Name=DevToys
Comment=A Swiss Army knife for developers.
GenericName=DevToys
Exec=/usr/bin/DevToys
Icon=/opt/devtoys/devtoys/Icon-Windows-Linux.png
Type=Application
StartupNotify=false
Categories=Development;Utility;Office
Keywords=devtoys

DevToys should be installed in /usr/bin/DevToys. The icon should be in /opt/devtoys/devtoys/Icon-Windows-Linux.png. DevToys also has dependencies on libwebkitgtk-6.0-4 and libadwaita-1-0.

Let's say the built binaries of DevToys, ready to be packed, are in /devtoys-bin on my local machine. What does the flatpak manifest should look like?

I'm not finding a clear answer on how to include the 2 dependencies I mentioned above. I'm also confused by the build-commands section in the manifest and what I should put there in my case. Can anyone please help me with it? :-)

Much appreciated!

badcel commented 2 months ago

I have not done it myself but there are already dotnet applications being distributed via flatpak:

Pinta has no GirCore release yet but the project is pretty well structured.
All nickvison apps (e.g. Denaro) are based on GirCore.

I would recommend comparing both ways to understand things (I have not done this myself yet). But i think it should get you started.

Additionally one day I found this on github: https://github.com/flathub/org.freedesktop.Sdk.Extension.dotnet8

I did not check it out in detail but perhaps it helps.

Im regard to libadwaita and webkitgtk: if you target the gnome runtime sdk in its current release 46 (this is what GirCore 0.5.0 is build for) it should just be available. See for example the Epiphany flatpak the GNOME web browser which uses both libraries.

badcel commented 2 months ago

One more thing: I don't know how it will work with extensions. There would probably be rights needed to access the home directory but if this is sufficient I don't know.

hockdudu commented 2 months ago

@veler You can take a look at https://docs.flatpak.org/en/latest/conventions.html. Flatpak expects a few conventions.

The Flatpak manifest doesn't only define where to install the application, but also how to build it. It should contain build instructions for the application.

Dependencies are also usually built from source in the manifest. But maybe they're on the GNOME runtime already? https://docs.flatpak.org/en/latest/available-runtimes.html

@badcel Thanks for the links. I have experimented a bit with Flatpak in the past, but didn't get to the point of publishing anything.

I have zero experience with .NET, that's what hinders me a bit here...

veler commented 2 months ago

@veler You can take a look at https://docs.flatpak.org/en/latest/conventions.html. Flatpak expects a few conventions.

The Flatpak manifest doesn't only define where to install the application, but also how to build it. It should contain build instructions for the application.

Dependencies are also usually built from source in the manifest. But maybe they're on the GNOME runtime already? https://docs.flatpak.org/en/latest/available-runtimes.html

@badcel Thanks for the links. I have experimented a bit with Flatpak in the past, but didn't get to the point of publishing anything.

I have zero experience with .NET, that's what hinders me a bit here...

Thanks @hockdudu . I will read this later today. Maybe it's already explained there but I will ask in case: what if the app is already built? I basically have all the binaries in a folder, ready to be packed in zip, Deb...etc

hockdudu commented 2 months ago

@veler You could use pre-built binaries, but AFAIK Flathub only allows binaries in exceptional circumstances. That's for auditing / safety reasons. By using sources, you can be sure the package doesn't do anything it shouldn't. With binaries, on the other hand, someone could have tampered it and added additional stuff (malware and the like).

See https://docs.flathub.org/docs/for-app-authors/requirements#best-practices

veler commented 2 months ago

@veler You could use pre-built binaries, but AFAIK Flathub only allows binaries in exceptional circumstances. That's for auditing / safety reasons. By using sources, you can be sure the package doesn't do anything it shouldn't. With binaries, on the other hand, someone could have tampered it and added additional stuff (malware and the like).

See https://docs.flathub.org/docs/for-app-authors/requirements#best-practices

Just to make it clear : are we talking about building the source code on the user machine, or in my CI ?

If the first, that sounds horrible. To run, DevToys had no dependencies needed outside of the 2 I mentioned earlier. But to build, it needs NodeJS,.NET, a total of 3 repositories... That will quickly cluter the user's machine.

hockdudu commented 2 months ago

Neither ;)

If submitted to Flathub, the code is built by Flathub themselves. You provide only the manifest and some metadata. Well, you build locally as well for testing, but the productive package is done by their Buildbot.

You can look into repositories for popular apps: https://github.com/orgs/flathub/repositories?q=sort%3Astars

The application is packaged by them and the end-user receives only the binaries. No clutter :D

I looked into a few repositories, and I've noticed many are based off on .deb packages. I suppose I interpreted the section "Best practices" about binaries wrong (maybe only additional, non first-party binaries are refrained upon?) and gave you misinformation regarding binaries, my bad.

You could probably look at e.g. https://github.com/flathub/com.visualstudio.code/blob/master/com.visualstudio.code.yaml for an example of an app built using .deb.

But as stated previously, I experimented with Flatpak a little bit, but I'm no package maintainer or anything. Do also write to the folks at the Matrix chat https://matrix.to/#/#flatpak:matrix.org, they can also help out on the whole process.

CodedOre commented 1 month ago

So, out of curiosity, I have experimented with creating a functional Flatpak manifest for DevToys. The results can be found in the attached archive.

First of all, I'm not an active maintainer of any Flatpak. I just wanted to experiment with packaging this app. And while it is a good starting point, it needs more improvements before it could be submitted to Flathub.

The archive contains a Flatpak manifest, which can, in theory, build the DevToys application from source and package it as a Flatpak.

However, there are a few caveats:

Firstly, Flatpak Builder pulls the dependecies before building, and then runs the build without a connection. AFAIK this is done to improve security and audibility of an Flatpak package.

However, this means dependencies need to cached by Flatpak Builder and the build tools need to use that cache. Thanks to some tooling from the Flatpak team, creating that cache is quite straightforward. The archive contains a index used by Flatpak Builder to cache the required packages, and the commands to update this index can be found in the Readme.

But this means also that I had to patch the build process to point towards said cache, so that Dotnet can build the app inside the build sandbox. Ideally, this patch should make it's way upstream as a build option.

Secondly, the archive also contains a data folder. This contains the desktop file, icons and a Appstream file. The latter is used by software centers like Gnome Software, KDE Discover or the Flathub website to display informations about the app.\ Flathub policy is normally that these files are maintained upstream, so this would most likely come up when submitting the app.

Lastly, while I technically have a build Flatpak with the DevToys application, it does not work right now. The application does not start, due to it missing some file.

I suppose though this is because I'm not running the build or install as expected by DevToys.[^1]\ Because I haven't found instructions for "building DevToys from a CLI", only from running from some IDE's, I've decided to show the current state for further discussion.

Finally, heres the archive for the Flatpak.

[^1]: Check lines 49 - 51 in the file app.devtoys.DevToys.yml for how I attempt to do it.

CodedOre commented 1 month ago

A few further notes: The reason the application did not start in the Flatpak was because is because it has a misdirection for a wwwroot folder. While running it still expects it from the source folder [^1], when it really is in the install folder [^2].

But I'm not sure why this happens.

@veler, it would help if you could provide some information on how you build the existing Linux releases, especially the portable zip. Might show why it expects the wwwroot in the wrong place.

[^1]: Flatpak Builder mounts the sources in /run/build/devtoys/, so the path is /run/build/devtoys/src/app/dev/DevToys.Blazor/wwwroot. [^2]: From the view of the app, it is installed under /app/lib/app.devtoys.DevToys, so the path should be /app/lib/app.devtoys.DevToys/wwwroot?

CodedOre commented 1 month ago

Managed to find why the app didn't want to start in the Flatpak. Removing the DevToys.Linux.staticwebassets.runtime.json file from the package makes it run.

With that, I now have a functional Flatpak manifest for DevToys: DevToys running from a Flatpak, with Gnome Software showing it's metadata

Here is the archive with the manifest.

In my view, the contents in the archive should be sufficient for submission to Flathub.

A few things to note:

As mentioned before, Flatpak builds without a connection. Therefore the patches to the build system (found in the patches folder) are neccessary to allow to build it in the Flatpak Builder environment.
The contents in the data folder should ideally be organized somewhere in upstream, as they would also be used by other Linux distribution methods.
If updates should arise, the dependencies need to be updated manually. Flathub does provide a mechanism to automatically check for updates to dependencies, but AFAIK this doesn't apply to Nuget oder NPM sources. The neccessary commands to update these sources are documented in the Readme.

veler commented 1 month ago

Hi @CodedOre and @hockdudu ,

Sorry for my lack of responsiveness here. Quite busy with many things this summer!

@CodedOre, this is impressive to me, and I have many interogation as Flatpak is still quite obscure to me. Thank you very much for exploring all this though! I really appreciate !

Managed to find why the app didn't want to start in the Flatpak. Removing the DevToys.Linux.staticwebassets.runtime.json file from the package makes it run.

That is interesting. I do not expect this file to be generated when using dotnet publish in Release mode. I looked at the archive you shared and saw that "$DOTNET_EXE" build is used in devtoys.patch (is that correct? is this an auto-generated file?)

Here is what I use in the CI I made (which is different that build.csproj in DevToys repository), perhaps it explains the difference:

dotnet publish DevToys.Linux.csproj --configuration Release --framework net8.0 --runtime linux-x64 --platform x64 --self-contained true /p:PublishSingleFile=false /p:PublishReadyToRun=false /p:PublishTrimmed=false /p:RuntimeIdentifierOverride=linux-x64 /p:Unpackaged=true

And then I delete output folder/wwwroot/_content. Then I copy LICENSE.md to the output folder. Finally, I extract the built NUPKG of DevToys.Tools to output folder/Plugins/DevToys.Tools.

CodedOre commented 1 month ago

Thanks for the reply, @veler!

Since I haven't much experience with .net, I have based the Flatpak Build on the build.sh script in the folder (which comes from the NUKE Execution Engine?), since it seemed like the expected build system to me.

The build command Flatpak Builder currently uses is this one:

./build.sh --configuration Release --target PublishApp

The $DOTNET_EXE is from said build script.

Here is what I use in the CI I made (which is different that build.csproj in DevToys repository), perhaps it explains the difference:
dotnet publish DevToys.Linux.csproj --configuration Release --framework net8.0 --runtime linux-x64 --platform x64 --self-contained true /p:PublishSingleFile=false /p:PublishReadyToRun=false /p:PublishTrimmed=false /p:RuntimeIdentifierOverride=linux-x64 /p:Unpackaged=true

I see. AFAIK this is more or less the same as with what NUKE eventually runs. But it may make sense to only run dotnet restore and dotnet publish, instead of the whole build system then.

I will look into this later this day.

If you have other questions about the Flatpak build, just ask away.

veler commented 1 month ago

Thank you :)

I guess, it would certainly help (with flatpak and other...) if I was publishing the CI on GitHub. The reason I kept it private is to try to deter some people from taking DevToys, which is under MIT license, and sell it without any significant change (it's something I saw happening in the past). Hoping that not publishing the CI would hide away the little details on how to publish the app smoothly.

The build command Flatpak Builder currently uses is this one:
./build.sh --configuration Release --target PublishApp

Oh I see where it's used now ! Well, since my CI also generates a ZIP, wouldn't be easier (and faster in CPU / server time) to just unzip / use the output folder instead of building, if I integrate this into my CI? How about the part in app.devtoys.DevToys.yml that points to Monaco editor (line 32)? Is it still needed if flatpak doesn't need to build DevToys from the sources?

Note that the CI I currently have privately also takes care of restoring nuget packages, node modules and Monaco Editor, and build DevToys.Tools properly.

CodedOre commented 1 month ago

Well, since my CI also generates a ZIP, wouldn't be easier (and faster in CPU / server time) to just unzip / use the output folder instead of building, if I integrate this into my CI? How about the part in app.devtoys.DevToys.yml that points to Monaco editor (line 32)? Is it still needed if flatpak doesn't need to build DevToys from the sources?

Note that the CI I currently have privately also takes care of restoring nuget packages, node modules and Monaco Editor, and build DevToys.Tools properly.

Well, in would be possible to build the Flatpak from a build output. You could take the output folder as directory source and only run the install commands with Flatpak Builder. [^1]

However, it also makes it less reproduceable and auditable, as it would basically take an unknown source with binary files.

As the Flathub docs state:

Applications should build all components of the manifest from source when possible.

If a distribution of a pre-build binary is wanted, the source would probably need to be publicly accessible, like the portable zip from the releases.

[^1]: "Install commands" mainly means to copy them to a location inside /app, as this folder is then packaged by Flatpak.

veler commented 1 month ago

I see. Building from the source might not be ideal in my case though, because the build process for release isn't as straight forward as it may sound like.

If a distribution of a pre-build binary is wanted, the source would probably need to be publicly accessible, like the portable zip from the releases.

Perhaps I misunderstand this. The portable ZIP and what the deb (on Linux), exe (on Windows) contains are the same. DevToys is literally built once per platform, then the binaries are stored into a zip, or packed in a deb, or in an installer for Windows. It would be the same thing here. The "source" is what there is on GitHub, either in each Release or in the main branch. It's already public. The current CI for generating what is then published on the GitHub Release page is NOT public at the moment. Is that the problem?

Also, are we talking about building DevToys on the customer machine in order to install Flatpak? If not, I'm not understanding how different from an auditing point of view it is to build the Flatpak from a build output instead of from the source (which... will generate the same build output my CI does, if configured properly).

Sorry for the confusion 😅

CodedOre commented 1 month ago

The user does not need to build the Flatpak on his machine.

In most cases, Flathub builds the Flatpak from the manifest on its own infrastructure on buildbot.flathub.org. Some, like OBS or Firefox, are build on their own CI's, with the Flatpak then published to Flathub.

The user then downloads and installs the package.

However, the manifest is publicly accessible. If you open any app on Flathub, you can find a link to the manifest file and check how its packaged, and could be audited on the package process.

Also, when submitting a new application, the manifest will be reviewed by the Flathub team.

Now, Flathubs policy is to prefer all components being build from source, if possible. So, quite a few apps, like OBS, Chromium or Gnome or KDE apps, have the sources and build instructions in the manifest.

It is possible to tell Flatpak Builder to use a archive of pre-buils binaries to create the package, which is what most proprietary and some unverified apps are doing.

This is less auditable (as they are binaries), but generally accepted by the Flathub team.

Still, this needs a accessible location from where to pull the binaries. And even if you were to package the binaries on your own CI, the reviewers would likely ask for the sources to be externally verifiable.

Which is why I mentioned the portable zip. If a binary distribution is preferred, it would make most sense to use this as source for the Flatpak package.

On a personal note: I choose to build from source because I generally like transparency over an build process, and also because just packaging the binaries wouldn't been a interesting challenge ;-)

But I could create a manifest which just packages the release archive.

As a bit of a sidenote, on how Flatpak Builder works:

Flatpak Builder first downloads the sources to an cache, found in the folder .flatpak-builder (generated in the working directory). Then, the build commands are run inside a sandbox, using the SDK set in the manifest. And at the end, the files located in the /app folder are packaged for release.

When the Flatpak is run, this /app folder will be mounted inside a sandbox and the set command is executed.

There are a few expectations on how the /app folder should look like, but those mirror Linux standards, just with /app instead of /usr as the prefix.

CodedOre commented 1 month ago

Alright, so I updated the manifest to use the publish command directly. I also found a way to update the nuget sources to the Flatpak-provided cache, so no patches are required anymore.

Here is the archive with the updated manifest: DevToys-Source-Flatpak.zip.

Also, I created a second manifest which takes the pre-compiled binaries from the Github Releases, for comparison: DevToys-Archive-Flatpak.zip

veler commented 1 month ago

Ooh, that makes more sense now! Thank you for the long explanation! I totally get why it's better to build it from the source in this scenario then! :D

I haven't look at the manifest, I will try to do that tonight.

Assuming it looks good, what next step would you recommend?

CodedOre commented 1 month ago

I have found a bug, introduced when switching from using the Build project to using the dotnet commands directly, as it does display the wrong version:

DevToys showing the wrong version

I noticed that there is a SetVersion step in the Build system, we probably need to run that or some similar functionality in the Flatpak build.

Then, there is the question of the contents in the data folder.

This folder currently includes a copy of the icons in the folder structure Linux expects, the Desktop file and the Appstream file.

As the Flathub docs states:

These metadata files should directly come from upstream whenever possible.

So, the question is if they could be brought upstream.

When those points are resolved, I think the next step would be submitting the app. The submission process is outlined in this document. But it would mostly mean that the files from my archive would be submitted via an Merge Request on the Flathub repository. After that it's up to the Flathub team to review and approve the application.

Lastly, there are a few steps to verify the Flatpak as official. In the case of DevToys, this would mean you getting a key from Flathub and storing it in a specific location on your website. Flathub would then use the key to verify that the Flatpak is official. The instructions for that process are located here.

That's basically it!

veler commented 1 month ago

I have found a bug, introduced when switching from using the Build project to using the dotnet commands directly, as it does display the wrong version:

Ah! Yes! I forgot that... In the CI I made, we also set the version number in some files (and the syntax of it varies). Here is the steps the CI performs:

Update the changelog in DevToys.Blazor/Assets
Update the version number in DevToys and DevToys.Tools
Update the splash screen (it changes based on whether it's a preview version or not)
Restore dependencies (nuget, monaco, npm...)
Build the solution (dotnet build)
Run tests
Publish the solution (dotnet publish and pack)
Generate installer, deb, or zips

Ultimately, and based on my understanding of your previous explanation, a good thing to try would be to:

Make the CI public, except whatever I don't want to be public, but there's nothing private for the Linux build as far as I remember.
Update the CI so we can optionally omit steps 4, 5 and 7.
Change the Flatpak manifest so it runs this CI with these omitted steps, and use the generated output as what to install on the user machine, which is ultimately the exact same thing that end up in the published portable Zip.

How does that sound?

These metadata files should directly come from upstream whenever possible.

I'm not sure what they expect. Is the upstream the GitHub repository?

Thanks for sharing your next step! Hopefully I can automate that more with the CI. Currently, I upload the generated packages to GitHub Release manually from each OS (Windows, Mac, Linux), publish an update on Microsoft Store, Homebrew and Winget manually, and soon I will add Chocolatey, and now Flatpak. It starts to add up 😅

CodedOre commented 1 month ago

Ah! Yes! I forgot that... In the CI I made, we also set the version number in some files (and the syntax of it varies). Here is the steps the CI performs:

Update the changelog in DevToys.Blazor/Assets

Update the version number in DevToys and DevToys.Tools

Update the splash screen (it changes based on whether it's a preview version or not)

Restore dependencies (nuget, monaco, npm...)

Build the solution (dotnet build)

Run tests

Publish the solution (dotnet publish and pack)

Generate installer, deb, or zips

Ultimately, and based on my understanding of your previous explanation, a good thing to try would be to:

Make the CI public, except whatever I don't want to be public, but there's nothing private for the Linux build as far as I remember.

Update the CI so we can optionally omit steps 4, 5 and 7.

Change the Flatpak manifest so it runs this CI with these omitted steps, and use the generated output as what to install on the user machine, which is ultimately the exact same thing that end up in the published portable Zip.

How does that sound?

That would sound good.

These metadata files should directly come from upstream whenever possible.

I'm not sure what they expect. Is the upstream the GitHub repository?

In most cases, it is the GitHub repository of the app. However, even a additional repository in the DevToys organization is likely fine.

AFAIK, the main point behind this policy is to ensure that the information in the data files, especially the Appstream file, matches the other distributions of the app.

Thanks for sharing your next step! Hopefully I can automate that more with the CI. Currently, I upload the generated packages to GitHub Release manually from each OS (Windows, Mac, Linux), publish an update on Microsoft Store, Homebrew and Winget manually, and soon I will add Chocolatey, and now Flatpak. It starts to add up 😅

Well, submission and verification are steps you only need to do once.

Though updates to the manifest remains a thing.

CodedOre commented 1 month ago

I do have to ask: This CI you mentioned, it isn't the NUKE build environment I've used before? As this seems to perform most steps you've mentioned.

veler commented 1 month ago

I do have to ask: This CI you mentioned, it isn't the NUKE build environment I've used before? As this seems to perform most steps you've mentioned.

Yes it is. It's a NUKE build. I should probably not use the CI term in here. I understand myself 😅.

CodedOre commented 1 month ago

I see. Then I should revert to use the ./build.sh script in the manifest. I do think I can avoid reintroducing the patches, now that I found the trick with updating the nuget source.

One thing I noticed: You mention as your first step "Update the changelog in DevToys.Blazor/Assets". Is this a step done by the CI?

Reason is: As mentioned before, the Appstream file is used by the Flathub website and Linux software centers to display information about the app. This also involves informations about releases and the changelog (Check the contents of the <releases> tag). My question now is: If you already have code that updates a changelog from a specific source, could we update said Appstream versions with that as well?

veler commented 1 month ago

You mention as your first step "Update the changelog in DevToys.Blazor/Assets". Is this a step done by the CI?

No, I do that manually before running the CI. Example: https://github.com/DevToys-app/DevToys/commit/1c60b39fe9f317bc02fb55249fb8ab8d5c6c31ee

CodedOre commented 1 month ago

You mention as your first step "Update the changelog in DevToys.Blazor/Assets". Is this a step done by the CI?

No, I do that manually before running the CI. Example: 1c60b39

Alright, so the Appstream file will just have to be updated manually.

I have today looked again at using the NUKE build.sh, but after looking at it today, it has opened a few questions on the best approach. So, let's go through the steps:

Update the version number in DevToys and DevToys.Tools The build.rs in this repository does start an SetVersion task, but this does only affect the application, but not the version number of the tools. Actually, since Application and Tools are in separate repositories, Flatpak Builder currently runs them in separate steps, and the Tools repository currently has no SetVersion task. We could combine these two steps if needed though.
Update the splash screen (it changes based on whether it's a preview version or not) Not sure if this is currently present in the publicly visible build instructions, so I'm don't know what exactly this entails.
Restore dependencies (nuget, monaco, npm...) Mostly unproblematic, expect for the monaco part. The Restore-MonacoEditor.sh script always wants to download currently. So, there would be a switch needed, which uses the pre-loaded archive when running from Flatpak Builder.[^1]
Build the solution (dotnet build) Fairly unproblematic, expect for the fact that build.sh will not run it when you set the target to PublishApp.
Run tests
Publish the solution (dotnet publish and pack) Well, this is the difficult part. As of now, the build.sh wants to publish four versions, portable and not, for two architectures, on each run. In the case of Flatpak Builder the architectures part is a bit annoying. The Flatpak Builder would run separate for each architecture, with a base SDK on the other architecture. But with the current build.sh this leads to the publish step wanting to build an ARM build on the x64 SDK, which would require more pre-loaded Nuget packages for no advantage, and vice versa. So, we would want to only run publish for the architecture currently being build.
Generate installer, deb, or zips That's fairly simple again, as it would only mean to move the finalized files to the right locations inside /app.

All these observations are based on the build system as it can be run from the build.sh script in the DevToys repository. Since you've mentioned that you use a different CI for the actual builds, I wanted to wait with patching this build system until feedback.

[^1]: Sidenote: We can detect if the build runs inside a Flatpak (as with the build) by checking that the environment variable container equals flatpak.

CodedOre commented 1 month ago

@veler, how do you want to proceed with this?

I see three options for this:

Use the external pipeline you've mentioned. I assume this is a third project, aside of DevToys and DevToys.Tools, which is building both elements. The manifest could be updated to use this, if it's similar to the build pipelines in the DevToys repository.
Updating the existing build systems in DevToys and DevToys.Tools with the points mentioned in my last comment.
Just repackage the released archive.

Since there are multiple options, you would need to decide which option to take.

veler commented 1 month ago

Hello @CodedOre :)

Sorry for the late reply. I finally took the time to remove from our CI what I didn't want to get public, and published 99% of it on GitHub: https://github.com/DevToys-app/Publish

The unpublished part should not be needed unless you want to publish the binaries on Apple Store or Microsoft Store.

With this, my suggestion is to edit that CI to add some logic for the flatpak. There may be a bit of refactoring needed. I do admit that the code in this repository is literal garbage compared to DevToys and DevToys.Tools code. A good entry point could be here: https://github.com/DevToys-app/Publish/blob/f8d47e1cc833beac5c57e45084ac7ea50f1735a3/src/build/Submodules/DevToys/Packing/Debian/GuiPackingDebian.cs#L14

I'd be happy to help integrating the flatpak to this. You seem to be advanced on the topic now. I'm also happy to let you draft a PR at your own rhythm. Please let me know what you prefer and how I can help :)

Thank you again for everything!

CodedOre commented 1 month ago

Thanks for publishing the CI, @veler.

I will look at adding the Flatpak part in the coming days.

CodedOre commented 1 month ago

Alright, to give an update:

I have looked at the CI, and unfortunately, the refactoring required would be more than just "a bit"...

Let's just go through to the ideas I tried in the last few days:

1. Adding Flatpak to the CI

The idea here was to add code to the CI to package the compiled program in a Flatpak and publish this to Flathub.

I did talk today with a few Flathub maintainers over on the Flathub Matrix room, and this seems to be an unlikely option. While some apps are build from external CI's, these are "exceptions for well-known projects". And it was highly recommended to use the normal submission process.

Apart from that, running Flatpak Builder inside the NUKE build environment proved to be surprisingly difficult...

So, I would drop this idea.

2. Running the CI inside Flatpak Builder

The idea here was to run the NUKE build system inside Flatpak Builder.

Well, this would need some major refactoring. Especially when looking at the Publish repository, there a lot of points where it wants to connect to the internet to load additional files, like updating the submodules, or even installing system packages (!)... The fact that a sudoapt command is present is problematic for multiple factors, like security or incompatibility with other distributions like Fedora or Flatpak.

All of these are points that make it difficult to run it in the environment of Flatpak Builder. And this is ignoring the fact that it is relatively bloated, building every step and multiple arches, and even the build system on each run...

So, where does this leaves us?

I would suggest on of two options:

Option 1: Rework building entirely

As one of the Flathub maintainers put it:

it would be much simpler to focus on making your build system reliable and off line so that it can reproduce its artifacts. Most of the "we want to build it ourselves" boils down to a fragile build process which lead to more issues

I can't speak much about how this would look for Windows or Mac, but for Linux packaging, especially Flatpak, the ideal case would be an build system that can use pre-loaded dependencies and then builds offline. The build would then be started from the packaging software.

The last DevToys-Source-Flatpak archive already does this in most parts by ignoring the NUKE build. What's basically missing is a script handling the versioning update.

Option 2: Just package an existing release

Alternatively, we could just use the solution in the DevToys-Archive-Flatpak archive, use the released archives and package this. Not ideal, but better than trying to package the Flatpak inside the NUKE CI.

So, that's for the update.

badcel commented 1 month ago

I still did not look it up myself but for example Nickvionapps are Gir.Core based apps wich are packaged on flathub.

Perhaps it helps to get started.

I myself started to use F# and it's script files as build system in combination with the dotnet commandline tool. Those scripts can be called directly via dotnet fsi. They do not require any setup scripts and are providing a complete programming language at your fingertips.

CodedOre commented 1 month ago

I still did not look it up myself but for example Nickvionapps are Gir.Core based apps wich are packaged on flathub.

Perhaps it helps to get started.

The Nickvision apps use the Cake build system. And more importantly, don't have the build system adapting the version number on build... Thats more or less the issue.

I myself started to use F# and it's script files as build system in combination with the dotnet commandline tool. Those scripts can be called directly via dotnet fsi. They do not require any setup scripts and are providing a complete programming language at your fingertips.

That sounds good. I think changing the version and splash screen updates to a F# script that can be run without the NUKE build would resolve that part of the issue. Will look at this later today. @veler, your thoughts?

veler commented 1 month ago

Hello,

Thanks for your follow up. :) I appreciate the time you spent on that.

Argh. Yeah this is tough. I wasn't aware that connecting to the internet would be an issue. See, this Publish project is design to run in a pipeline, not on a local machine of the customer.

Question: From what I was understanding, Flatpak packaged are being installed by building sources on a local machine. That requires all the sources to be present. Is it a bad thing that a few files are being changed on the fly so they have the right version number?

For the things downloaded from the internet, it "could" be avoided through a setting in the command line that makes skipping the part where we run the sudo apt commands.

Also, I realize the Publish pipeline contains a bunch of Windows-specific binaries, which makes this quite dirty for Linux.

Really, it's NOT designed to be shipped to end users. We'd indeed need to rewrite the whole publication pipeline just to take consideration of Flatpak. I'm also not super fan of the second option because a bunch of currently manual steps would need to be done.

My ultimate wish / desire for the future is it have a 1-click CI that does everything, including creating the pull requests to update Homebrew and WinGet packages, pushes the MSIX to the MS Store...etc (which I currently do manually, and it takes quite some time), and include Flatpak in all this.

CodedOre commented 1 month ago

Question: From what I was understanding, Flatpak packaged are being installed by building sources on a local machine.

Not really. They are build before-hand, but in the case of Flathub, they are build by an CI externally of the project.

That requires all the sources to be present. Is it a bad thing that a few files are being changed on the fly so they have the right version number?

While it is, unconventional, it in itself should not be that problematic from a technical perspective. The main issue is that these steps are tied to the pipeline, which also includes a lot of steps not working well inside Flatpak Builder (e.g. downloading while building).

I'm also not super fan of the second option because a bunch of currently manual steps would need to be done.

The second option I've mentioned would just mean that the Flatpak manifest would load the released binaries from GitHub and package it, so I'm not sure about the "manual steps" you're concerned about.

If its about updates, we can add parameters to the manifest which will tell the Flathub CI to automatically check for an update and prepare the changes needed to the manifest in an pull request.

Its still not fully automatic, but manageable.

DevToys-app / DevToys

Flatpak version #1199

What feature or new tool do you think should be added to DevToys?

Why do you think this is needed?

Solution/Idea

Comments