search

dxx-rebirth / dxx-rebirth

Descent-Rebirth and Descent II-Rebirth
Other
493 stars 117 forks source link

Feature request: publish on flathub.org for SteamDeck #691

Open JoeNotCharles opened 1 year ago

JoeNotCharles commented 1 year ago

It would be great to package dxx-rebirth as a flatpak and publish it through flathub.org (https://github.com/flathub/flathub/wiki/App-Submission).

flathub.org is the main 3rd-party feed for the SteamDeck, so this would let users easily install dxx-rebirth there.

vLKp commented 1 year ago

This was previously requested in #680, but the requester never responded. It's not clear to me from the linked page how I would test that Flatpak built the program correctly, which makes me disinclined to have them redistribute it.

JoeNotCharles commented 1 year ago

Looks like there are more complete docs at https://docs.flatpak.org/en/latest/building.html. I'll see if I can get a build set up.

vLKp commented 1 year ago

Thank you. To elaborate on my comment above: I am uncomfortable posting for distribution a program I cannot actually test. For Windows and for traditional Linux distributions, I can readily create a virtual machine of the appropriate environment to try it out. When a Linux package maintainer packages Rebirth without me (which I have no objection to them doing), they (usually) direct their users to report problems to them first, and to upstream second, so if the package is broken due to a problem in the distribution, the relevant distribution maintainer hears about it and can deal with it. When I post something directly, users reasonably expect they can come straight to me with a problem. Thus, I don't want to post things which may be broken and give people the impression that I expect it to work. This is compounded in the case of Flatpak, which appears to have a strange distribution and sandboxing model, so there are more opportunities for things to go wrong.

I tried reading through their setup documentation, and it does not look very promising. I started at the link you gave, which starts off by telling me to read Building your first Flatpak first. I went there, and it starts off by telling me that I first need to read and complete setup guide on flatpak.org. I went there, and it's just an ugly mess of icons, several of them far too large (Rocky Linux alone is 4096 pixels tall), and most are not properly labelled. At that point I decided the rabbit hole was too deep and gave up. I don't care to spend this much time just trying to get set up to learn how to build yet another package format.

I wish you luck with the setup, and I'm happy to answer any questions you may have about build problems. I hope the flatpak build environment is typical enough that following the usual Linux instructions will work for you.

JoeNotCharles commented 1 year ago

Yeah, the docs for flatpak are pretty poor. Now that I've dug into then further, it's much simpler than it appears. Ignoring all the details of sandboxing, etc, flatpak is basically an environment of its own that sits on top of any Linux distro, so if it can be built and tested on one distro it should just work everywhere. (And if somebody reports that it doesn't work on their distro's version of flatpak, that's a flatpak bug that should be reported to the flatpak maintainers.)

I should have a fixed version number to package in the flatpak though. The latest tag is 0.60.0-beta2 which is pretty old. Is there a stable version?

Also are there any automated tests?

On Sun, Jan 8, 2023, 11:39 vLKp @.***> wrote:

Thank you. To elaborate on my comment above: I am uncomfortable posting for distribution a program I cannot actually test. For Windows and for traditional Linux distributions, I can readily create a virtual machine of the appropriate environment to try it out. When a Linux package maintainer packages Rebirth without me (which I have no objection to them doing), they (usually) direct their users to report problems to them first, and to upstream second, so if the package is broken due to a problem in the distribution, the relevant distribution maintainer hears about it and can deal with it. When I post something directly, users reasonably expect they can come straight to me with a problem. Thus, I don't want to post things which may be broken and give people the impression that I expect it to work. This is compounded in the case of Flatpak, which appears to have a strange distribution and sandboxing model, so there are more opportunities for things to go wrong.

I tried reading through their setup documentation, and it does not look very promising. I started at the link you gave, which starts off by telling me to read Building your first Flatpak https://docs.flatpak.org/en/latest/first-build.html first. I went there, and it starts off by telling me that I first need to read and complete setup guide on flatpak.org https://flatpak.org/setup/. I went there, and it's just an ugly mess of icons, several of them far too large (Rocky Linux alone is 4096 pixels tall), and most are not properly labelled. At that point I decided the rabbit hole was too deep and gave up. I don't care to spend this much time just trying to get set up to learn how to build yet another package format.

I wish you luck with the setup, and I'm happy to answer any questions you may have about build problems. I hope the flatpak build environment is typical enough that following the usual Linux instructions will work for you.

— Reply to this email directly, view it on GitHub https://github.com/dxx-rebirth/dxx-rebirth/issues/691#issuecomment-1374877144, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMDPP2ILL3Q2I4A3XNLM7LWRLUUXANCNFSM6AAAAAATUJC2SE . You are receiving this because you authored the thread.Message ID: @.***>

vLKp commented 1 year ago

No one has designated any particular commit as a release in years. I would not recommend using any of the existing tags as a base. I generally try to keep master in a good state, such that any snapshot from it should be fine. The mess with the audio resampler is the first time in quite a while that I've had a known issue in master that was visible enough to players to lead to a problem report.

There are some unit tests that I wrote for my own convenience, but they are entirely around verifying the correct behavior of some utility code that I added. Most of the core game is untested. If you want to build the unit tests, they can be enabled by adding register_runtime_test_link_targets=1 to the scons command line. Some of the tests are static_assert based and so are triggered by compiling them. Others result in test programs in $builddir/common/, and are all named test-*. Boost.Test is required to build the unit tests.

JoeNotCharles commented 1 year ago

I've got a first draft working, but it's missing MIDI support (and I haven't tested gamepad/joystick issue or networking yet)

... Using SDL_mixer library v2.6.0 sdl-joystick: 0 joysticks DXX-Rebirth: OpenGL: disabling automatic GL sync since VSync is turned off Music descent.hmp could not be loaded: Unrecognized audio format

SDL uses flatpak's stock setup which is:

SDL2 Configure Summary: Building Shared Libraries Enabled modules : atomic audio video render events joystick haptic hidapi sensor power filesystem threads timers file misc locale loadso cpuinfo assembly Assembly Math : mmx 3dnow sse sse2 sse3 Audio drivers : disk dummy oss alsa(dynamic) pipewire(dynamic) pulse(dynamic) jack(dynamic) Video drivers : dummy offscreen x11(dynamic) kmsdrm(dynamic) opengl opengl_es2 vulkan wayland(dynamic) X11 libraries : xcursor xdbe xinput2 xinput2_multitouch xfixes xrandr xscrnsaver xshape Input drivers : linuxev linuxkd Enable virtual joystick APIs : YES Using libsamplerate : YES Using libudev : YES Using dbus : YES Using ime : YES Using ibus : YES Using fcitx : YES

Is there an SDL option or additional dependency I'm missing?

vLKp commented 1 year ago

SDL support for MIDI is optional. Perhaps Flatpak's maintainers built without it. It appears to be controlled by the configure option --enable-music-midi.

JoeNotCharles commented 1 year ago

Work in progress at https://github.com/JoeNotCharles/flatpak-dxx-rebirth

maggu2810 commented 1 year ago

I know this topic is about flatpak and not AppImage but if it is just about a working solution for the Steam Deck:

For my Steam Deck I created AppImage files for d1x-rebirth and d2x-rebirth. Upstream already contains a package for linux script that creates AppImages.

Mine differs from upstream one:

I created a Containerfile that can be used with podman to build the AppImage. Upstream uses GitHub workflow.

-- edit --

I gave it a try and migrated to a GitHub workflow, too. First result: ... deleted, because of further improvements ... Does it work for you?

Would upstream be interested in?

-- edit --

https://github.com/maggu2810/dxx-rebirth/actions/runs/4594866874

vLKp commented 1 year ago

@maggu2810 wrote:

  • I bundle "all" required shared libraries, so they need not to be present on target machine (for me Steam Deck). Upstream does not bundle required shared libraries.

I thought AppImage (added in #652) did this by default. That is the point of both Flatpak and AppImage, to include everything necessary to run the program.

  • Upstream uses 20.04.

I believe this was intentional, because older versions lacked a sufficiently recent compiler. Looking at your AppImage commit, it looks like you encountered exactly that problem, and had to resort to external personal package archives to get a working compiler and usable Python. I dislike this, as those personal package archives could become unavailable or incorrect without warning. Using a more recent base system would provide those with the system repository. System archives have a documented lifecycle, and can be expected to continue offering usable packages until the end of the lifecycle. What are you trying to run on that Ubuntu 20.04 was too new? Is your target system even supported upstream anymore?

First result: https://github.com/maggu2810/dxx-rebirth/suites/11965850856/artifacts/627689625 Does it work for you?

No. That link fails with a 404 for me.

Would upstream be interested in?

If by upstream you mean the dxx-rebirth project, which in practice just means me, then maybe. I dislike adding yet another dependency (linuxdeploy) in a workflow I cannot run, so I would prefer to see the AppImage Github workflow fixed such that it bundles the required libraries directly. I would be particularly pleased to see the AppImage Github workflow improved such that I can actually run it. Presently, it opens a blank window with a never-ending spinner, no status messages, and no error messages. No artifacts are ever produced. This makes testing the workflow impossible, so I am reluctant to apply changes that I cannot validate by visual inspection.

maggu2810 commented 1 year ago

After forking your repo, I was able to run the GitHub workflow by trigger the action manually. So, no modification from my side. It works. Strange that the same did not work in your repo.

18.04, 20.04, ... I personally do not care as all my systems are (much) more recent. AppImage officially states, you should consider to use the oldest LTS version that are still supported (AppImageKit itself is using 14.04).

The original commit copies image, desktop file and binary for the AppImage, no more - no shared libs.

"linuxdeploy" is just another tool to create AppImage files. I removed two downloads in the package script and replaced two by one (linuxdeploy). So IMHO it is not more bad then before.

All in all, no problem if you dislike it. I did my excurse / learning AppImage - how it works and how it is created. I got executables for my Steam Deck and will perhaps repeat it for other games I like to play.

I get your point about that personal ppa repositories. How do you decide which version of C / C++ compiler, Python and SCons is the favorited one (the one that should work)? The way I used allows to select for all stuff an explicit non distribution specific version.

vLKp commented 1 year ago

I do not know what to tell you about the workflow. As I said, it neither works, nor produces any status messages or error messages that could explain why it does not work. As it produces no messages, there is nothing I can do to try to debug it. As best I can tell, it does not run silently, as no artifacts appear to be produced.

I understand the idea of "oldest supported," but since the older LTS releases lack multiple tools required to build the program, I do not consider them fully supported. In particular, when I build locally, one of the libraries that Rebirth depends on is libgcc_s.so, which is provided by gcc itself. I would not be surprised to read that the libgcc_s.so shipped with Ubuntu 18.04 lacks symbols that are required to support a program built with a post-18.04 gcc. If so, building on Ubuntu 18.04 does not actually make the program compatible with Ubuntu 18.04, so pretending to use Ubuntu 18.04 and installing modern tools is just added complexity for no benefit. On the other hand, if building on Ubuntu 18.04 (even with modern tools backported in) produced a binary that did run on Ubuntu 18.04, that might justify the extra complexity. Even there, I am doubtful, since "Canonical recommends that users and organisations upgrade to the latest LTS release." Have you checked whether a binary built using your AppImage can run on an Ubuntu 18.04 system?

The original commit copies image, desktop file and binary for the AppImage, no more - no shared libs.

I see. Since the workflow never starts, I had not been able to observe that. From what I understand of AppImage's supposed self-contained properties, that seems like a bug in appimagetool-x86_64.AppImage that it produces an image that is not, in fact, self-contained. However, I admit that I am not familiar with the tool. Perhaps its documentation warns that it does not by default create a working AppImage.

"linuxdeploy" is just another tool to create AppImage files.

True, but I thought, perhaps wrongly, that the existing non-working workflow pulled tools maintained by the AppImage project. As such, I trusted that those programs would continue to be available as long as AppImage was viable, and that they would do the right thing. linuxdeploy is some other project, which may have different lifecycle plans.

How do you decide which version of C / C++ compiler, Python and SCons is the favorited one (the one that should work)?

My target is that a user on the then-current Debian stable (which is presently Bullseye), using only packages available in the main repository, can get the tools needed to build the game. I would have already raised the minimum gcc version higher than gcc-10, but gcc-10 is the highest available in Bullseye. When Debian declares their next release (Bookworm) stable, then after a grace period (the length of which I have not decided), I will likely raise the minimum gcc version based on what is offered in Bookworm (gcc-12).

maggu2810 commented 1 year ago

It is sad, that the workflow mechanism does not work for you. As long as you cannot test it yourself, I agree that it is difficult to judge about it.

From the official project side providing appimagetool:

appimagetool is used to generate an AppImage from an existing AppDir.

It is up to the caller what is the content of the AppDir.

linuxdeploy is able to create / enhance an AppDir and (with the bundled output plugin) to create AppImage.

I do not know how "official" linuxdeploy is. At least AppImageKit is also "just" the reference implementation of the AppImage specification. It also seems that the user TheAssassin works for both.

For me I found IMHO a good solution now. I created another workflow so it does not conflict with any existing upstream file. So I can easily merge all upstream changes into my branch using the GitHub UI without running into merge conflicts. My new workflow builds AppImage for linux and the Windows and Mac OS archives. After the builds succeeded a job is started that provides the results as a continuous release (so it can be downloaded from not logged in users, too - which is not the case for action artefacts).

See

If we further discuss about the packaging, should we create a separate issue? Is there anything to say?

fpiesche commented 6 months ago

To bring this back on track with Flatpaks: I've been building Flatpak distributions for various free game projects over the past few months (in part, as with the OP, because I bought a Steam Deck and it's just a nicer way to distribute software on there as it comes with automatic updates and launcher shortcuts unlike AppImages) so I can probably advise on a few things here, and as @JoeNotCharles seems to have dropped this for the time being I could pick up the work to finish the build.

To answer your question from early on:

This was previously requested in https://github.com/dxx-rebirth/dxx-rebirth/issues/680, but the requester never responded. It's not clear to me from the linked page how I would test that Flatpak built the program correctly, which makes me disinclined to have them redistribute it.

The way publishing on Flathub works is this:

This means that if the test build after an update fails for any reason, nothing changes about the live distribution, and maintainers for the manifest will be notified in the PR of the failure and can see the build log on Flathub's infra, e.g. https://buildbot.flathub.org/#/builders/6/builds/104939

Crucially though, as @JoeNotCharles points out, Flatpak distributions are intended for stable builds rather than development snapshots. The way to do this with a project like this would be to pin the manifest to a specific commit to use as a build base; however the automated update won't just pull new commits so it would be necessary to either start creating tags in the repo again whenever you decide that the current state is "a release", or to manually update the Flatpak manifest to a new commit ID periodically to keep the Flatpak builds reasonably up to date.

With a little clever workflow work however, we could have a very nice process here where when you create a new tag matching, say, v##.##.##, GitHub workflows can automatically create a new "release" in the Github repo and retain the artifacts from the CI workflow run for that tag build indefinitely as release versions. I could help out with that too if you'd like :)

fpiesche commented 6 months ago

I now have working Flatpak builds for both D1X-Rebirth and D2X-Rebirth - if given the above information about how distributing software via Flathub works you're happy for me to submit these for publication, I would be happy to do so (once the question of where screenshots for the metadata should go is sorted out) :)

vLKp commented 6 months ago

Historically, I tried to keep all development snapshots suitable for general use. There have been commits where I failed to live up to that, but in most cases, the report that I let a bug out arrived so long after the bad commit was published that, if I had been issuing regular releases, there is a good chance the bad code would have been declared a release before the bug was reported. Often, it would not only have been released, but several more releases would have been issued in the interim.

Work in #759 has added a set of GitHub Actions, one group to perform a build + artifact publication on every push, and one that, if triggered manually, would declare a release. An earlier GitHub Action, added in #652, attempted to provide a manually triggered workflow. However, I was never able to get it to execute (attempts to start it would just hang, with no errors). I spent a lot of scarce time trying, with no success, to get it to work, so for quite a while afterward, I considered GitHub Actions to be unusable and not worth my time. After seeing the more recent workflows actually work, I am not averse to using GitHub Actions, but I have seen no indication that manually triggered workflows were ever made usable, so any workflow added would need to be automatically triggered.

fpiesche commented 3 months ago

Sorry for dropping off the face of the earth for a bit there; I ended up stuck in a house move for a while. I've just started trying to sort out those workflows and figured I'd chime in to note that, as well as try and answer some more of your questions from earlier in this thread.

Notably, "how would I test a Flatpak build?" - the answer is, like any other package. Once a new release tag is created and Flathub's infrastructure has noticed this update, their build infra will automatically open a PR on the flathub/io.github.dxx_rebirth.d[1|2]x-rebirth repository to update the Flatpak build manifest and kick off a test build on Flathub's infra.

If the test build fails, the build logs can be viewed over on flathub's side; if the build failed because of a problem with the manifest (e.g. a dependency needs updating, or a dependency has been updated and that broke the build), this can be fixed over in the PR on the flathub-side repo and the test build repeated.

Once the test build is successful, it can be installed to any Linux system supporting Flatpak (which is basically every major distro nowadays); their build bot will even comment with the command to do so on the PR.

Once someone (which would be me or yourself or anyone else with contributor access to the Flathub build repo) is happy with said test build(1), the PR for the update can be merged, at which point Flathub will push the build out to their live repository and update the "store" page for it.

(1) a Flatpak build should function exactly like a regular Linux build; Flatpak's packaging is conceptually very similar to AppImage's and the runtime environment is very similar to Ubuntu - it's generally safe to assume that if the build has succeeded it works as well as an un-Flatpaked Ubuntu build would.

vLKp commented 2 months ago

It looks like testing flatpak builds locally would be more trouble than I care to undertake. I've had no use for it to date, and it has a lot of dependencies that I otherwise don't install or use.

However, I have no objection to you posting Flatpak builds of Rebirth on Flathub.

Regarding the footnote (1): it's funny you should mention that. I just recently had a report ( https://github.com/dxx-rebirth/dxx-rebirth/issues/793 ) that an AppImage build was failing to find files it ought to find, and the user resorted to running the Windows build under Wine rather than work with me debug it.

fpiesche commented 2 months ago

I'd be more than happy to maintain the Flatpak builds as is - as the person submitting the app to Flathub I'd be the default maintainer for the repo anyway :)