anura-engine / anura

Anura Engine
Other
382 stars 77 forks source link

Out of tree modules #209

Closed teto closed 6 years ago

teto commented 6 years ago

Hi,

I am trying to package argentumage for nixos but the assumption of anura that games have to be in a submodule kinda defies the nixos assumption that packages should be immutable, aka installing a new anura-based game would trigger a recompilation of the engine which seems like a waste.

I wonder how hard it would be for anura to accept any arbitrary "modules" path ? like anura --module=$HOME/mygames

Related discussion: https://github.com/NixOS/nixpkgs/pull/31511#issuecomment-343741862

DDR0 commented 6 years ago

That certainly sounds reasonable. In the meantime, would a soft link work? (I've commented with more details in the linked discussion.)

DDR0 commented 6 years ago

After some discussion we've decided to promote the soft link to the status of Official Solution. It should be easy, reliable, and widely supported.

lukateras commented 6 years ago

See https://github.com/NixOS/nixpkgs/pull/31511#issuecomment-344689886: as long as Anura engine directory is read-only, symlinking game modules wouldn't work. That applies to Nix as well as SquashFS images.

DDR0 commented 6 years ago

I was thinking you'd have to symlink the modules folder itself, first, so it was writable. Then you'd symlink the individual module folders in there. Would that work?

hagabaka commented 6 years ago

@DDR0 If the Anura package's modules directory is a symlink pointing to another directory, and doesn't create it, the module packages would have to create that directory to be installed. Then, you'd run into the same problem if you try to install a second module package.

teto commented 6 years ago

Nixos is one of a kind (well there is Guix): all packages are read-only once installed, and they are installed in funny folders for instance /nix/store/h2a3x774k6zx32b0lxcqd55brxxyw9ja-qtscript-5.9.1, i.e. the content of the package is hashed and the hash is used in the path; i.e. updating argentum for instance would require to update the symlink, and change anura's hash hence trigger a reinstall of anura. Maybe there is some hackish way around that but it will have bad side effects on nixos side. Having anura use arbitrary module path would be a lot cooler.

lukateras commented 6 years ago

Nixos is one of a kind (well there is Guix) [...]

This breaks system-wide Anura installation on any distribution, not just NixOS.

lukateras commented 6 years ago

[...] i.e. updating argentum for instance would require to update the symlink,

@DDR0 proposes a symlink to a static location, e.g. /run/anura-modules, like what NixOS does with OpenGL. This isn't user-friendly, dangling symlink is a security nightmare, and it would break in multi-user environments.

DDR0 commented 6 years ago

After some discussion, it has been decided that for NixOS it would probably be best to package Anura modules with their own version of Anura. (I expect that the modules path is somewhat embedded in the engine, and difficult for us to change, although I'm not sure.) Since we're not letting ~/modules/ be configurable, I'm going to close this issue.

lukateras commented 6 years ago

Again, this is not specific to NixOS. There is no Linux distro that would be able to package Anura in its current form.

ghost commented 6 years ago

That's right, there are a number of Linux distros, if not all, to which the model of distribution Argentum Age is using is radically alien.

A quick analysis of the components at stake

Not everything is strange but these cases are for sure weird, some distros may accept these, others may forbid, some could even panic at the thinking of:

  1. The auto updated executable binary (and here) A.K.A. the program A.K.A. the game engine,
  2. The library dependencies for a predictable execution of the auto updated executable binary,
  3. The high level source code (and here and here) meant to be interpreted by the auto updated executable binary,
  4. The not meant for execution assets (and here and here) meant to be used by the auto updated executable binary;

By what I have learned in the past months of occasionally collaborating to Argentum Age, there is no doubt that the continuously changing nature of (1. to 4.) is not going to change in any short or mid term.

As long as I'm not aware that any of the prominent distros has released any statement about where belongs each of these (1. to 4.) type of files (or any similar past cases not related to Argentum Age) in the (whatever UNIX school or flavor) directory tree structure they have adopted, I will think that all discussion about packaging is wasted effort.

The good part is that some parts already got their place:

  1. The auto updater executable binary (and here) belongs in /usr/bin/, packagers can do this.
  2. The library dependencies of the auto updater executable binary belong in /usr/lib/. These are handled by a number of packagers. If at some point Argentum Age uses a combination of libraries that can not be handled by a distro release for reasons of dependency conflicts, or there are execution errors, the Argentum Age package gets removed from the distro release.
  3. (Potentially?) high level source code meant to be interpreted by the auto updater executable binary belongs in /usr/share/argentumage/ (I think).
  4. Eventually not meant for execution assets could be used by the auto updater executable, I guess these can belong in /usr/share/argentumage/.
  5. Local chat history, and the guest/s cookie/s, and such likes, all of them belong, if I'm not in mistake, in $HOME/.config/argentumage/.

Me personally I love using Debian GNU/Linux, so I will use Debian as an example from here. I would tell the Debian people (I mean all the people, not necessarily packagers):

For the distros for which the answer is yes, I would ask:

In any case, I could ask both distro people and Argentum Age people:

TL;DR

For good and bad, the Argentum Age software distribution model is more similar to that of some desktop apps and specially a lot of mobile apps made by companies that didn't exist (or have changed substantially) when the prominent GNU/Linux distros were born.

The whole software distribution model in question didn't even exist (or if it did exist, practical use of it was unthinkable) when the prominent distros were born.

And yet I think something can be done to cope with this colliding views and there is a mid way GNU/Linux distros can use to package Argentum Age, but first (me personally) I would need to know what do distro people think today (not by the guidelines in scriptures by the ancestors written fifteen or more years ago) about each questionable part of what Anura is doing now.

teto commented 6 years ago

The autoupdate thing works well on windows but I don't know any linux app doing this, or the few that do require to install their own repository where they store their own package. The autoupdate thing can work if files are written in a user-writable folder ($HOME), which isn't the case for argentum since it has to write things in its own subfolders (and distrib will install it in readonly folders, be it debian or nixos).

ghost commented 6 years ago

The autoupdate thing works well on windows but I don't know any linux app doing this, or the few that do require to install their own repository where they store their own package.

This scheme would be nice. A lot of software vendors do this, and Argentum Age can fit in this, but doesn't do, there is no foreseeable moment to embrace this, I don't think there is manpower to do this, I'm not aware of a lot of interest from Argentum Age to change in this (I'll turn over this later), and to add up, Argentum Age solves the problem with a drastic measure that not only just works, it does in a more predictable way than doing what the GNU/Linux recipes mandate, and without causing actual trouble to the GNU/Linux users. The only problem is that it is just illegal. From a short sighted read of standards now obsolete, but these standards could be extended to help this without breaking the normal works.

Each time it's run and updates are available it potentially pulls:

All these parts could technically be handled using the DEB and RPM systems. I'm sorry to Gentoo, Arch and others for being oversimplifying here.

There is the problem that each time it's run and updates are available it potentially pulls also:

A bad thing here is that this leads to a system with duped libraries installed. Twenty years ago this would have been clearly madness. Now it's something allowed in mobile but illegal on GNU/Linux, even though is not really hurting free disk space anymore, and it's helping stability and easening maintenance. Cellphones are doing this, and GNU/Linux can not? Not a chance. It is just not allowing the possibility, either by chance or by commandment.

By the way, the really bad thing here is the engine doesn't ask before pulling new code, not even once at the time of first install.

The autoupdate thing can work if files are written in a user-writable folder ($HOME), which isn't the case for argentum since it has to write things in its own subfolders (and distrib will install it in readonly folders, be it debian or nixos).

Readonly folders? There is no such thing in my GNU/Linux box. Let a package want to write something to an illegal place, just pop up a shell or desktop environment dialog asking for permission and root elevation if necessary.

Whatever. Do you really think that a couple people (a couple, literally, and one of them is missing most the time) can sort out this auto update scheme, have it working for the big three desktop platforms (using an infrastructure anyone would laugh at), and be incapable of reworking a little where each type of stuff is read from disk?

I think the problem here is more a don't want to fix it yet mentality. I'm not sure where it comes from, but I get more and more convinced it comes from the fact that there is no really any well known place towards where to clearly direct this to, specially in regards to outdated choices by GNU/Linux about duping binary libraries. Have someone from the distros side taking care of previously needed decisions, and it will sort out, I guess..?

It should be easy for Anura to probe several directories for a file, the same as every operating system iterates $PATH since aeons, and it should be easy for distros to provide root elevation when Anura needs it (and let the user have the final decision once properly re-informed about the possible consequences).

I understand distro people not wanting to open this door because ten years from now GNU/Linux could have become a bloated-catastrophe.js, but I doubt every distro will have this door closed forever, given that mobile is actually working like this.

AI0867 commented 6 years ago

Steam and Google Chrome both store all their auto-updated binaries and data under $HOME/.local/share/ on my machine, despite being (originally) installed through apt, so there is at least a precedent for this approach.

ghost commented 6 years ago

I'm sorry to have written that the local chat history, and the guest/s cookie/s should be stored in $HOME/.config/argentumage/. A better location would be $HOME/.local/share/argentumage/.

I had forgotten that for hosting the downloaded files the Linux distributable now uses $AA_EXTRACT_DIR/aa/ instead of $HOME/.citadel/. It's nice it has been so flexible.

[...] for NixOS it would probably be best to package Anura modules with their own version of Anura. [...]

So no multiplayer Argentum Age for NixOS unless using the Linux Argentum Age package? If so, then the login controller is missing changes to remove any sight of multi player (auto logging into the offline mode), and the single player experience is missing some work such as https://github.com/davewx7/citadel/pull/157.

[...] The autoupdate thing can work if files are written in a user-writable folder ($HOME), which isn't the case for argentum since it has to write things in its own subfolders [...]

A previous version used $HOME/.citadel/ for storing the auto updated assets. Could a future version do that again? Should this be flexible from Anura side? Would this potentially fix for NixOS or they will keep wary of downloading unverified binaries?

Steam and Google Chrome both store all their auto-updated binaries and data under $HOME/.local/share/ [...]

I support that $HOME/.local/share/argentumage/ can generally be a good place for the downloaded files, even though this is not conforming filesystem traditions.

I think something like /usr/$DLC/anura/$URL:$PORT/$GAME_MODULE_NAME/ (with familiar subdirectories such as bin/ and lib/, and custom directories like modules/) would be better, if at some point this becomes possible, or if an equivalent directory structure at some point becomes a widespread idiom to suit this type of software.

Whenever the distro can prefer hosting the auto updated assets in a subdirectory of the auto updater, this is a great idea as long as they exceptionally relax good read only after install doctrines in the most controlled possible way and necessarily warning the user.

AI0867 commented 6 years ago

I support that $HOME/.local/share/argentumage/ can generally be a good place for the downloaded files, even though this is not conforming filesystem traditions.

According to modern freedesktop.org (XDG) standards, various environment variables, with defaults such as $HOME/.config and $HOME/.local/share have superseded the traditional $HOME/.$PROGRAMNAME, so configuration, assets and such can be separated (so you can backup all your configs, without all the game textures) and to keep the number of dot-directories under control.

ghost commented 6 years ago

I support that $HOME/.local/share/argentumage/ can generally be a good place for the downloaded files, even though this is not conforming filesystem traditions.

According to modern freedesktop.org (XDG) standards, various environment variables, with defaults such as $HOME/.config and $HOME/.local/share have superseded the traditional $HOME/.$PROGRAMNAME, so configuration, assets and such can be separated (so you can backup all your configs, without all the game textures) and to keep the number of dot-directories under control.

Yes, $HOME/.$PROGRAMNAME is now obsolete.

I meant that storing binary programs and binary libraries under $HOME/.local/share/ defies the old tradition of storing binary programs mainly under /bin/, /usr/bin/ and /usr/local/bin.

Using /usr/$DLC/.../bin/ (at whichever necessary depth but under $DLC) for auto updated binaries would help respecting a tradition that userland programs are under /usr/ (unless under /opt/) instead of under $HOME (so no duplicates caused by multi user). There might be no need of root elevation if the write permission there is granted to the auto updater by executing it using an anura user.

ghost commented 6 years ago

I have been reading parts of the latest FHS lately. My current opinion, unfortunately not neutral, is that on the grounds of an extremely free interpretation of FHS, Anura could use /var/lib/$ANURA/ for all the auto updated contents.

Anura does not fit naturally in FHS. No packager is ever going to risk their time to package a software that is very bad positioned to fight its way into the distro repos.

I picture a way to fight for closing the packaging problem, but realistically many of the steps can not happen and should not happen any sooner than toolchains are available for releasing to Android, iOS, and what comes next.

At last I can tell you I will not dedicate any more time to the topic for years.