Allow using the system's wine version instead of the bundled one

[sharkwouter]

This may improve compatibility and performance, but it also has the potential to break games in unexpected ways, so it might be good to make it configurable.

Currently a gametile will compile a command to execute when starting a game based on which files are in the game's directory. It does already support native scummvm and dosbox installations. Wine games are already detected right now, but they aren't launched in a different way from other games.

The wine prefix is always in a directory called prefix from what I've seen. The location of the game's binary is probably best to retrieve by parsing the start.sh script.

Examples of wine packaged games being sold on GOG:

• Flatout
• Flatout 2

[namandixit]

A better alternative (though a bit more complicated to implement) might be to add support for Steam Play's compatibilitytools.d. This way, not only Windows games would be playable with up-to-date Proton/Wine but community-made tools such as Boxtron, luxtorpeda, Roberta could also be used for games that ship with outdated versions of DOSBox, SCUMM, etc.

[sharkwouter]

It's true that proton is better than standard wine, but I wouldn't want the client to depend upon Steam. Using the system wide installation is also a bit easier and less likely to break with future releases of wine in some ways (Valve might change how proton is launched in the future, for wine the cli is stable).

Dosbox and SCUMMVM games launched through Minigalaxy already use the versions installed on the system if available. This was added in 0.9.2.

[namandixit]

It's true that proton is better than standard wine, but I wouldn't want the client to depend upon Steam.

It won't have to depend upon Steam, see here. The software will have to parse the VDF files that describe the tool itself.

Valve might change how proton is launched in the future, for wine the cli is stable

That is true. One would hope that since third party tools have cropped up that use the Steam Play interface, they will not break compatibility, but there are obviously no guarantees.

Dosbox and SCUMMVM games launched through Minigalaxy already use the versions installed on the system if available. This was added in 0.9.2.

Right, but support for these had to be added manually, while supporting the Steam Play interface will mean that minigalaxy will automatically support any other similar tools in the future (given they are compatible with Steam Play), see luxtorpeda for example.

[namandixit]

I am documenting what I have figured out in regards to this:

There are two files in a Steam Play compatible tool: compatibilitytool.vdf and toolmanifest.vdf.

Sample `compatibilitytool.vdf:

"compatibilitytools"
{  // C++ style comments seem to be supported
  "compat_tools"
  {
    "Tool_s_Internal_Name"
    {
      "install_path" "." // The directory in which the commands mentioned in `toolmanifest.vdf` exist.
      "display_name" "Fancy Name for fancy people"
      "from_oslist"  "windows" // Boilerplate?
      "to_oslist"    "linux" // Boilerplate?
    }
  }
}

Sample `toolmanifest.vdf:

"manifest"
{ // `/run_command` below stands for whatever script/executable the tool consists of
  "commandline" "/run_command" // To simply run the game (exists relative to `install_path` above)
  "commandline_waitforexitandrun" "/run_command --wait-for-exit-and-run" // Wait for the process to quit?
  "commandline_getnativepath" "/run_command --get-native-path" // Return Unix-style path of game executable mounted inside Wine bottle (only applicable for Proton)
  "commandline_getcompatpath" "/run_command --get-compat-path" // Return Windows style path of game executable mounted inside Wine bottle (only applicable for Proton)
}

A usual command takes the form of:

STEAM_COMPAT_DATA_PATH="${ProtonPrefix}" "${install_path}/${commandline}" getnativepath "${ExecutableFile}"

For example,

STEAM_COMPAT_DATA_PATH="${HOME}/.local/share/Steam/steamapps/compatdata/3830/" ./proton getnativepath "${HOME}/.local/share/Steam/steamapps/common/Psychonauts/Psychonauts.exe"

STEAM_COMPAT_DATA_PATH holds the path to the directory with Proton prefix with some extra metadata. This should be different for each game.

What do we need to implement?

For simple use cases, all that needs to be done is to parse the two vdf files, find the full path of run_command above, create a unique directory for the executable passed to it (maybe hash the EXE) to be set as STEAM_COMPAT_DATA_PATH, and then run a command like the above example to launch the game.

Pinging @dreamer, is the above correct?

@sharkwouter, will you be interested in taking PR for this?

[dreamer]

@namandixit You are 99% correct. Few tiny (but important) details:

  "from_oslist"  "windows" // Boilerplate?
  "to_oslist"    "linux" // Boilerplate?

These are not boilerplate, they actually instruct Steam which depots to use for installing game resources (so theoretically it's possible to write a compatibility tool, that runs macOS games on Linux or the other way around). In context of non-Steam launchers - it might be relevant only for installation/post-installation steps.

manifest file:

"commandline" "/run_command" // To simply run the game (exists relative to `install_path` above)
"commandline_waitforexitandrun" "/run_command --wait-for-exit-and-run" // Wait for the process to quit?

commandline can be used for running the game, but usually Steam uses it to trigger various installation steps (like triggering installation of DirectX for some old Windows games or injecting some values into the Wine registry). It would be advisable for non-Steam launchers to not use it for starting the games.

commandline_waitforexitandrun Steam uses this command to run the game. "wait for exit and run" is there to accomodate usecases like:

• waiting for previous game proces to stop, so that cloud saves are properly synchronized before starting the same game again
• run the game as soon as installation steps for this SteamAppId are finished

(there are more details about how SteamAppId is passed from Steam to compatibility tool to accomodate these usecases)

getnativepath and getcompatpath - I've never actually found the exact purpose of these commands.

Supporting this interface would assure that tools like Boxtron, Luxtorpeda or various Proton forks will work "out of the box", without launcher having to support them explicitly.

@sharkwouter Supporting this interface does not mean depending on Steam - but it'll allow for supporting community-developed Proton forks and other compat. tools. The annoying part is parsing Valve's VDF format, I guess.

Dosbox and SCUMMVM games launched through Minigalaxy already use the versions installed on the system if available. This was added in 0.9.2.

For ScummVM it is perfectly reasonable; I have developed Roberta to have Steam compat. tool, for this usecase, but ScummVM offers nice interface for autodetection and running the games on its own.

For DOSBox, unfortunately, it is not enough :( Simply invoking dosbox command with parameters supplied by Steam or GOG will fail in an overwhelmingly large number of cases. There are many reasons for that (I can explain in detail if needed) - in Boxtron I tried to work around as many issues as I could, but eventually I hit the wall - that's why I am working on fixing the remaining ones in dosbox-staging (but I suspect my improvements will never be merged into upstream).

[sharkwouter]

@dreamer could you be a bit more specific on the issues with DOSBox? All the games I have tested have worked with the version in Debian 10, but barely any of them are able to launch with the GOG supplied DOSBox version(expected library versions don't match). Do you know some examples of games which do not work like this?

To clarify how it currently works. I use the same command to launch DOSBox the start.sh script would use but with the system's DOSBox instead. I do make sure the working directory is set to the game's directory as well. So far I haven't received any bug reports about it yet.

@sharkwouter, will you be interested in taking PR for this?

@namandixit I will take a PR, but since I want to make sure to not change the user experience too much, it would need to meet the following requirements:

• If it adds additional dependencies, they should be available in the repositories of the supported distributions
• Using proton should be optional, since certain games might not run better on it compared to the wine version shipped by GOG. This could be done by adding an option to preferences
• It should probably be disabled by default
• By default, Minigalaxy should still only show games which have Linux downloads (Not sure if being able to run all Windows games is your goal here, so I wanted to include this)
• All strings, except for error messages in the log (currently that's stdout, still need to implement a better system), have to be translatable (this is a requirement for all PRs)

I think it's a very interesting concept, but I'm not sure if it is implementable without adding dependencies which aren't widely available. If it turns out using the system's DOSBox doesn't work like expected I'm even more interested, though, since what GOG ships doesn't work on a lot of systems.

[dreamer]

@sharkwouter sure:

• DOSBox .conf files are not portable - mounting directories and images requires platform-specific path to mount/imgmount command; upstream DOSBox maintainers refuse to acknowledge it as a problem. Boxtron translates Windows paths left by publishers into Unix paths (original games use DOS paths, so in practice there's never a conflict unless user manually creates a conflicting file).
• .cue files for mounting image files (e.g. for CD-DA emulation) also are platform-specific and have inter-dependency on current directory mounted inside dosbox (while still using platform-spacific paths) (Boxtron does the translation for these files as well)
• .cue files referencing .mp3 and .flac files cause various kinds of breakage - besides DOSBox bugs (fixed in dosbox-staging) this is often result of using SDL_sound 1.2 (which is basically unmaintained nowadays). dosbox-staging fixes all known problems in this area.
• MIDI synthesiser port detection does not work at all on Linux (Boxtron scans open ALSA sequencer ports and redirects DOSBox to the correct one)
• fullscreen does not work correctly when using more than 1 display (this is SDL 1.2 issue with an easy fix - upstream does not acknowledge the problem, Boxtron implements workaround, dosbox-staging avoids the problem by using SDL2)
• alt-tab does not work at all - fixed in dosbox-staging
• DOSBox 0.74-3 defaults to output=surface which has serious performance implications - fixed in dosbox-staging (uses output=texture which defaults to SDL2 2D acceleration using OpenGL as backend)
• option aspect defaults to false because of scaling algorithm deficiencies. In result almost all games from early 90s are stretched horizontally (Boxtron has workaround, we're working on a fix in dosbox-staging)
• users who want to inject overlays (like Steam overlay or some FPS overlays) experience a range of issues (fixed in dosbox-staging by using SDL2)
• random keyup events are being injected to some games when playing in fullscreen mode (it makes some games totally unplayable: Tomb Raider I, Megarace). Fixed in dosbox-staging.
• modern controllers do not work out of the box SDL 1.2 does not support them, sooo… situation in dosbox-staging is significantly improved via SDL2 (however, I believe it's not fixed completely yet)
• autodetection of cycles is too conservative, leading many users to believe that DOSBox is slow when in fact dosbox often autoconfigures itself to use cycles=3000 which utilizes like 5% of modern CPU - high-profile games experiencing it are e.g. Wolfenstein 3D or X-Wing series.
• DOSBox 0.74-3 does not support overlay mounts; GOG team submitted patches upstream and they were accepted ~2 months before 0.74-3 release, but they were merged only to trunk, not to 0_74 branch. dosbox-staging supports overlays.

So these were DOSBox issues… now publisher bugs - these are common only for DOSBox games distributed via Steam:

• publishers often hardcode platform-specific options in bundled .conf files (like output=ddraw) or options for their own forks of DOSBox (it's easier to fork DOSBox than to cooperate with upstream). Boxtron overrides those with a sane default configuration with some game-specific tweaks.
• publishers tend to misuse/misconfigure cycles option (probably as a result of outdated DOSBox manual); Boxtron corrects by changing default cycles value - in result games from 90s work correcly instead too slow, but games from 80s run too fast instead of "just ok".
• many publishers wrap their DOSBox games with .bat files to invoke the game, sometimes with various workaround inside, and these are universally Windows-specific - Boxtron implements an interpreter handling a small subset of .bat functionality, that is able to invoke the Linux version of DOSBox for such games

(probably like 10 smaller issues I forgot about) ;)

[sharkwouter]

Oh wow, that makes the DOSBox upstream sound like quite a pain. Have you had no success getting patches into DOSBox at all?

So if I understand correctly DOSBox configurations are portable, but only between systems which run the same operating system? That would mean the current solution should always work, since Minigalaxy only installs Linux games at the moment.

If we want to make Windows games installable as well, this solution will not work.

[dreamer]

Oh wow, that makes the DOSBox upstream sound like quite a pain. Have you had no success getting patches into DOSBox at all?

I am trying for several months now (link) :( So far out of 78 patches upstream accepted only few trivial ones (some patches got completely rewritten). There is no estabilished review nor contributing process… Co-maintainer of dosbox-staging tried his patches for several years without proper feedback from upstream.

So if I understand correctly DOSBox configurations are portable, but only between systems which run the same operating system? That would mean the current solution should always work, since Minigalaxy only installs Linux games at the moment.

That is correct. If minigalaxy installs only Linux games, then many (not all) problems I mentioned go away (generally all SDL 1.2 - related issues are still present). There are very rare cases of games which provide Linux version and broken .conf file (e.g. Megarace 2), but they are exceptions.

Sometimes DOS games are not published for Linux at all. On Steam it depends on publisher - e.g. Bethesda never publishes on Linux (even classic id games like Doom).

On GOG, lack of Linux version of DOS game usually signifies that DOSBox had some non-obvious bugs preventing the game from being playable (e.g. Mortal Kombat 3 had game-breaking performance problems, but only after 1-2 rounds - in Boxtron+dosbox-staging it works without serious issues) or publisher bundled some Windows software (Warcraft 1; works great in Boxtron).

If we want to make Windows games installable as well, this solution will not work.

From Steam: nope - Steam will always install the game according to user settings inside the Steam client.

From GOG: I have implemented a script to help me install Windows versions of DOS games from GOG (after manual download) - it seems like a good fit for minigalaxy (after some changes, I believe); this script is not part of main Boxtron functionality, only a helper script to help me with testing; if I could replace it with minigalaxy for my usecase I would be quite happy :).

I was also cooperating with maintainer of GameHub for a while to enable integration of that launcher with Boxtron/Roberta/Luxtorpeda, but that effort stalled.

[namandixit]

@sharkwouter There shouldn't really be any dependencies. Only new component we need is a VDF parser; however, the relevant VDF files don't use a whole lot of features of the format, so writing a rough-and-ready parser is pretty easy.

Since I am not very familiar with GTK, I can write up the logic first (making sure that it stays compatible with the present GUI) and submit it as a PR, then work on the exposing the functionality through GUI later. Will that be acceptable?

[sharkwouter]

@namandixit that sound pretty good.

If it helps, I did start writing some developer documentation, but it still needs a lot of work. I'll see if I can write a part of adding a setting to the preferences windows next weekend. Honestly, I was not expecting more people would use this code when I started the project 😅. It is really cool to see people investing into it.

@dreamer thanks for the detailed description! That gives some great insight into DOSBox. It makes a lot of sense to me now that you started DOSBox-Staging.

[kekonn]

I just ran into wanting this feature and I might be able to offer an idea. GloriousEggroll makes a wine version that has all his proton stuff packed into a wine prefix. If you package that version by default, that will basically make a lot more games compatible.

EDIT: link to the version