Ability to install directories and files into GameRoot directories

linuxgurugamer commented 4 years ago

This is needed for the SoundtrackEditorForked (see KSP-CKAN/NetKAN#7666). It needs to put a few directories in the game root, as well as one directory/file into the KSP_Data/ (or KSP_x64_Data)

See also KSP-CKAN/NetKAN#5953, in which the original fork of this mod was found to be installed wrong and frozen.

HebaruSan commented 4 years ago

This was done deliberately to discourage installation of files into GameRoot:

https://github.com/KSP-CKAN/CKAN/pull/83#discussion_r19053307

Relevant bits of the spec:

GameRoot (which should be used sparingly, if at all).
A path to a given subfolder location can be specified only under GameData (v1.2);
Subfolder paths under a matched directory will be preserved, but directories will only be created when installing to GameData, Tutorial, or Scenarios.

DasSkelett commented 4 years ago

I see two clean solutions for this: 1) Create a special handling for KSP_Data. For a stanza like "install_to": "KSP_Data/Mono" CKAN automatically decides based on which OS it is run on whether it should go into KSP_Data/Mono or KSP_x64_Data/Mono. This assumes the folder structure in this two directories is the same (I hope so). 2) Create a os property. CKAN installs mods with this property set only if it matches the current OS (and always if it's not set at all).

HebaruSan commented 4 years ago

This mod's current release structure could not be indexed correctly anyway (same version number assigned to two different downloads for different game versions):

https://github.com/linuxgurugamer/SoundtrackEditorForked/releases/tag/4.6.1.2

Also the ZIP files appear to be misbuilt, every top level folder contains a copy of itself with duplicated contents:

Before we make any changes here, I think we should dig in and understand what this mod is doing and make sure that it has to be done this way.

Could these DLLs simply be installed to the root of GameRoot?

    {
      "find": "KSP_Data",
      "install_to": "GameRoot"
    },
    {
      "find": "KSP_x64_Data",
      "install_to": "GameRoot"
    },

For reference, AdvancedFlyByWire installs two DLLs directly into GameRoot in what looks like a similar situation:

https://github.com/KSP-CKAN/NetKAN/blob/master/NetKAN/AdvancedFlyByWire.netkan

    "install": [ {
        "file":       "SDL2.dll",
        "install_to": "GameRoot"
    }, {
        "file":       "SDL_LICENSE",
        "install_to": "GameRoot"
    }, {
        "file":       "XINPUTDOTNET_LICENSE",
        "install_to": "GameRoot"
    }, {
        "file":       "XInputInterface.dll",
        "install_to": "GameRoot"

And do these folders need to be in GameRoot, or could they be in GameData? It looks like Music contains audio assets used by the mod and Playlists contains some .cfg files, and it's not clear why those would need to live outside GameData.

    {
      "find": "Music",
      "install_to": "GameRoot"
    },
    {
      "find": "Playlists",
      "install_to": "GameRoot"
    }

linuxgurugamer commented 4 years ago

The reason for both the KSP_Data and KSP_x64_Data is to support Linux & Macs, which don't use the x64 folder. The intention is to only install one depending on the system, but I hadn't gotten that far.

Second, the DLLs need to be installed where they go because that is the only way they will work. I don't pretend to understand the reasons behind that right now.

Finally, the folders "Licenses". "Music", & "Playlists" could certainly be moved, if that was the only thing stopping this from being done.

This is actually a very old mod, and those folders are there because there wasn't a GameData directory when originally written. I have no problem rework that if requested and needed

JonnyOThan commented 1 year ago

This would help with KerbalVR too!

JonnyOThan commented 2 months ago

I assume there's not a big downside to installing both KSP_Data and KSP_x64_Data; though DasSkellet's proposal for a new install_to option would be nice.

The Music and Playlists folders should definitely be moved into GameData (possibly under a PluginData folder if you don't want KSP to automatically load/process them)

HebaruSan commented 2 months ago

Second, the DLLs need to be installed where they go because that is the only way they will work. I don't pretend to understand the reasons behind that right now.

Hi @linuxgurugamer, has there been any progress on understanding this?

JonnyOThan commented 2 months ago

So trying to install both directories to GameRoot didn't quite work, because it refuses to create subdirectories:

https://github.com/KSP-CKAN/NetKAN/actions/runs/10081264830/job/27872895738?pr=10116

But we could let it through with just the x64 one, or use DasSkellet's idea.

HebaruSan commented 2 months ago

Create a special handling for KSP_Data. For a stanza like "install_to": "KSP_Data/Mono" CKAN automatically decides based on which OS it is run on whether it should go into KSP_Data/Mono or KSP_x64_Data/Mono. This assumes the folder structure in this two directories is the same (I hope so).

I've been looking into this a bit, and there are some complications here that call into question the viability of this suggestion:

"based on which OS it is run on" would fail for Proton-based Windows game instances on Linux, so it would have to be based instead on which directories actually exist in the game dir
Sounds like the files to install might have to be OS-specific, which means having a single install_to directive for one consistent set of files wouldn't necessarily work anyway? And if we tried to install OS-specific files, Proton again makes that awkward (presumably CKAN would need to know to select the files for Windows on Linux, somehow?).
Macs use KSP.app/Contents/Resources/Data, not either of the folders mentioned so far (not a deal-breaker, just noting for the record so we don't get tunnel vision with just the two options)

At this point, finally cleaning up SoundTrackEditor (migrating from libmpg123 to some Unity API, moving the folders from game root to GameData like every other mod) is looking like a superior option to doing tons of confusing coding in CKAN to enable a bunch of hacks.

HebaruSan commented 2 months ago

This would help with KerbalVR too!

Elaborating for future reference (only Windows DLLs seem to be included):

https://github.com/FirstPersonKSP/Kerbal-VR/releases

linuxgurugamer commented 2 months ago

"The reason for both the KSP_Data and KSP_x64_Data is to support Linux & Macs, which don't use the x64 folder. The intention is to only install one depending on the system, but I hadn't gotten that far."

It doesn't hurt to install both, only one will be loaded. Probably given that the 32 bit code has been gone for a long time,could probably just install to the KSP_x64_Data, but that would require testing

Second, the DLLs need to be installed where they go because that is the only way they will work. Essentially, that's on the OS, since we don't have access to the KSP source code (might even be Unity, not sure), we can't tell the game where to load the real DLLs (native code, not C# compiled code).

However, if it's just a matter of the directories not being able to be created, I wouldn't mind modifying the soundtrack mod to put them somewhere else.

HebaruSan commented 2 months ago

It doesn't hurt to install both, only one will be loaded. Probably given that the 32 bit code has been gone for a long time,could probably just install to the KSP_x64_Data, but that would require testing

If you're talking about support for non-Windows users, then no, that's wrong. Linux uses KSP_Data because it never had a 32-bit build, and Mac uses KSP.app/Contents/Resources/Data as noted above.

Second, the DLLs need to be installed where they go because that is the only way they will work.

Is that true, though? It sounds like so far nobody has actually tested this, and instead we've been going by guesses and assumptions; notably AdvancedFlyByWire is installing analogous files to game root and seems to work fine on Windows. I would really like to see at least some minimal effort put towards investigating whether this mod actually needs this feature before we add it.

JonnyOThan commented 2 months ago

we can't tell the game where to load the real DLLs (native code, not C# compiled code).

Well, you can, Principia does this. It's just more convenient to dump them somewhere that it's already loading things from (note that principia also spews benign imageformatexceptions when installed. Perhaps these could have been suppressed if they put the native libraries in PluginData, I'm not sure.

What if we just changed these rules slightly?

A path to a given subfolder location can be specified only under GameData (v1.2); for example: GameData/MyMod/Plugins. The client must check this path and abort the install if any attempts to traverse up directories are found (eg: GameData/../Example). Subfolder paths under a matched directory will be preserved, but directories will only be created when installing to GameData, Tutorial, or Scenarios.

That is, allow installing to KSP_x64_Data, KSP_Data, and whatever godforsaken place MacOS loads native libs from. CKAN should just install all of them as specified without trying to figure out what platform the game is actually running on.

And yeah, strictly speaking, mods that only support windows don't need any new features at all from CKAN because they can just dump things into the game's root directory (I'm pretty sure it's a windows "feature" that the exe file's location automatically gets added to the library search path. I have no idea if Linux and MacOS do the same).

HebaruSan commented 2 months ago

That is, allow installing to KSP_x64_Data, KSP_Data, and whatever godforsaken place MacOS loads native libs from. CKAN should just install all of them as specified without trying to figure out what platform the game is actually running on.

Again, this fails when those directories are specified and don't already exist.

JonnyOThan commented 2 months ago

Again, this fails when those directories are specified and don't already exist.

Er, no? CKAN could create them? The game will ignore the ones that are for other OSes.

HebaruSan commented 2 months ago

Well now we're back to the core unanswered, surprisingly simple question: Is that actually needed?

linuxgurugamer commented 2 months ago

It doesn't hurt to install both, only one will be loaded. Probably given that the 32 bit code has been gone for a long time,could probably just install to the KSP_x64_Data, but that would require testing

If you're talking about support for non-Windows users, then no, that's wrong. Linux uses KSP_Data because it never had a 32-bit build, and Mac uses KSP.app/Contents/Resources/Data as noted above.

Second, the DLLs need to be installed where they go because that is the only way they will work.

Is that true, though? It sounds like so far nobody has actually tested this, and instead we've been going by guesses and assumptions; notably AdvancedFlyByWire is installing analogous files to game root and seems to work fine on Windows. I would really like to see at least some minimal effort put towards investigating whether this mod actually needs this feature before we add it.

Admittedly I'm not an expert on DLLs, if Principia and ABFW can do it then obviously it's possible.

Among other things, ABFW has different versions for each OS. Maybe just require that any mod that has OS specific files needs to have a separate install for each OS?

HebaruSan commented 2 months ago

It sounds like nobody's going to actually investigate this. If that does happen, and if the investigation indicates that these DLLs don't work when installed in game root, please let us know and we can re-open this.

JonnyOThan commented 2 months ago

Right, I think a study of the use cases would be useful:

SoundtrackEditorForked

Uses libmpg123-0.dll. I'm fairly certain that since the file in KSP_Data is identical to the one in KSP_x64_Data, it doesn't actually work on linux. There are builds of this library available for linux, but we don't know if the mod just works by replacing that file. So, right now, we can assume that this mod is effectively windows-only.

Kerbal-VR

Uses native openvr dlls, only ships with windows versions. Like libmpg, there should be versions of these libraries available for other operating systems but no one has asked for them. As such, effectively windows-only.

AFBW

Uses SDL and XInput. Dumps native libraries into GameRoot. The windows flavor also contains XInputDotNetPure.dll inside the mod's GameData folder (presumably this is a manged DLL interface to XInput? No idea what it would do if installed on linux). Has alternative download for linux: does not include native windows libraries nor the XInputDotNetPure.dll. The mod's manged dll is also not bitwise identical but I don't know if there's actually any differences in the code. According to documentation, requires that the user manually installs SDL on their system.

Principia

Does their own thing by dumping native libraries for all 3 platforms into their GameData folder and manually loads them as appropriate. Spews benign BadImageFormatExceptions as KSP's assembly loader tries and fails to load them.

Craft2FBX (unreleased)

Uses UnityFbxSdkNative. Seems like there is a build of this available at least for mac, not sure about linux. But I don't have a way to test or support them, so effectively windows-only.

So, it's probably fine for windows-only mods to just continue dumping DLLs into GameRoot. If Principia ever wants to integrate with CKAN or if one of the above mods wants to actively support a non-windows OS we should revisit this.

KSP-CKAN / CKAN