[Feature]: Symlinks for GameData

[ProgrammerFailure]

Problem

When using multiple KSP instances, the squad & squadexpansions folders, along with some others, are duplicated, wasting space.

Suggestion

My idea is for that all mods in a specific instance, and any save files etc, would be saved outside KSP. Then, when you click "Launch Game", the data for that instance would be symlinked in.

For example, I have a KSP installation in C:/Games/Kerbal Space Program. I create a instance using this in CKAN, named "KSP test". When I install a mod, it goes to %appdata%/Local/CKAN/KSP test/GameData. Inside the CKAN/KSP test you would also have saves and ships.

Now, I click launch game in CKAN. Symlinks are created from %appdata%/Local/CKAN/KSP test & its folders to the respective folders in C:/Games/Kerbal Space Program. When I play the game, KSP sees exactly what it usually would and runs accordingly.

The advantage of this is saving 3-4 GB of space per installation, which is quite significant. It also isolates the actual KSP folder, which might be useful to prevent needing a reinstallation (unsure of this part).

I've been doing something similar with .bat files myself, and so far it works well!

Alternatives

No response

Additional context

I believe most Skyrim mod installers do something similar to this, which could be useful for some help.

[JonnyOThan]

Hmm, I'd actually propose a slightly different solution:

The "create new instance" feature could be extended to set up a lightweight instance that just symlinks certain folders back to some other instance. That way, there's no ongoing work to swap out stuff when you decide to launch the game, which would likely be a big maintenance burden and possible point of failure.

I already do this locally with a few of my instances, symlinking GameData\Squad and KSP_x64_Data, etc.

[HebaruSan]

Sounds like that would fit well as a checkbox here:

I wonder if .NET has cross-platform support for symlinks...

[HebaruSan]

... nope. There's a File.CreateSymbolicLink, but not for us:

[ProgrammerFailure]

... nope. There's a File.CreateSymbolicLink, but not for us:

You could have CKAN create, run, and delete a .bat file to make the symlinks? Or whatever the MacOS/Linux version is.

The .bat file would need to:

• delete all previously created symlinks (delete any folder except Squad and SquadExpansion in GameData)

• Add a symlink from inside an instance-specific mods folder to GameData for each mod (instance/mods/modA to GameData/modA)

-Somehow deal with mods creating files directly inside GameData

That last point is the main weakness of symlinks. For example, ModuleManager creates a MM.configcache file in GameData. This file won't go into the instance specific folders; it'll stay in the actual GameData folder, messing up tons of other stuff. IIRC, the same is true for Wild Blue Playmodes.

[JonnyOThan]

You could have CKAN create, run, and delete a .bat file to make the symlinks? Or whatever the MacOS/Linux version is.

Maybe..but probably also lots of issues related to OS permissions etc.

The original proposal is likely prohibitively complex. But a lightweight instance where the unchanging large stock files are shared certainly seems feasible. At least as long as we create the symlinks when creating the instance.

[HebaruSan]

Hmm, that comment was a duplicate when I deleted it. I'll see if it'll let me undelete...

... nope, doesn't look like it. Sorry about that, @ProgrammerFailure, total permanent removal wasn't my intention.

[ProgrammerFailure]

Wait, my comment got deleted by accident. The idea was that stock files are stored by CKAN, symlinked into the mods folder, then the whole mods folder is symlinked into KSP root named as GameData.

[HebaruSan]

I have to say I am firmly, resolutely against messing with any stock files. That is a recipe for absolute disaster with the slightest unexpected bug, and "CKAN bricked my install" reports all up and down the bug tracker.

[JonnyOThan]

Wait, my comment got deleted by accident. The idea was that stock files are stored by CKAN, symlinked into the mods folder, then the whole mods folder is symlinked into KSP root named as GameData.

What does this achieve that symlinking certain stock directories/files back to another instance doesn’t? Because one of these is orders of magnitude more complex than the other.

[HebaruSan]

Realistically, the audience for this feature would probably be a fraction of the people who could homebrew it themselves in a couple of minutes. It's a fun thought experiment, though.

Also, imagine a typical user's thought process:

1. Cool, CKAN can run the game out of a much smaller folder! I can save so much disk space!
2. (Makes a symlinked game instance)
3. (Deletes the original instance where the symlinks point)
4. WTF, why doesn't the game run anymore????

[ProgrammerFailure]

Wait, my comment got deleted by accident. The idea was that stock files are stored by CKAN, symlinked into the mods folder, then the whole mods folder is symlinked into KSP root named as GameData.

What does this achieve that symlinking certain stock directories/files back to another instance doesn’t? Because one of these is orders of magnitude more complex than the other.

It's not too complicated; you only need three symlinks:

CKAN/BaseGameFiles/Squad to CKAN/Instances/InstanceA/mods

CKAN/BaseGameFiles/SquadExpansion to CKAN/Instances/InstanceA/mods

CKAN/instances/InstanceA/mods to KSP root, named as GameData.

As to purpose, it makes the CKAN managed mods folder serve as the entire GameData for all KSP purposes. There's a specific reason that just doing mods/ModA to GameData won't work, I'll clarify when I'm not on mobile.

[ProgrammerFailure]

Realistically, the audience for this feature would probably be a fraction of the people who could homebrew it themselves in a couple of minutes. It's a fun thought experiment, though.

Also, imagine a typical user's thought process:

1. Cool, CKAN can run the game out of a much smaller folder! I can save so much disk space!
2. (Makes a symlinked game instance)
3. (Deletes the original instance where the symlinks point)
4. WTF, why doesn't the game run anymore????

Yeah, didn't think about the "customer" support aspect. Or would it be user support?

You are correct that anyone who understands how it works could homebrew it, but the typical user wouldn't be able to.

As for deleting instance and the game doesn't run anymore, could be addressed by a small popup?

[ProgrammerFailure]

I have to say I am firmly, resolutely against messing with any stock files. That is a recipe for absolute disaster with the slightest unexpected bug, and "CKAN bricked my install" reports all up and down the bug tracker.

I mostly agree. However, my proposed solution doesn't change stock files; to restore, the user just has to copy Squad and SquadExpansion back to GameData. That's within most users capabilities surely?

[HebaruSan]

CKAN/BaseGameFiles/Squad to CKAN/Instances/InstanceA/mods

CKAN/BaseGameFiles/SquadExpansion to CKAN/Instances/InstanceA/mods

CKAN/instances/InstanceA/mods to KSP root, named as GameData.

I don't want to crush your dreams here, but: No. We're not moving stock files out of their canonical locations. To pick one reason at random among a vast number...

Steam users. Steam installs those files itself, and if you right click the game and select "verify integrity of game files", it will check them and re-download them to ensure they're valid. CKAN absolutely should not be in the business of disrupting that.

And for a game that has not had its dev team disbanded yet (remember that CKAN supports multiple games and will probably add more in the future), it would be insane to remove such files from where Steam can update them when there are new game versions.

Please let that go lest this suggestion veer into total impracticality.

Another note, there are users who install mods via CKAN and then launch the game via Steam. They shouldn't be locked out of this feature by making it depend on "click launch game in CKAN".

Making the clone-instance feature support symbolic links for stock files would have none of the above problems, but it still faces the lack of cross-platform support for symbolic links in the version of .NET that CKAN uses.

[ProgrammerFailure]

Nah, fair enough. I can do it myself, this was a suggestion, and it had large issues making it impractical. Nothing wrong with that.

I will note that once symlinks are made the game can be launched any way you want; that's not an issue.

I'm gonna implement this for myself, if I figure out a solution that doesn't need modification of game files, I'll post it here.

Edit: I just realized what you meant about clone instance supporting stock files. Yes, that keeps duplication very, very low. That leaves the issue of cross-platform support. I know nothing about that so if you guys are interested, I leave it to you.

[HebaruSan]

I will note that once symlinks are made the game can be launched any way you want; that's not an issue.

That was specifically regarding this in the OP:

Now, I click launch game in CKAN. Symlinks are created from %appdata%/Local/CKAN/KSP test & its folders to the respective folders in C:/Games/Kerbal Space Program. When I play the game, KSP sees exactly what it usually would and runs accordingly.

[ProgrammerFailure]

That was specifically regarding this in the OP:

Now, I click launch game in CKAN. Symlinks are created from %appdata%/Local/CKAN/KSP test & its folders to the respective folders in C:/Games/Kerbal Space Program. When I play the game, KSP sees exactly what it usually would and runs accordingly.

That's for first run/on switching instances. Afterwards, it'll work until symlinks are deleted (ideally on switching instances)

Also, my original idea would have had an issue similar to this. Symlinks on clone instance won't.

[JonnyOThan]

Hey if you find out any good info on programmatically creating symlinks let us know. I have a hunch that we could probably do it with a small amount of platform-specific code, but I haven’t looked into it at all.

[HebaruSan]

Yeah I think it's basically this. Haven't tested yet; the params may or may not work as strings, and not sure what to do about MacOS.

        [DllImport("libc")]
        private static extern int symlink(string src, string dest);

        [DllImport("kernel32.dll")]
        private static extern bool CreateSymbolicLink(string dest, string src,
                                                      SymbolicLink kind);

        private enum SymbolicLink
        {
            File      = 0,
            Directory = 1,
        }

        public static bool MakeSymLink(string src, string dest)
            => Platform.IsWindows ? File.Exists(src)      ?  CreateSymbolicLink(dest, src, SymbolicLink.File)
                                  : Directory.Exists(src) && CreateSymbolicLink(dest, src, SymbolicLink.Directory)
             : Platform.IsUnix && symlink(src, dest) == 0;

[ProgrammerFailure]

Yeah I think it's basically this. Haven't tested yet; the params may or may not work as strings, and not sure what to do about MacOS.

        [DllImport("libc")]
        private static extern int symlink(string src, string dest);

        [DllImport("kernel32.dll")]
        private static extern bool CreateSymbolicLink(string dest, string src,
                                                      SymbolicLink kind);

        private enum SymbolicLink
        {
            File      = 0,
            Directory = 1,
        }

        public static bool MakeSymLink(string src, string dest)
            => Platform.IsWindows ? File.Exists(src)      ?  CreateSymbolicLink(dest, src, SymbolicLink.File)
                                  : Directory.Exists(src) && CreateSymbolicLink(dest, src, SymbolicLink.Directory)
             : Platform.IsUnix && symlink(src, dest) == 0;

Does this cover the name of the resulting symlink? I don't see a parameter for name in there.

[HebaruSan]

@ProgrammerFailure do you mean dest?

[ProgrammerFailure]

No, I mean the name of the symlink. For example, a symlink from A/B/C to 2/3/4 named "blue" would result in the following path: A/B/C/blue, where the contents of blue are the contents of 4.

See this command: https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/mklink . It's the windows symlink command. You'll notice a link field (the symlink is created in the current working directory, so no dest).

[HebaruSan]

Yeah, that's dest. Your example would be MakeSymLink("2/3/4", "A/B/C/blue"). There's no need for an extra parameter to specify the last piece of the path (especially since neither of the underlying platform-specific functions work that way).

[ProgrammerFailure]

Ok, got confused by the naming I guess.

Edit: oh, the last name in dest is automatically the symlink name. Brain fart there.

[JonnyOThan]

I'd suggest using the same names as in the win32 api to avoid confusion: symlinkFileName and targetFileName.

https://learn.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-createsymboliclinka

[HebaruSan]

What about naming the param to CloneInstance that determines whether to copy or symlink the stock files? Right now I have bool shallow = false, but I don't like it very much.

[JonnyOThan]

shallowCopy is fine, or name it deepCopy or fullCopy and default to true. Other possibilities: lightweight or shareStockFiles.

[HebaruSan]

Hoping to avert disasters with a tooltip warning (the accept/cancel buttons are still there underneath):

[JonnyOThan]

Not bad. I’d probably add a note in there about why someone would want to do this, and maybe an estimate of how much it saves per instance

[ProgrammerFailure]

Just to be clear, are you guys planning on adding this? Or is it more of a hypothetical?

[HebaruSan]

Heh, don't expect that kind of clarity until we have a merged pull request that closes this issue. I'm interested in the idea and have been looking at it but I can't yet guarantee that it will work well enough to be added.

[ProgrammerFailure]

Fair enough. If/when you close it GitHub sends me a ln email right? I'd be interested in using the feature myself.

(My computer is VERY limited in disk space. That's the whole reason I suggested it.)

[HebaruSan]

Sure, in fact, I'll tag you in the pull request if/when it's ready so you can test a build with the changes before it goes out to everybody else.