qsniyg / ksp_stuff

GNU General Public License v3.0
9 stars 4 forks source link

Option to create hardlinks instead of symlinks #32

Closed smirgol closed 3 years ago

smirgol commented 3 years ago

If interested, I've patched your script to create hard-links instead of symlinks. It's triggered with: --hard_links. If omitted, it will create the usual symlinks.

It can be omitted when calling --unvfs, as the option is written to the logfile, so hard-links can properly be cleaned up if the user used that option when creating the links but forgot to specify the option when asking for a clean-up. Of course it will only work if both src and dst are on the same drive, I have not added a check for that, might be a good idea though...

I also added a few calls to close() whenever a file has been opened for reading/writing. Not sure if absolutely necessary, but I thought it might be a good idea. Then I've replaced the calls to create/check/remove links with new functions, as I needed to add additional logic to the checks to handle hard-links and its cleaner that way. Finally, the log file is now read earlier, as I require its data for the various checks in the new functions. I left in the additional check in you function when removing the vfs, but moved the reading of the log to the main function.

The pull requests consist of 2 commits, as I forgot to remove some debug prints() on my first commit, as usual. :)

qsniyg commented 3 years ago

Wow thank you very much! I'll take a closer look at it soon and hopefully merge it! :D

smirgol commented 3 years ago

Thank you. I'm still playing around with it, but so far I'm super happy with your script! I have added roundabout 50 mods with a total of ~50k hard-linked files and it works like a charm. Still trying to figure out why the FNIS stuff breaks when I unlink+relink, but I will find out. :) I added an additional check before unlinking a file, because it happened to me that a linked file got removed and it broke the script. Then I'm trying to not overwrite files that exist already, because I had a mod overwrite some SKSE script files with an old version - but I'm not yet sure if that is a good idea. Works for me, but I need to put more thoughts into that.

smirgol commented 3 years ago

I added your proposed changes and added another enhancement - it will now save a timestamp when the linking was done and when unlinking, it will skip all files in the data directory that were changed after the linking. This should preserve FNIS and CBBE data that is generated after the linking. It will print those files when the unlinking is done, not sure yet if we should do that or just silently skip those files.

qsniyg commented 3 years ago

I agree, printing those files seems like a good ea, that way the user is aware of it. Looks good to me, thanks again!

smirgol commented 3 years ago

Hm one question: I've noticed that my patch to preserve existing files does not work, because you rename existing destination files in updatelink(). Is there a special reason that you do that? I'd remove it, but maybe there is a good reason for this and I should not touch it?

qsniyg commented 3 years ago

You're referring to the .unvfs files? This is in case there were files that were already present in the Data folder, which the VFS has overwritten.

smirgol commented 3 years ago

Yes. My question would be if there is a good reason to backup the files to .unvfs before overwriting/linking. I get that it's generally a good idea to backup things before overwriting, but is it a pure pre-cautious thing or is there a real use-case why you do it?

Theoretically speaking there is nothing in a game directory that is going to be overwritten by the mod files. It's a blank game with only the game data files, which should not be overwritten by mods anyway. So, again theoretically speaking, there should be no reason to backup things.

My point is, when I have linked everything, create the FNIS and CBBE files and then unlink, right now it will keep those files so you don't have to re-create the files when you link again. This was an issue before, but I fixed that with checking the timestamp of the files before unlinking and keep those in "Data", that were created/modified after linking (usually the FNIS and CBBE files). So far, so good. But when you re-link the mods, it will backup those modified files and copy in the old files from the mod, which forces you to re-do the FNIS and CBBE.

I did disable the backup in my script and for me it works good - I can link+unlink as much as I want, it keeps the FNIS and CBBE files. But before trying to push that change I wanted to know if there is a reason for the backup that I might be missing. :)

qsniyg commented 3 years ago

Maybe then we can add an option? Like --overwrite_existing, which would keep the current behavior of backing it up, and if it's not present, then it'll avoid overwriting?

smirgol commented 3 years ago

Sounds good to me, I'll add that. :)

ajventer commented 3 years ago

I added that feature originally when I first developed this script. The reason for the .unvfs files is to support users who choose to install some mods manually or with a different mod manager, if an MO mod then overwrites a part of a such a mode - then reverting the script should restore the original. Consider for example mods that are only available via steam workshop, if one of those retextures an object, but the user then installs a texture pack that also retextures the same object via MO2 - then the CORRECT behaviour should be that, with the script run the MO2 mod's file is present, but if you do an unvfs and then run the game the steamworks mod's version should be present, and if you uninstall that was well only THEN do you get the skyrim original.

But that was the intention behind it: to support files that come from mods NOT installed by MO2 but which may be overriden by MO2 mods. This is a particularly big issue with some older games like Oblivion and such where some mods can ONLY be installed by other mod managers.

smirgol commented 3 years ago

I see, thank you for clearing that up! :) That behaviour should be preserved with the new --overwrite_existing flag. By default it will not rename files and will not overwrite any existing file that is neither a sym- nor a hardlink, that really helps with things like FNIS and BodySlide, as it keeps the files modified by them so you don't need to re-run it every time you relink. Question would be what makes more sense to make it the default behaviour.

ajventer commented 3 years ago

I've been experimenting a bit with it, I think the don't overwrite by default works well - but be aware that it prevents installing certain mods - like UPOT for example which overrides the texture BSAs from skyrim itself, if you (like me) only have one affected mod - the easiest way is to back the affected files up yourself and just remove them from the target directory.

ajventer commented 3 years ago

Your keeping of files created post linking is problematic, it turns out that Skyrim itself regularly updates the timestamps on a bunch of files - for example every SKSE plugin, and that means that these files don't get unlinked either. That means if you later uninstall a mod that added such a file - the files get left behind in your game. I've specifically caught this with SKSE plugins and scripts. I suspect the timestamp on these get updated whenever they are loaded. I don't know what else may be affected though.

smirgol commented 3 years ago

Hm interesting observation, thank you. I didn't notice that with Skyrim SE. After unlinking 113k files from roundabout 120 mods all that is left are the files changed by BodySlide, FNIS and of course all log and configuration files of the plugins/mods along with their various folders, which do pollute the Data directory but should cause no harm. The mod files were all removed otherwise.

Admittedly there are two FNIS pex files in Data/Scripts, but when there is no FNIS, they should not get loaded, so I assume. But that's all I could find. One contains a script to return the FNIS version, the other some "AA" lists, whatever that is good for. It's not exactly nice of FNIS to store this information in dynamic script files, which could easily be written to a config file, but that's the way it is done.

Well, it's complicated I think. You cannot have one without the other. You either re-run BodySlide and FNIS and similar things after each linking, which can get really annoying when you're installing mods incrementally or need to narrow down a mod error by removing/adding mods, which requires relinking, or you keep the changed files, risking that it will pollute your Data directory.

One could be more specific as for which files to keep, but I think that's a too specific approach that will break in some unforeseen cases. Another thing could be to restore the default behaviour of renaming files and make keeping the files an option, so it's more fail-safe for the regular user. A last idea could be to log all the files that were not removed when unlinking and add an option to purge these, but that's not without risks either. A game update would mess that up and while one possibly could circumvent that for Bethesda games, where the game data files and their location are pretty much known, it won't work for other games.

I'm open to suggestions. :)

ajventer commented 3 years ago

This may come down to what modding style and tools you use. FNIS is really deprecated now, Nemesis does everythign FNIS did and does it much better -while opening up mods that FNIS can't run like Combat Gameplay Overhaul. But Nemisis creates a LOT of scripts and meshes. Personally I don't mind re-running nemesis and bodyslide after every linking - it takes about 90 seconds to do both on my SSD, the linking itself takes a lot longer (about 55-thousand files for me), and that's nothing compared to the more than 5m required to re-build my bashed patch which you always have to do regardless.

Personally I commented that block out - and it finally allowed me to completely uninstall VIGOR which, even after uninstalling, was still causing injuries in-game because of some script left behind.

One way you can avoid rerunning those tools unless you install mods that require it is to use a FUSE overlay filesystem first, so when you run say BodySlide it thinks it's writing to your Data folder but in fact all the files it changes go into a different folder. Then you create an empty mod in MO2, move all the files from that folder into the empty mod and activate it. Much like you would have done using MO2 overwrite directory in normal use.

smirgol commented 3 years ago

Hm I see.

So I'm proposing two things to restore the initial behaviour:

  1. Change the default behaviour from --overwrite_existing from False to True, so it will backup existing files by default and replace them.
  2. Add a new option --keep_modified_files, which default to False, to keep any file that initially was linked by the script, but got overwritten afterwards.

So, if you run it without those two parameters, it should behave like it initially did, but giving you the option to not overwrite files when linking and keep modified files when unlinking.

If you have chosen to overwrite existing files and backups were created, when you unlink and specify to keep modified files and the file was actually modified, it will delete the backup file instead of restoring it. If there was no modification to the linked file, the backup will be restored.

But, and I might be wrong here, it never did remove any files that were not linked initially, right? So any log/configuration/script/... files do remain one way or another, because the script simply does not know about them, so it can't safely remove them.

Good information about Nemesis, I'll look into this. I didn't even knew about FNIS until I started modding Skyrim, it wasn't a thing with Fallout 4 which I usually play.

ajspandigital commented 3 years ago

Personally I just commented out the datestamp check block :) but that sounds like a good sollution, and if you are happy with your version of the behavior then it will also support that.

smirgol commented 3 years ago

Forgot to push the changes... I added a new flag keep_modified_files which default to False and changed default behaviour of overwrite_existing from False to True. It should now behave like it initially did, but giving you the option to keep the files modified by BodySlide, FNIS or whatever.