[Feature Request] COW Copy On Write usvfs.

[whraven]

The problem

Files that are updated in VFS are written to the original location instead of the Overwrite directory. This is a major change from previous Mod Organizer behaviour, and invalidates much of the current documentation for modding, when assumes updated files will also appear in Overwrite along with newly created files.

Environment

• Mod Organizer Version that exhibits the issue: 2.1.3
• Desktop OS/version used to run Mod Organizer: Windows 10

Details

Most likely due to changes in how usvfs handles file updates.Workaround would require advance knowledge of what files will be updated and placing them in a new mod or in Overwrite prior to running whatever utility will update them. This affects SSE Edit, Wrye Bash, and RSPatch, and may affect BodySlide and FNIS (I'm not certain because I don't know if they use file updates or only create new files).

Requested Feature

Ability to configure updated file location to Overwrite instead of in place.

[whraven]

This may require modification of usvfs dll's, and may require a new issue for that code as well.

[Al12rs]

You are invited to read the 2.1.2 changelog. We intentionally changed the behaviour to avoid consistency issues. The behaviour before was actually flawed and did not protect the original files at all. If you want to know more and discuss about it I invite you to join the mo2 Dev discord server. Do note that we intend in the future to change the behaviour to make mods completely safe and untouchable by external programs to guarantee integrity, but this kind of change would require major work and still has involved technical issues.

[whraven]

Understood. Were those changes made in USVFS or the MO2 hooks? I consider the planned behaviour you mention an urgent matter for two reasons: one, it invalidates current practice, and two, while some may disagree, I believe protection of original mod contents is important.

[Al12rs]

The changes were done to usvfs. Mo2 does not have any code concerning hooking and virtualization except for the preparation of the mapping of the files to give to usvfs. As I said there are major unsolved obstacles to an approach like that one and sincerely we have studied the cases where the "new" behavior could be negative but in the end the end there was nothing especially bad. We are currently very occupied with personal life and severely undermanned. We would really need more devs to help out as right now it is difficult to stay on top of bugs, major usvfs (one of the most complex and low level libraries you can find) rewritings are not really possible right now. Before usvfs changes are considered again (except for bug fixes of course) there ire still other things that take priority such as BSA conflicts detection and all the gui that comes with it. Feel free to come over at the Mo2 Devs discord https://discord.gg/cYwdcxj to speak with us more

[whraven]

Sadly, I don't do Windows programming, although I do have code in the Linux Kernel and several Linux software packages. However, I am looking at the usvfs code, and perhaps I can help in some way.

[Al12rs]

Any help would be greatly appreciated (and needed). Feel free to ask us things and we will try to answer as well as we can. If you want to know how to set up a dev environment for usvfs you will need to use umbrella (Mo2 build script) probably, and at that point you might as well build all of Mo2

[AnyOldName3]

The long-term solution to this would be to get Kortex's VFS fully working and then (as it uses Dokany) it should be a lot easier to add copy-on-write features to that than to USVFS. Kortex is already slower than USVFS, though, and if it were to gain COW features, I'm not sure the game would be playable while they were enabled.

I know that it's possible to do something like that as I've done it before, but that system was absolutely ridiculously slow to the point that it wasn't even usable with BodySlide, let alone the game.

[whraven]

I'm just starting to dig into the code but I have a few ideas. The primary one is that we might use file filters to monitor for one of two things: first write to a file, or opening a file read-write or write-only (not sure which would be better at this point). At that point, the filter would hold the I/O, copy to Overwrite, and allow the write to continue using the new link. Subsequent writes would involve no overhead. We would probably want the legacy filters to maintain compatibility with Win8 and Win7. However, this is only based on my first look, and filters may not be appropriate from a UAC standpoint. Second, where should the code live? As I said, I'm just starting to look at USVFS (if I'm successful at doing a reverse engineering UML diagram, I'll provide it to the group), but I wonder if the file filter code should exist where the VFS functions are called rather than where they are implemented (or am I talking nonsense here since I don't know the code yet?).

[AnyOldName3]

USVFS only affects functions which create file handles and list which files are in directories. When a hooked application tries opening a file, USVFS just gives a file handle to the real file on disk and lets the application use it normally. That means that we never see the first write to a file as it's already out of our control. That means we can't do anything based on copy-on-first-write.

That leaves copying a file when it's opened with write access. Unfortunately, most applications always open every file they access with read-write access even if they never plan on writing to it. If we copied all files opened with write access, we'd basically just end up with an unmodified version of every single file in the overwrite directory.

[whraven]

Thank you for the insight. It tells me where to dig further and to start looking at alternatives.

[Al12rs]

Another big thing to note is that Loot or xEdit could simply edit all the plugins filetimes (to enforce loadorder for older games) and this could cause a theoretical copy on write version of usvfs to copy all the active plugins to overwrite just to edit their timestamp. That is not something desirable

[AnyOldName3]

@Al12rs That depends on how it's done. If you want really serious COW that completely protects everything, it will protect filetimes. If you want a less serious version of COW, you could let filetimes and similar metadata get edited. If you wanted something in between, then it's possible to have a database in the overwrite directory of edits to file metadata without copying the whole file.

@whraven If you want to make any kind of progress with this, it's going to be a lot of work, and it's very unlikely that it'll be feasible with something based on USVFS. You probably want to look at https://github.com/KerberX/KxVirtualFileSystem (which Kortex uses) and https://github.com/AnyOldName3/LVFS/blob/master/LayeredDirectoryMirror/OneWay/SimpleOneWayContentMirror.cs (which I wrote as part of my degree, and does actually have copy-on-write features, but is designed to be flexible rather than fast, so even if you turn off COW features, it's far too slow to use as anything other than a proof-of-concept).

[Al12rs]

@whraven GitHub is not too good for an efficient conversation, if you want to discuss this with us directly its better to do it on Discord in the development channels. @erasmux is one of the devs that dived the most into usvfs and he had already made some experiments with changing usvfs to protect mods. He has been busy lately but if you want to know more about usvfs and the mod protection he told me you should PM him directly on Discord. His discord name is @erasmux#3933