Linux-Wine USVFS compatibility thread

[jarrard]

OP Update: Current MO2 via WINE status;

(refer to end thread for current status)

At this point in time MO2 mostly works with wine5/staging5, people will experience odd problems where MO2 will fail to load the VFS+EXE (tools/game) randomly and need to restart MO2.

These issues are likely some odd permissions failure happening with WINE and could even be due to me using NTFS for these games/files.

I intend to upgrade to straight EXT4 partition in the future (waiting on new NVMe) and will be able to test under native conditions then.

NOTE: Some window managers will cause semi-messed up font (ie. XFCE)

The problem

At the moment Mod Organizer 2.1.1 standalone install appears to work find under Wine 3.8 Staging (and other versions) without any need for .net installation, however the usvfs component appears to not mount the files correctly. I can't quite determine whats going wrong but the console seems to suggest the usvfs is at least loading.

Note: V2.1.2 breaks Linux compatibility, as always is a wine bug but that doesn't mean changes made can't be done in a wine friendly manner and still do what they need to do.

Environment

• Mod Organizer Version : 2.1.1
• Desktop OS/version used to run Mod Organizer: Linux Antergos Wine 3.8++

Details

Link to Mod Organizer logs

USVFS

The log files appear to be all empty, I guess it never initialized so doesn't log things.

MO Interface

mo_interface.log

I will try and obtain those log files, no idea where they could have gone too unless its a specific option somewhere?

PS. Fallout4/SkyrimSE give 50-70fps at 4k and high detail under DXVK and wine, its 100% worth getting working under Linux now! (and script extenders work, with patch)

[qsniyg]

I believe NtQueryDirectoryFileEx is a red-herring, as the devs have pointed out.

Newer versions of USVFS expect CreateProcessInternalW, which Wine currently doesn't implement (older versions hook CreateProcessA/W instead, which Wine of course implements)

Oops, looks like Github's out-of-date indexing strikes again. It is properly implemented now.

There may be other differences as well, I'll update this comment if I find any more

[qsniyg]

Here's a patch I wrote: https://gist.github.com/qsniyg/ec21d68f9407991a1043a71fa6ffd4cf (reddit post with more information). With this USVFS works properly under wine.

[jarrard]

Nice, hopefully the remaining issues can be figured out over time. And perhaps sometime it will appear in Proton-GE build.

[qsniyg]

@jarrard The explorer++ issue will hopefully be figured out soon, but I don't think the cwd one (the one requiring you to make fake directories) will be really possible in any sane way, due to fundamental differences in the way windows and linux work. I tried to work around this when I wrote my linux version of USVFS, but I ended up with a number of issues. The only way I believe it could be fixed for linux is if USVFS did something on their end, and I'm not sure if that could cause other issues.

I have no problem submitting it there, but I'm guessing the only way to do it is through a pull request?

[jarrard]

Yeah not sure on how to add it to proton-GE but sure maybe pull request is worth a try.,

I don't understand why USVFS can't create directories, HOW does it work on NTFS partitions? I tend to run my stuff off NTFS since I dual boot back to windows now and again for many reasons.

[qsniyg]

@jarrard Sorry, I didn't explain very well.

The issue is that it tries to load the executable (such as FNIS) with the cwd (current working directory) to a directory that doesn't exist (Data\tools\GenerateFNIS_for_Users). Under windows, this succeeds for some reason, but linux doesn't allow it on a kernel level. For example, if you wanted to run ls with a specific cwd, you'd do:

$ cd directory
$ ls

More or less the same happens in C (fork; chdir; exec). So if the directory doesn't exist:

$ cd nonexistentdirectory
cd: no such file or directory: nonexistentdirectory

It'll fail. Windows on the other hand, doesn't need chdir, the working directory is set as an argument to CreateProcess, and it doesn't seem to check for its existence before running the program.

[jarrard]

What happens if you just EXPLORE (MO2 function) the data directory and run FNIS from there manually? Or doesn't that explorer mode work either?

Maybe another patch in wine can detect this behaviour and insert the requirement for linux kernel to do this? I really am talking out of a hole in my head at this stage tho :)

[qsniyg]

@jarrard Actually explore (explorer++) doesn't work for an unrelated reason, but I would also expect it not to work.

insert the requirement for linux kernel to do this

This is the problem though, I don't know what the requirement could be. There are technically ways to go around this, but they would likely cause a number of other issues that may or may not be able to be solved, and would require restructuring a lot of the code. However, fixing this in USVFS itself may be possible.

[jarrard]

So to get this working well I need to basically manually create all the directory structure in my game DATA folder before running MO2 USVFS? So that is the major requirement atm correct? except for specific programs like FNIS needing a Virtual windows box to do its thing. (fnis is SLOW in virtualbox btw).

It should be noted that 32bit MO1 works without these issues btw, shrug.

[qsniyg]

@jarrard No, only for executables. Like FNIS and BodySlide (i.e. create empty folders named tools/GenerateFNIS_for_Users and tools/BodySlide in Data). Other than that, you don't need to touch the game folder :)

Also, FNIS should work if you install dotnet40 through winetricks.

It should be noted that 32bit MO1 works without these issues btw, shrug.

Does the data folder have folders for FNIS/Bodyslide? If not, how are they set up?

[jarrard]

Yes they do, but I don't use MO1, but I know it works from peoples reports. Possibly worth testing out with old skyrim or fallout3 to see its behaviour.

[qsniyg]

@jarrard I believe the reason why MO1 works is because it doesn't redirect directories. It actually suffers from both the same problems that the new version suffers from as well, but due to it not redirecting directories, it manages to avoid the issues surfacing. So the fact that the old version worked is actually an accident haha.

[Metalhead33]

Any progress? I am tempted to reinstall SSE and try out MO2 with the most recent Wine, if no one has tried.

[qsniyg]

@Metalhead33 Yes, the latest wine will (mostly) work, one of the two patches required for it to work has been merged into the latest wine (4.18). Here's a thread explaining it in more detail if you're interested: https://www.reddit.com/r/linux_gaming/comments/d3493y/i_got_mod_organizer_2s_usvfs_running_under_wine/

[jarrard]

Yeah seems we are about 90% of the way there or more. Just a couple small things to note as mentioned in that reddit post.

• Also you will want to read some stuff on WineHQ on howto get nexus mods website links opening directly into your MO2 install.
• I set MO2 up in Steams Proton system (use glorious-eggroll proton 4.19+) which means it generates a game ID for it, with protontricks you can then installed dotnet472 and vcrun2017 as per required.
• You may also need to install python 2.7 or something in the wine container (proton in my case) but I'm not 100% sure....

[cyro666]

I have used the patches to build 4.18 and a 4.20 version of wine. Simple mods now work, so uvfs seems to work with Skyrim. But the problem is other programs, like patchers, batch scripts, etc.

I can't get the Requiem mod working, for example. You were supposed to run something called reqtificator, which is essentially a skyproc patcher, if I understand correctly. But the patcher just doesn't see the files it needs, despite that fact that uvfs works with Skyrim.

It's the same problem as with explorer++, at least it seems to me like this.

Unless I'm missing something?

[jarrard]

You need to manually create folders in the Data directory for any programs you run. (explore the application and its install data, make all the folders manually)

Also Wine 4.19 does not have patch 2, so it does not work with USVFS atm. Should be up-streamed sometime, so this means we must manually compile our own wine or proton installs.

[qsniyg]

@cyro666 Strange... I've never used reqtificator, so I don't know what the issue is, sorry.

With explorer++, the issue is that wine's shell32 implementation uses linux native APIs, instead of invoking the windows APIs, which means that USVFS will can't affect it in any way. It's ~2000 lines of code, and I'm not really proficient enough with the windows API to know how to rewrite it (as well not having enough time to really do so). If reqtificator uses the shell32 APIs, that may be why the issue exists.

That being said, as @jarrard noted, you do need to create the folders in Skyrim's real Data folder. Looking at a tutorial for reqtificator, you'll probably need to create SkyProc Patchers\Requiem under Skyrim's Data folder.

[mysticfall]

It could be I'm saying something totatlly irrelevalent, but have you tried running Reqtificator using WINE?

Currently, I'm playing Skyrim with Requiem on Linux and I had trouble running Skyproc patchers using native Java. However, when I downloaded Windows version of JRE and run it via WINE, it could successfully run those patchers.

It works with symlinks created by movfs4l.py scripts, so I suppose it may have a chance to work with UVFS, if it has been made to work with the base game.

[cyro666]

@mysticfall Yes, the latest versions of Requiem include a Java executable anyways.

@qsniyg I tried re-installing the wine prefixes and now I can't get any mod to be recognized at all. Do you maybe know if the uvfs patches in tk-glitch wine builds work? Or do I need to apply them differently/myself?

[qsniyg]

@cyro666 Sorry, what do you mean by reinstalling the wine prefixes? Did you use a new wine version?

Unfortunately the TK-Glitch ones don't include the second USVFS patch anymore. You'll need to compile it from scratch or binary patch the compiled kernel32 DLLs.

This being said, I've sent in the second patch directly to wine, so hopefully it will get upstreamed in the next release.

[cyro666]

@cyro666 Yes, I tried to re-install the whole wine prefix just in case I setup anything wrong. I think I might have used a build from Tk-Glitch that still had the patch in the past, but now it doesn't.

In any case, I might try applying the patches myself then, or just wait and we'll see.

[cyro666]

Tried applying the patch myself (or rather, just editing the dlls/kernel32/file.c by myself) and it worked. The patcher still didn't see the necessary files though. I created the folder structure it needed by myself in the Data directory. I tried only the root of Data first, but it seems it also needed the Requiem folder in SkyProc Patchers. After this, the patcher at least seems to have worked. Will test with some gameplay.

Would be nice if a Proton build had this, could run all of this with a click directly from a Linux version of Steam.

[qsniyg]

The patch just got in git master for wine! https://source.winehq.org/git/wine.git/commit/1658d5a20de4b12df539f3e5ecc11b81984684a4

Unless it gets removed for some reason, it'll be in the next release.

[qsniyg]

It should work properly in wine 4.21 :)

[MagicRB]

It does work, but there is a small issue. It seems to me like no symlinks or anything are actually created and the USVFS files somehow appear only to the game (in my case Fallout 4). Issue is that F4SE fails to load the dlls, when they don't really exist. If I manually copy them, it works. I know that this has nothing to do with you guys (you could add a workaround but it's not your issue). I just thought it would be to inform anyone whose googling and finds this issue.

[Silarn]

I have noticed a problem with script extender DLLs loading through USVFS in wine, but symlinks are fundamentally not how MO2 works. The mod directories are virtualized and USVFS redirects finding and opening files to the appropriate mod directories for the requested paths.

I'm pretty sure most of the current issues are due to wine redirecting ~ntdll~ shell32 requests to native linux functions - which bypasses USVFS since it's only hooking Windows APIs.

[qsniyg]

@Silarn

I'm pretty sure most of the current issues are due to wine redirecting ntdll requests to native linux functions

That's the issue for the shell32 DLLs, not the ntdll ones. Unless something recently changed in wine, the ntdll/kernel32 ones still get hooked properly.

I haven't tried F4SE under a recent version of wine yet, it used to work properly with a (then-)USVFS-patched wine. If it doesn't now, it's likely due to a regression. I'll try to look into this.

[Silarn]

Right. Well, I recently set up a Linux instance and was unable to load SKSE64 plugins that were not in the Data directory due to it being unable to find a dependency (I'm assuming this was the SKSE DLL in the game directory, though I didn't find out for sure). Requiring the directories to exist for tools is also not normal behavior, so there's obviously a difference there as well.

[qsniyg]

@Silarn

Requiring the directories to exist for tools is also not normal behavior

As I wrote in the reddit post, it's almost impossible to solve this reasonably under wine, due to differences with how the linux and windows kernels work. This is why I asked on Discord if it would be possible to start the executable in a real directory, and then set the working directory after USVFS has been injected. I believe it wouldn't change the behavior at all, but it would allow it to work under wine.

[jarrard]

Wine 4.21 has some fixes for this USVFS system and allows it to mostly work.

I'll have to test it to see if DLL plugins are loaded.

Seems to work on some things? Can't verify if SKSE is loading dlls correctly.

> skse64 plugin preloader - d3dx9_42
> exe path: D:\SteamLibrary\steamapps\common\Skyrim Special Edition\SkyrimSE.exe
> loaded into SkyrimSE.exe or SkyrimVR.exe, proxying SkyrimSE d3dx9_42 funcs and registering preload hook
> success
> hook triggered, loading dll plugins (meh's loader)
> failed to search dll plugin directory
> DLLPlugin loader finished
> hook triggered, loading skse plugins
> found plugin "PrivateProfileRedirector" for preloading
> attempting to load found plugins
> attempting to load "D:\SteamLibrary\steamapps\common\Skyrim Special Edition\Data\SKSE\Plugins\PrivateProfileRedirector.dll"
> loaded successfully
> loader finished

On further inspection it does appear SKSE64 dlls may not be loading correctly because I installed the achievement unlocker plugin and the game still warns me at the menu that starting a new game with mods will disable achievements. Pretty sure its not meant to say that with that mod installed!

[qsniyg]

@jarrard I don't have SSE to test unfortunately. Does F4SE work properly for you?

[jarrard]

Ok more testing done and Iv'e found the issue. SKSE64 will load correct if executed directly within proton/steam but if you run it via MO2 it will fail to load in. I tested by manually copying the DLL into the SKSE/Plugins folder and tried both methods, MO2 SKSE failed while Steam SKSE works.

Not sure whats going on here but it seems SKSE64 is failing when launched via MO2. No idea how to fix this as of yet.

[Silarn]

It wont load your MO2 mods if you run it through not MO2? For me it would run SKSE just fine, but it would only load plugins in the Data directory, not virtualized. That being said I'm just using a plain old wine prefix, not proton through native Linux steam. I don't know enough to get it all to work through proton and native steam, so I went with just wine.

[jarrard]

Ok SKSE64 does appear to load because in console skse shows the correct version.

What we are seeing here is MO2 failing to link the dll files or something thus it doesn't load.

The INTERESTING thing that is happening is even if you place the dll files manually into the SKSE/Plugins folder and load via MO2, it STILL won't load the DLL's, even when SKSE has been confirmed to be loaded.

[Silarn]

I think more accurately I'd say whatever mechanism should let it correctly locate the path to the dll relative to the game directory isn't working. This is probably similar to the reason tools will not load on the virtualized directory without creating the directory inside data.

[Silarn]

It actually works just fine for me if they are in the 'real' data/skse/plugins. They need to not be inside an MO2 mod directory. Only inside Data.

[jarrard]

is that Skyrim Special Edition tho? because I'm testing SKSE64 which uses different wine patching to get it to work. Anyway it seems for me MO2 blocks all plugins loading even tho skse64 appears to inject fine. Wish there was a log file. (doesn't generate one)

[Silarn]

Yes, SSE. Remove them from any MO2 mod. It will load for me at that point. It will redirect to the mod DLL otherwise.

[jarrard]

I disabled all my SKSE64 mods then moved AchievementsModsEnablerLoader.dll into /Data/SKSE/Plugins/ and loaded SSE via MO2 and still SKSE doesn't inject that dll.

[Silarn]

Shouldn't need that old plugin with Engine Fixes, but that requires an additional loader installed in addition to SKSE64. All I can tell you is that I was able to load SKSE64 plugins from Data.

[jarrard]

How are you loading SSE? via proton? is MO2 linked inside proton? or are you doing all this external via wine?

[Silarn]

As I already said, I have everything set up in a wine prefix, including Steam. It's the lutris flavor of 4.21.

[jarrard]

ahh ok, that makes a big difference. Yeah sorry I'm using steam play system to do everything, probably why our experiences are wildly different.

[Silarn]

Not that different, same DLL issue if virtualized.

[jarrard]

nar its hugely different, steam has a truck tonne of libraries it pulls in and custom environmental settings, plus f-sync / e-sync which isn't always enabled via lutris runners.

Anyway from what I can tell proton/mo2 via proton steamplay steam directly is breaking skse64 somehow and preventing it from loading plugins. I'll mess around with disabling some stuff to see if I can get anything to happen.

Turns out logs are in this folder for steamplay. _drivec/users/steamuser/My Documents/My Games/Skyrim Special Edition/SKSE/

plugin D:\SteamLibrary\steamapps\common\Skyrim Special Edition\Data\SKSE\Plugins\AchievementsModsEnablerLoader.dll (00000001 AchievementsModsEnabler 00000001) disabled, fatal error occurred while loading plugin

Well that doesn't help. Maybe that plugin is borked, lol. Time to try another.

[qsniyg]

@jarrard Could you report back if another plugin gives the same error?

[Silarn]

@jarrard That particular plugin is outdated. In theory it's supposed to be version agnostic, but that doesn't always work with SKSE, so you'd probably need to use the custom loader to make it work.

Here's a handy spreadsheet with most of the SKSE plugins and their compatibility status: https://docs.google.com/spreadsheets/d/1SxFQ1qKjrlHjyFekZeZhaA8tY7hv_yvkdcLxCeynffA/edit#gid=0

[jarrard]

Loading Fuz Ro D-oh via MO2 produces this error: So I dunno, does this mean it actually linked correctly? will try without MO2 now.

couldn't load plugin D:\SteamLibrary\steamapps\common\Skyrim Special Edition\Data\SKSE\Plugins\Fuz Ro D'oh.dll (Error 126)

It would seam to work directly via SKSE (from steamplay) but for some reason it hangs on registering plugin listener 1 of 2.... not sure whats going on there.

[qsniyg]

@jarrard Awesome, thanks for reporting! 126 = ERROR_MOD_NOT_FOUND, which means that LoadLibrary failed to find the module somehow. Now to go down the long path of trying to figure out why it's failing... LoadLibrary should call the patched LoadLibraryExW, so I don't quite understand why it's not working yet.