qsniyg / ksp_stuff

GNU General Public License v3.0
9 stars 4 forks source link

[Performance] of using symlinks on a ntfs partition #8

Open jarrard opened 5 years ago

jarrard commented 5 years ago

I wanted to pose this question after using this script to mount skyrim-se mods on my windows NTFS NVMe drive which also has large_writes enabled etc to increase performance.

The issue I have seen is that skyrim-se takes up to 5 minutes to load up at the start, once its loaded it seems to run fine, BUT under Windows10 it takes less then 30 seconds.

DO YOU KNOW what could be going on here? is the linux symlink system slowing down something? can a alternative solution be tested for better ntfs performance? (if that's the issue, yet to test on ext4)

It's one of the main reasons I have decided to not use Linux for modded gaming atm (at least Bethesda type mod system).

Mod Organiser 2 for exmaple does some sort of trickery in the computers memory to map everything virtually so its able to process 200-1000 mods within less then a couple seconds whereas this script would take considerably longer as it has to go through each folder and file and create physical links,.

qsniyg commented 5 years ago

My primary guess is how Wine handles case insensitivity. I'm sure the Wine developers have their reasons, but if it fails to find the file, it checks every directory, without a cache. So let's say Skyrim tries to access this file:

C:\Skyrim\Data\Meshes\Actors\Character\CharacterAssets\FemaleBody.NIF

But the actual file is:

~/.wine/drive_c/skyrim/data/meshes/actors/character/characterassets/femalebody.nif

It will check through ~/.wine/drive_c/ for a directory that can be stricmp'd to Skyrim, and then check through ~/.wine/drive_c/skyrim for a directory that can be stricmp'd to Data, etc.

This happens every single time it tries to find a file.

This could be fixed with a directory cache (one is actually implemented in this script to make the VFS process go faster), but considering how long the Wine developers have spent trying to fix this issue, I'm sure there's a good reason they didn't use one (although an option would be nice).

I did try to make performance slightly faster by not renaming everything to be lowercase, which might improve performance in some unique cases where it's able to find the file directly, but it's not a perfect solution, especially when mod is looking for File.NIF, and another is looking for file.nif.

For now, I think the solution is to wait until Linux 5.2 comes with per-directory case-insensitivity for EXT4. Once that comes, in theory we should just be able to setattr +F ~/.wine/ and reap the performance benefits :)


Referring to the second point (how long it takes to build the physical VFS), I completely agree that's a pretty big pain point. I'm considering creating my own script that will hook into wineserver and replace paths dynamically (with a case-insensitivity cache of course). I think this would be the best of both worlds, until USVFS manages to work under wine again.

It wouldn't replace USVFS, but it would work like this one, where you manually specify the mods directory, and it would create the VFS from there. Another nice improvement this would bring would be that it would automatically write to the overwrite directory, so no need to copy files manually after using BodySlide (or constantly re-running FNIS).

jarrard commented 5 years ago

For now, I think the solution is to wait until Linux 5.2 comes with per-directory case-insensitivity for EXT4. Once that comes, in theory we should just be able to setattr +F ~/.wine/ and reap the performance benefits :)

This would be nice, will keep a eye out for this. May not help when using ntfs partitions?

I wish wine had a option for turning off this file path seeking method you mention, that would be nice. (or allow one to be cached)

I've had issues running FNIS via wine, it often spits out some error 5 and doesn't output anything, had to run it via a windows VM and that was quite a slow process compared to how fast it is under native windows (10seconds compared to 5minutes in the VM!!!). Dunno why I can't get it to perform right under wine, looks like it uses a background command app to process files and that may be failing.,

qsniyg commented 5 years ago

May not help when using ntfs partitions?

I think the idea is to implement it for EXT4 for now, and move it to other filesystems later. I read somewhere that it was implemented mostly in the core kernel, meaning it shouldn't be too much work to port it to another filesystem. But I'm not a kernel developer, so I could be completely wrong.

That being said, from a quick Google search, apparently you can enable case insensitivity through lowntfs-3g. I've never tried it though.

I wish wine had a option for turning off this file path seeking method you mention

Unfortunately that would mean the vast majority of files would be seen as missing, as the casing changes constantly between mods and the ES{M,P} entries. It's a complex issue :(

I've had issues running FNIS via wine

What I did is that I copied the contents of the FNIS mod to the Data directory directly, and disabled the mod in ModOrganizer, and now it works fine for me (although it has to be regenerated every single time you update the mod configuration, as it's not in the overwrite folder, as I mentioned above). I also installed some dotnet thing under winetricks (I forget which), but I don't think it was necessary (and it also unfortunately caused issues with other programs).

This issue may be fixed by #5 (meaning you can just add it as a normal mod under MO), but I've only tested it with BodySlide for FO4.

jarrard commented 5 years ago

Unfortunately that would mean the vast majority of files would be seen as missing, as the casing changes constantly between mods and the ES{M,P} entries. It's a complex issue :(

Maybe wine can implement a directory caching system to get around this, I dunno

What I did is that I copied the contents of the FNIS mod to the Data directory directly, and disabled the mod in ModOrganizer, and now it works fine for me

Yeah I have like a multitude of mods, and I'd rather not do that, defeats the whole purpose of modding in this fashion, to keep the core game directly clean and not confused with scattered files that need to be overwritten in a very specific way by other mods..

Ultimately I'd like to know exactly what is going wrong with FNIS, I believe dotnet472 or something is needed to actually launch the app, however sometimes my wine prefixes refuse to install dotnet, saying its already installed when its not.. shrug.

qsniyg commented 5 years ago

defeats the whole purpose of modding in this fashion

Actually I just checked, I only copied the contents of the "Tools" directory. Once I get Skyrim running again (I screwed it up somehow with my mod setup) I'll see if #5 fixes it, it probably should.

The issue is that wine can't handle running exe's from symlinks, so if they're copied (or hardlinked), it runs fine.

saying its already installed when its not.. shrug

Did you install it through winetricks, and if so, have you tried the --force flag?

ajventer commented 5 years ago

Sorry, on this one I have no ideas either way. My games live on an EXT4 SSD - I have no NTFS partitions to test or compare with.

jarrard commented 5 years ago

Did you install it through winetricks, and if so, have you tried the --force flag?

yeah tried that.. I dunno, I think it only happens when I try and install dotnet40 might be ok with 472