search

qsniyg / ksp_stuff

GNU General Public License v3.0
9 stars 4 forks source link

Exploring USVFS compatibility #28

Closed qsniyg closed 4 years ago

qsniyg commented 4 years ago

Patch finished! https://gist.github.com/qsniyg/ec21d68f9407991a1043a71fa6ffd4cf

While not strictly related to this project, I figured this would be the best place to start jotting this down.

Support for USVFS has improved in wine. Programs now launch properly (thanks to CreateProcessInternalW having been implemented in Wine 3.19), and accessing individual files (via cmd.exe) works without issue.

However, directory listings currently fail. Under windows XP, cmd.exe completely fails to do so (likely a fault lying with USVFS), but under Windows 10, both cmd.exe and explorer++ are able to list the directories, but only the unmodified ones.

Log:

10:37:09.271 [I] mod_organizer_instance_11 opened in process 93
10:37:09.286 [I] attached to mod_organizer_instance_11 with 186115 nodes, size 67108864
10:37:09.286 [I] inv_mod_organizer_instance_1 opened in process 93
10:37:09.286 [I] attached to inv_mod_organizer_instance_1 with 249 nodes, size 65536
10:37:09.286 [I] Process registered in shared process list : 93
10:37:09.286 [I] Windows version 10.0.17134 sp 0 platform 2 (BuildLabEx reg value not found?!)
10:37:09.287 [I] hooked GetFileAttributesExW (0x7f9d5be190ac) in C:\windows\system32\kernelbase.dll type 64-bit hot patch
10:37:09.287 [I] hooked GetFileAttributesW (0x7f9d5be190d4) in C:\windows\system32\kernelbase.dll type 64-bit hot patch
10:37:09.287 [I] hooked SetFileAttributesW (0x7f9d5be1c2d8) in C:\windows\system32\kernelbase.dll type 64-bit hot patch
10:37:09.287 [I] hooked CreateDirectoryW (0x7f9d5be18350) in C:\windows\system32\kernelbase.dll type 64-bit hot patch
10:37:09.287 [I] hooked RemoveDirectoryW (0x7b432168) in C:\windows\system32\kernelbase.dll type 64-bit hot patch
10:37:09.287 [I] hooked DeleteFileW (0x7f9d5be189dc) in C:\windows\system32\kernelbase.dll type 64-bit hot patch
10:37:09.287 [I] hooked GetCurrentDirectoryA (0x7b42e064) in C:\windows\system32\kernelbase.dll type 64-bit hot patch
10:37:09.287 [I] hooked GetCurrentDirectoryW (0x7b42e088) in C:\windows\system32\kernelbase.dll type 64-bit hot patch
10:37:09.287 [I] hooked SetCurrentDirectoryA (0x7b43284c) in C:\windows\system32\kernelbase.dll type 64-bit hot patch
10:37:09.287 [I] hooked SetCurrentDirectoryW (0x7b43286c) in C:\windows\system32\kernelbase.dll type 64-bit hot patch
10:37:09.287 [I] hooked ExitProcess (0x7bc4889c) in C:\windows\system32\kernelbase.dll type 64-bit hot patch
10:37:09.287 [I] hooked CreateProcessInternalW (0x7b42becc) in C:\windows\system32\kernelbase.dll type 64-bit hot patch
10:37:09.287 [I] hooked MoveFileA (0x7b4313a4) in C:\windows\system32\kernel32.dll type 64-bit hot patch
10:37:09.287 [I] hooked MoveFileW (0x7b431470) in C:\windows\system32\kernel32.dll type 64-bit hot patch
10:37:09.287 [I] hooked MoveFileExA (0x7b4313c8) in C:\windows\system32\kernel32.dll type 64-bit hot patch
10:37:09.287 [I] hooked MoveFileExW (0x7b4313f0) in C:\windows\system32\kernelbase.dll type 64-bit hot patch
10:37:09.287 [I] hooked MoveFileWithProgressA (0x7b431494) in C:\windows\system32\kernel32.dll type 64-bit hot patch
10:37:09.287 [I] hooked MoveFileWithProgressW (0x7b4314c0) in C:\windows\system32\kernelbase.dll type 64-bit hot patch
10:37:09.288 [I] hooked CopyFileExW (0x7b42b800) in C:\windows\system32\kernelbase.dll type 64-bit hot patch
10:37:09.288 [E] failed to hook CopyFile2: No Error
10:37:09.288 [I] hooked GetPrivateProfileStringA (0x7b42f12c) in C:\windows\system32\kernel32.dll type 64-bit hot patch
10:37:09.288 [I] hooked GetPrivateProfileStringW (0x7b42f158) in C:\windows\system32\kernel32.dll type 64-bit hot patch
10:37:09.288 [I] hooked GetPrivateProfileSectionA (0x7b42f084) in C:\windows\system32\kernel32.dll type 64-bit hot patch
10:37:09.288 [I] hooked GetPrivateProfileSectionW (0x7b42f100) in C:\windows\system32\kernel32.dll type 64-bit hot patch
10:37:09.288 [I] hooked WritePrivateProfileStringA (0x7b434090) in C:\windows\system32\kernel32.dll type 64-bit hot patch
10:37:09.288 [I] hooked WritePrivateProfileStringW (0x7b4340bc) in C:\windows\system32\kernel32.dll type 64-bit hot patch
10:37:09.288 [I] hooked GetFullPathNameA (0x7b42e7b8) in C:\windows\system32\kernelbase.dll type 64-bit hot patch
10:37:09.288 [I] hooked GetFullPathNameW (0x7b42e7e4) in C:\windows\system32\kernelbase.dll type 64-bit hot patch
10:37:09.288 [I] hooked FindFirstFileExW (0x7b42d1ec) in C:\windows\system32\kernelbase.dll type 64-bit hot patch
10:37:09.288 [I] hooked NtQueryFullAttributesFile (0x7bc466b0) in C:\windows\system32\ntdll.dll type 64-bit hot patch
10:37:09.288 [I] hooked NtQueryAttributesFile (0x7bc46598) in C:\windows\system32\ntdll.dll type 64-bit hot patch
10:37:09.289 [I] hooked NtQueryDirectoryFile (0x7bc46600) in C:\windows\system32\ntdll.dll type 64-bit hot patch
10:37:09.289 [E] failed to hook NtQueryDirectoryFileEx: No Error
10:37:09.289 [I] hooked NtOpenFile (0x7bc461d8) in C:\windows\system32\ntdll.dll type 64-bit hot patch
10:37:09.289 [I] hooked NtCreateFile (0x7bc458e4) in C:\windows\system32\ntdll.dll type 64-bit hot patch
10:37:09.289 [I] hooked NtClose (0x7bc45800) in C:\windows\system32\ntdll.dll type 64-bit hot patch
10:37:09.289 [I] hooked NtTerminateProcess (0x7bc473c8) in C:\windows\system32\ntdll.dll type 64-bit hot patch
10:37:09.289 [I] hooked LoadLibraryExW (0x7b430fd4) in C:\windows\system32\kernelbase.dll type 64-bit hot patch
10:37:09.289 [I] hooked GetModuleFileNameW (0x7f9d5be193d8) in C:\windows\system32\kernelbase.dll type 64-bit hot patch
10:37:09.289 [I] inithooks in process 93 successful

From WINEDEBUG=+relay, it doesn't appear NtQueryDirectoryFileEx or CopyFile2 is called, but rather just FindFirstFileW, so I don't think either of these are the issue.

One possibility is that NtQueryDirectoryFile is called directly, as opposed to remotely. ntdll.dll calls a function at some ordinal or something, and my guess is that usvfs is replacing that function. But some functions in ntdll call NtQueryDirectoryFile based off the address of that function inside the .so file, instead of calling the one from the dll.

This may be completely wrong, but this is my guess so far as to why it's not working. I'm not sure what the solution is. This doesn't explain why earlier versions of USVFS work however.

The hook seems to be correctly called, and detects a large (but correct-looking) number of virtual files for NtQueryDirectoryFile.

One possible explanation as to why NtQueryDirectoryFile doesn't work is because of https://github.com/wine-mirror/wine/blob/master/dlls/kernel32/file.c#L804:

if (!has_wildcard || info->data_len < info->data_size - max_entry_size)

This sets info->data_size = 0;, which prevents FindNextFile from re-querying NtQueryDirectoryFile

EDIT: That is exactly what the issue was! Removing the second check (info->data_len < info->data_size - max_entry_size) allows it to work properly. FO4 runs modded up to the main menu (haven't tried the game yet). However, for whatever reason, explorer++.exe still doesn't work, and FNV still crashes.

Still, progress!

TESV can't find Skyrim.esm

Likely the main reason why the new USVFS doesn't work is because of the FindFirstFileExW hook

Found the other issue I believe. Not sure how to fix it though:

USVFS's hooked FindFirstFileExW will first try to find a inputted file through the original path. If it doesn't work, then it'll try with the path hooked to the overwrite path instead. Looking at USVFS's log, this makes sense:

18:39:12.569 <90:91> [D] hook_FindFirstFileExW [lpFileName=DATA\Fallout - Textures.bsa] [originalPath.c_str()=...\DATA\Fallout - Textures.bsa] [res=FFFFFFFF] [callContext.lastError()=2]
18:39:12.569 <90:91> [D] hook_FindFirstFileExW [lpFileName=DATA\Fallout - Textures.bsa] [finalPath.c_str()=C:\users\username\Local Settings\Application Data\ModOrganizer\FalloutNV\overwrite\Fallout - Textures.bsa] [res=FFFFFFFF] [callContext.lastError()=2]

But Fallout - Textures.bsa exists in the game directory, why does the first one return -1 then? Because internally, wine calls NtOpenFile, which is also hooked by USVFS:

18:39:12.569 <90:91> [D] ntdll_mess_NtOpenFile [source=...\DATA\] [rerouted=\??\C:\users\username\Local Settings\Application Data\ModOrganizer\FalloutNV\overwrite\] [*FileHandle=000001C8] [OpenOptions=33] [res=ok]

We can see this in the wine log too:

005b:trace:ntdll:FILE_CreateFile handle=0x1a0ecc access=80100000 name=L"\\??\\C:\\users\\username\\Local Settings\\Application Data\\ModOrganizer\\FalloutNV\\overwrite\\" objattr=00000040 root=(nil) sec=(nil) io=0x32e554 alloc_size=(nil) attr=00000000 sharing=00000003 disp=1 options=00000021 ea=(nil).0x00000000
005b:trace:file:wine_nt_to_unix_file_name L"\\??\\C:\\users\\username\\Local Settings\\Application Data\\ModOrganizer\\FalloutNV\\overwrite\\" -> "/home/username/.wine-fnv64/dosdevices/c:/users/username/Local Settings/Application Data/ModOrganizer/FalloutNV/overwrite/"
005b:trace:file:NtQueryDirectoryFile (0x1c8 (nil) (nil) (nil) 0x32df10 0x1a0f0c 0x0000025e 0x00000003 0x00000000 L"Fallout - Textures.bsa" 0x00000000
005b:trace:file:append_entry long L"." short L"" mask L"Fallout - Textures.bsa"
005b:trace:file:append_entry long L".." short L"" mask L"Fallout - Textures.bsa"
005b:trace:file:init_cached_dir_data mask L"Fallout - Textures.bsa" found 0 files
005b:trace:file:NtQueryDirectoryFile => c000000f (0)
005b:trace:file:FindFirstFileExW L"C:\\users\\username\\Local Settings\\Application Data\\ModOrganizer\\FalloutNV\\overwrite\\Fallout - Textures.bsa" 0 0x32ebac 0 (nil) 0
005b:trace:file:RtlDosPathNameToNtPathName_U_WithStatus (L"C:\\users\\username\\Local Settings\\Application Data\\ModOrganizer\\FalloutNV\\overwrite\\Fallout - Textures.bsa",0x32e54c,0x32e548,(nil))
005b:trace:file:RtlGetFullPathName_U (L"C:\\users\\username\\Local Settings\\Application Data\\ModOrganizer\\FalloutNV\\overwrite\\Fallout - Textures.bsa" 520 0x32e178 0x32e548)
005b:trace:ntdll:NtQueryInformationFile ((nil),0x32df28,0x32df30,0x00000018,0x00000005)
005b:trace:ntdll:FILE_CreateFile handle=0x1a174c access=80100000 name=L"\\??\\C:\\users\\username\\Local Settings\\Application Data\\ModOrganizer\\FalloutNV\\overwrite\\" objattr=00000040 root=(nil) sec=(nil) io=0x32e554 alloc_size=(nil) attr=00000000 sharing=00000003 disp=1 options=00000021 ea=(nil).0x00000000
005b:trace:file:wine_nt_to_unix_file_name L"\\??\\C:\\users\\username\\Local Settings\\Application Data\\ModOrganizer\\FalloutNV\\overwrite\\" -> "/home/username/.wine-fnv64/dosdevices/c:/users/username/Local Settings/Application Data/ModOrganizer/FalloutNV/overwrite/"
005b:trace:file:NtQueryDirectoryFile (0x1c8 (nil) (nil) (nil) 0x32df10 0x1a178c 0x0000025e 0x00000003 0x00000000 L"Fallout - Textures.bsa" 0x00000000
005b:trace:file:append_entry long L"." short L"" mask L"Fallout - Textures.bsa"
005b:trace:file:append_entry long L".." short L"" mask L"Fallout - Textures.bsa"
005b:trace:file:init_cached_dir_data mask L"Fallout - Textures.bsa" found 0 files
005b:trace:file:NtQueryDirectoryFile => c000000f (0)

This likely also explains why the old USVFS worked, as the old one didn't reroute directories (https://github.com/TanninOne/usvfs/blob/master/usvfs/hooks/ntdll.cpp#L104 vs https://github.com/ModOrganizer2/usvfs/blob/Develop/src/usvfs_dll/hooks/ntdll.cpp#L179 )

According to the USVFS devs:

It doesn't make any sense to me, if findFirstFile is looking for a file why it it opening the data folder with ntOpenFile? The fact that ntOpenFile is rerouted to overwrite when looking ofr Data is correct. We need that to happen or other things break.

So FindFirstFileExW needs to somehow open a handle for NtQueryDirectoryFile without running NtOpenFile (https://github.com/wine-mirror/wine/blob/master/dlls/kernel32/file.c#L755)