search

mstorsjo / msvc-wine

Scripts for setting up and running MSVC in Wine on Linux
Other
683 stars 83 forks source link

RFC: Add basic support for automated WDK installation #111

Closed ravenexp closed 9 months ago

ravenexp commented 9 months ago

Add a new command line option '--with-wdk-installers' pointing to the directory where the MSI packages downloaded by wdksetup.exe are located.

Extract the WDK packages after unpacking the Windows SDK packages. They both install files in the same versioned directories.

Unpack the included VS extension files onto the MSVC tree.

Fix the WDK directory structure after extracting MSI packages with msiextract on Linux.

Resolves https://github.com/mstorsjo/msvc-wine/issues/28

ravenexp commented 9 months ago

Could we add some minimal testing of it in the CI setup? E.g. enable the --with-wdk-installers in one setup, and then e.g. try compiling and linking some minimal test driver? (Doesn't need to be a full test driver from somewhere, just something minimal with a single entry point function or similar, would be enough, as long as it testruns including a kernel mode header and linking a kernel mode library.)

I thought about doing this, but it will add ~700 MB of downloads to the CI job and ~ 1.5 GB to the install size. It also requires downloading wdksetup.exe from the Microsoft site and running it in automated mode in WINE. Initially I had all this implemented directly in vsdownload.py, but then decided against it and changed it to work with just the .msi packages that are somehow obtained. Perhaps I could add a small helper script that downloads wdksetup.exe, runs it with the right flags and downloads the WDK packages directly into the cache directory?

As for the test driver, I could take a MIT licensed example driver project from Microsoft GitHub, just as I did with MSBuild tests. They only provide .vcxproj build files, but I can add a Makefile or CMakeLists.txt to do the same thing with plain cl.exe and link.exe.

mstorsjo commented 9 months ago

I thought about doing this, but it will add ~700 MB of downloads to the CI job and ~ 1.5 GB to the install size.

Right, it probably would make the CI job a bit slower. The install size should probably be fine, as we don't distribute the installed directory anywhere anyway (and aren't allowed to anyway). OTOH if it needlessly hammers Microsoft servers, where they aren't prepared to handle very frequent extra downloads, it might not be so nice...

It also requires downloading wdksetup.exe from the Microsoft site and running it in automated mode in WINE. Initially I had all this implemented directly in vsdownload.py, but then decided against it and changed it to work with just the .msi packages that are somehow obtained.

Oh, I missed this detail.

Perhaps I could add a small helper script that downloads wdksetup.exe, runs it with the right flags and downloads the WDK packages directly into the cache directory?

That would certainly be appreciated indeed.

As for the test driver, I could take a MIT licensed example driver project from Microsoft GitHub, just as I did with MSBuild tests. They only provide .vcxproj build files, but I can add a Makefile or CMakeLists.txt to do the same thing with plain cl.exe and link.exe.

Sure, that would probably be good.

ravenexp commented 9 months ago

I getting strange mergeTrees() failures on macOS. I don't know where to start debugging this, since this command is run by the native Python, not in WINE. Is it OK to skip testing WDK install on this platform?

mstorsjo commented 9 months ago

I getting strange mergeTrees() failures on macOS. I don't know where to start debugging this, since this command is run by the native Python, not in WINE. Is it OK to skip testing WDK install on this platform?

That sounds odd.

Yes, it’s ok to not test it on all platforms, as long as it’s tested in at least one configuration in CI.

That said, if you want to debug it, I would add lots of debug printouts in vsdownload.py, to show what files it sees and how it progrfilesesses. But it’s a bit surprising to hear about such failures in unpacking(?) files. I wonder if it’s due to a case insensitive file system, or if it is due to a different version of Python.

ravenexp commented 9 months ago

But it’s a bit surprising to hear about such failures in unpacking(?) files. I wonder if it’s due to a case insensitive file system, or if it is due to a different version of Python.

Wait, does macOS have a case-insensitive file system? Isn't it a kind of Unix system since Mac OS X days? That would definitely explain it, because the failing code merges "Build" and "build" sub-directories created in the same directory.

mstorsjo commented 9 months ago

But it’s a bit surprising to hear about such failures in unpacking(?) files. I wonder if it’s due to a case insensitive file system, or if it is due to a different version of Python.

Wait, does macOS have a case-insensitive file system? Isn't it a kind of Unix system since Mac OS X days? That would definitely explain it, because the failing code merges "Build" and "build" sub-directories created in the same directory.

Yes, it is very much a Unix (even a certified one), but they have retained the case insensitiveness. It is possible to set the apple file systems to case sensitive mode too (although I think it might be kinda cumbersome as I’m not sure if it’s possible to change it other than when formatting a partition), but I’ve heard issues with e.g. some Adobe applications misbehaving on case sensitive file systems. For external volumes it’s certainly doable though.

ravenexp commented 9 months ago

This is very unfortunate, because my /build/ tree merging logic pretty much depends on being executed on a case-sensitive file system, since the extracted Build and build directories must be treated differently due to some bug in the WDK MSI installers.

ravenexp commented 9 months ago

I've rebased on master and fixed the Ubuntu CI job. macOS tests are not enabled because the WDK installation does not work on case-insensitive file systems with the current implementation. Is it OK to leave this as a Linux-only feature? I don't know if there's any demand for it, since developing Windows drivers on macOS is not something people usually do.

mstorsjo commented 9 months ago

I've rebased on master and fixed the Ubuntu CI job. macOS tests are not enabled because the WDK installation does not work on case-insensitive file systems with the current implementation. Is it OK to leave this as a Linux-only feature? I don't know if there's any demand for it, since developing Windows drivers on macOS is not something people usually do.

Yes, that's totally ok - if it's a less central feature, I'm quite happy with it being tested only in one configuration. As long as it is tested in at least one configuration, it won't be dead code that I don't know how to test.

I'll try to get to re-reviewing this PR within a few days, hopefully!