search

openzfsonwindows / openzfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
471 stars 17 forks source link

Attempting to build using CMake 3.24.2 and Clang 15 yields unexpected results #139

Open kwvg opened 2 years ago

kwvg commented 2 years ago

System information

PS C:\OpenZFS\openzfs> cmd /c ver
Microsoft Windows [Version 10.0.19043.1889]

PS C:\OpenZFS\openzfs> clang -v
clang version 15.0.0
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: C:\Program Files\LLVM\bin

PS C:\OpenZFS\openzfs> cmake --version
cmake version 3.24.2

Environment variables

Reveal * `%PATH%` * `C:\Program Files\PowerShell\7` * `C:\Windows\system32` * `C:\Windows` * `C:\Windows\System32\Wbem` * `C:\Windows\System32\WindowsPowerShell\v1.0` * `C:\Windows\System32\OpenSSH` * `C:\Program Files\CMake\bin` * `C:\Program Files\Git\cmd` * `C:\Program Files\LLVM\bin` * `C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.16.27023\bin\HostX64\x64` * `C:\Program Files (x86)\Windows Kits\10\App Certification Kit` * `C:\Program Files (x86)\Windows Kits\10\bin\10.0.19041.0\x64` * `C:\Users\Computer\AppData\Local\Microsoft\WindowsApps` * `%INCLUDE%` * `C:\Program Files\LLVM\include` * `C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\cppwinrt` * `C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\km` * `C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\shared` * `C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\ucrt` * `C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\um` * `C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\winrt` * `C:\OpenZFS\openssl\include` * `C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.16.27023\include` * `C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.16.27023\atlmfc\include` * `%LIB%` * `C:\Program Files\LLVM\lib` * `C:\Program Files (x86)\Windows Kits\10\Lib\10.0.19041.0\km\x64` * `C:\Program Files (x86)\Windows Kits\10\Lib\10.0.19041.0\um\x64` * `C:\Program Files (x86)\Windows Kits\10\Lib\10.0.19041.0\ucrt\x64` * `C:\OpenZFS\openssl\lib` * `C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.16.27023\lib\x64` * `%WindowsSdkDir%` * `C:\Program Files (x86)\Windows Kits\10` * `%UseEnv%` * `true` * `%TargetPlatformVersion%` * `10.0.19041.0`

Describe the problem you're observing

Attempting to build openzfs (branch windows at https://github.com/openzfsonwindows/openzfs/commit/27e584dab7bf4d098dfa91f32ac83009632c5740) results in oddball behavior. Build instructions were based on windows-build-test.yml and attempts to reference ZFSin's README.md were made to no avail. The system was a vanilla installation of Windows 10 Pro (21H1, build 19043.1889) with any changes after being made specifically to help openzfs compile.

Describe how to reproduce the problems

Based on the guidance provided by windows-build-test.yml, andrewc12/openssl was cloned at C:\OpenZFS\openssl and OpenZFS was cloned at C:\OpenZFS\openzfs.

First build attempt #### Command ```powershell PS C:\OpenZFS\openzfs> cmake -B ".\out\build\x64-Debug" . ` -DOPENSSL_ROOT_DIR="C:\OpenZFS\openssl" ` -DCRYPTO_STATIC_TEST="C:\OpenZFS\openssl\lib\VC\static\libcrypto64MTd.lib" ` -DLIB_EAY_DEBUG="C:\OpenZFS\openssl\lib\VC\static\libcrypto64MTd.lib" ` -DLIB_EAY_RELEASE="C:\OpenZFS\openssl\lib\VC\static\libcrypto64MT.lib" ` -DOPENSSL_INCLUDE_DIR="C:\OpenZFS\openssl\include" ` -DSSL_EAY_DEBUG="C:\OpenZFS\openssl\lib\VC\static\libssl64MTd.lib" ` -DSSL_EAY_RELEASE="C:\OpenZFS\openssl\lib\VC\static\libssl64MT.lib" cmake --build ".\out\build\x64-Debug" --verbose -j 8 ``` #### Build output ``` [...] CMake Error at C:/Program Files/CMake/share/cmake-3.24/Modules/CMakeTestCCompiler.cmake:69 (message): The C compiler "C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.16.27023/bin/HostX64/x64/cl.exe" is not able to compile a simple test program. It fails with the following output: Change Dir: C:/OpenZFS/openzfs/out/build/x64-Debug/CMakeFiles/CMakeTmp Run Build Command(s):C:/Program Files/Microsoft Visual Studio/2022/Community/MSBuild/Current/Bin/amd64/MSBuild.exe cmTC_d7677.vcxproj /p:Configuration=Debug /p:Platform=x64 /p:VisualStudioVersion=17.0 /v:m && MSBuild version 17.3.1+2badb37d1 for .NET Framework Microsoft (R) C/C++ Optimizing Compiler Version 19.16.27048 for x64 Copyright (C) Microsoft Corporation. All rights reserved. cl /c /Zi /W1 /WX- /diagnostics:column /Od /Ob0 /D _MBCS /D HAVE_STRLCPY /D HAVE_STRLCAT /D "CMAKE_INTDIR=\"Debug\"" /Gm- /RTC1 /MDd /GS /fp:precise /Zc:wchar_t /Zc:forScope /Zc:inline /Fo"cmTC_d7677.dir\Debug\\" /Fd"cmTC_d7677.dir\Debug\vc143.pdb" /external:W1 /Gd /TC /errorReport:queue -Wno-implicit-int -Wno-implicit-function-declaration -Wno-int-conversion -Wno-invalid-token-paste "C:\OpenZFS\openzfs\out\build\x64-Debug\CMakeFiles\CMakeTmp\testCCompiler.c" cl : command line error D8021: invalid numeric argument '/Wno-implicit-int' [C:\OpenZFS\openzfs\out\build\x64-Debug\CMakeFiles\CMakeTmp\cmTC_d7677.vcxproj] CMake will not be able to correctly generate this project. Call Stack (most recent call first): CMakeLists.txt:2 (project) [...] ```

This guided me to use clang (sourced from LLVM), alongside clang-cl which is provided by Visual Studio as the test program cannot be built due to cl's dislike for this particular style of arguments. So we move forward and add the necessary flags needed to prefer clang over msvc.

Second build attempt #### Command ```powershell cmake -B ".\out\build\x64-Debug" . ` -G "Visual Studio 17 2022" -T ClangCL -A x64 ` -DCMAKE_BUILD_TYPE=Debug ` -DCMAKE_C_COMPILER="C:/Program Files/LLVM/bin/clang.exe" ` -DCMAKE_CXX_COMPILER="C:/Program Files/LLVM/bin/clang++.exe" ` -DOPENSSL_ROOT_DIR="C:\OpenZFS\openssl" ` -DCRYPTO_STATIC_TEST="C:\OpenZFS\openssl\lib\VC\static\libcrypto64MTd.lib" ` -DLIB_EAY_DEBUG="C:\OpenZFS\openssl\lib\VC\static\libcrypto64MTd.lib" ` -DLIB_EAY_RELEASE="C:\OpenZFS\openssl\lib\VC\static\libcrypto64MT.lib" ` -DOPENSSL_INCLUDE_DIR="C:\OpenZFS\openssl\include" ` -DSSL_EAY_DEBUG="C:\OpenZFS\openssl\lib\VC\static\libssl64MTd.lib" ` -DSSL_EAY_RELEASE="C:\OpenZFS\openssl\lib\VC\static\libssl64MT.lib" cmake --build ".\out\build\x64-Debug" --verbose -j 8 ``` #### Build Output ``` [...] "C:\OpenZFS\openzfs\out\build\x64-Debug\ALL_BUILD.vcxproj" (default target) (1) -> "C:\OpenZFS\openzfs\out\build\x64-Debug\module\os\windows\OpenZFS.vcxproj" (default target) (3) -> "C:\OpenZFS\openzfs\out\build\x64-Debug\module\zfs\zfskern.vcxproj" (default target) (26) -> (ClCompile target) -> C:\OpenZFS\openzfs\module\zfs\arc.c(7248,3): error : pasting formed 'WPP_CALL_(', an invalid preprocessing token [-Winvalid-token-paste] [C:\OpenZFS\openzfs\out\build\x64-Debug\module\zfs\zfskern.vcxproj] C:\OpenZFS\openzfs\module\zfs\arc.c(7248,3): error : pasting formed ')7248', an invalid preprocessing token [-Winvalid-token-paste] [C:\OpenZFS\openzfs\out\build\x64-Debug\module\zfs\zfskern.vcxproj] C:\OpenZFS\openzfs\module\zfs\arc.c(7248,3): error : expected ';' after expression [C:\OpenZFS\openzfs\out\build\x64-Debug\module\zfs\zfskern.vcxproj] C:\OpenZFS\openzfs\module\zfs\arc.c(7248,10): error : called object type 'int' is not a function or function pointer [C:\OpenZFS\openzfs\out\build\x64-Debug\module\zfs\zfskern.vcxproj] [...] "C:\OpenZFS\openzfs\out\build\x64-Debug\ALL_BUILD.vcxproj" (default target) (1) -> "C:\OpenZFS\openzfs\out\build\x64-Debug\module\os\windows\OpenZFS.vcxproj" (default target) (3) -> "C:\OpenZFS\openzfs\out\build\x64-Debug\module\icp\icpkern.vcxproj" (default target) (4) -> "C:\OpenZFS\openzfs\out\build\x64-Debug\module\os\windows\spl\splkern.vcxproj" (default target) (5) -> (ClCompile target) -> C:\OpenZFS\openzfs\module\os\windows\spl\spl-kmem.c(959,27): error : incompatible integer to pointer conversion passing 'pc_t' (aka 'unsigned long long') to parameter of type 'const void *' [-Wint-conversion] [C:\OpenZFS\openzfs\out\build\x64-Debug\module\os\windows\spl\splkern.vcxproj] C:\OpenZFS\openzfs\module\os\windows\spl\spl-kmem.c(5234,2): error : call to undeclared function 'segkmem_abd_init'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration] [C:\OpenZFS\openzfs\out\build\x64-Debug\module\os\windows\spl\splkern.vcxproj] C:\OpenZFS\openzfs\module\os\windows\spl\spl-kmem.c(5292,2): error : call to undeclared function 'segkmem_abd_fini'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration] [C:\OpenZFS\openzfs\out\build\x64-Debug\module\os\windows\spl\splkern.vcxproj] "C:\OpenZFS\openzfs\out\build\x64-Debug\ALL_BUILD.vcxproj" (default target) (1) -> "C:\OpenZFS\openzfs\out\build\x64-Debug\cmd\os\windows\kstat\kstat.vcxproj" (default target) (14) -> "C:\OpenZFS\openzfs\out\build\x64-Debug\lib\os\windows\libkstat\libkstat.vcxproj" (default target) (15) -> "C:\OpenZFS\openzfs\out\build\x64-Debug\lib\libspl\libspl.vcxproj" (default target) (17) -> C:\OpenZFS\openzfs\lib\libspl\os\windows\posix.c(794,24): error : incompatible pointer to integer conversion passing 'HANDLE' (aka 'void *') to parameter of type 'int' [-Wint-conversion] [C:\OpenZFS\openzfs\out\build\x64-Debug\lib\libspl\libspl.vcxproj] "C:\OpenZFS\openzfs\out\build\x64-Debug\ALL_BUILD.vcxproj" (default target) (1) -> "C:\OpenZFS\openzfs\out\build\x64-Debug\lib\libtpool\libtpool.vcxproj" (default target) (25) -> C:\OpenZFS\openzfs\lib\libtpool\thread_pool.c(154,5): error : call to undeclared function 'clock_gettime'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration] [C:\OpenZFS\openzfs\out\build\x64-Debug\lib\libtpool\libtpool.vcxproj] 204 Warning(s) 5 Error(s) [...] ```

Ok, so we need the compiler to turn the other cheek on a few things, no worries, we can add -Wno-implicit-int -Wno-int-conversion -Wno-invalid-token-paste and move on but -Wno-implicit-function-declaration won't work as CMake is assuming we're using C99 but that's disallowed if we want implicit declarations so we need to downgrade the standard to one that does with -DCMAKE_C_STANDARD="90".

That takes care of a few other problems.

Third build attempt #### Command ```powershell cmake -B ".\out\build\x64-Debug" . ` -G "Visual Studio 17 2022" -T ClangCL -A x64 ` -DCMAKE_BUILD_TYPE=Debug ` -DCMAKE_C_STANDARD="90" ` -DCMAKE_C_FLAGS="-Wno-implicit-int -Wno-implicit-function-declaration -Wno-int-conversion -Wno-invalid-token-paste" ` -DCMAKE_CXX_FLAGS="-Wno-implicit-int -Wno-implicit-function-declaration -Wno-int-conversion -Wno-invalid-token-paste" ` -DCMAKE_C_COMPILER="C:/Program Files/LLVM/bin/clang.exe" ` -DCMAKE_CXX_COMPILER="C:/Program Files/LLVM/bin/clang++.exe" ` -DOPENSSL_ROOT_DIR="C:\OpenZFS\openssl" ` -DCRYPTO_STATIC_TEST="C:\OpenZFS\openssl\lib\VC\static\libcrypto64MTd.lib" ` -DLIB_EAY_DEBUG="C:\OpenZFS\openssl\lib\VC\static\libcrypto64MTd.lib" ` -DLIB_EAY_RELEASE="C:\OpenZFS\openssl\lib\VC\static\libcrypto64MT.lib" ` -DOPENSSL_INCLUDE_DIR="C:\OpenZFS\openssl\include" ` -DSSL_EAY_DEBUG="C:\OpenZFS\openssl\lib\VC\static\libssl64MTd.lib" ` -DSSL_EAY_RELEASE="C:\OpenZFS\openssl\lib\VC\static\libssl64MT.lib" cmake --build ".\out\build\x64-Debug" --verbose -j 8 ``` #### Build output ``` [...] "C:\OpenZFS\openzfs\out\build\x64-Debug\ALL_BUILD.vcxproj" (default target) (1) -> "C:\OpenZFS\openzfs\out\build\x64-Debug\cmd\zdb\zdb.vcxproj" (default target) (33) -> (Link target) -> lld-link : error : duplicate symbol: strlcpy [C:\OpenZFS\openzfs\out\build\x64-Debug\cmd\zdb\zdb.vcxproj] lld-link : error : duplicate symbol: strlcat [C:\OpenZFS\openzfs\out\build\x64-Debug\cmd\zdb\zdb.vcxproj] "C:\OpenZFS\openzfs\out\build\x64-Debug\ALL_BUILD.vcxproj" (default target) (1) -> "C:\OpenZFS\openzfs\out\build\x64-Debug\cmd\zfs\zfs.vcxproj" (default target) (34) -> lld-link : error : duplicate symbol: strlcpy [C:\OpenZFS\openzfs\out\build\x64-Debug\cmd\zfs\zfs.vcxproj] lld-link : error : duplicate symbol: strlcat [C:\OpenZFS\openzfs\out\build\x64-Debug\cmd\zfs\zfs.vcxproj] "C:\OpenZFS\openzfs\out\build\x64-Debug\ALL_BUILD.vcxproj" (default target) (1) -> "C:\OpenZFS\openzfs\out\build\x64-Debug\cmd\zstream\zstreamdump.vcxproj" (default target) (38) -> lld-link : error : duplicate symbol: strlcpy [C:\OpenZFS\openzfs\out\build\x64-Debug\cmd\zstream\zstreamdump.vcxproj] "C:\OpenZFS\openzfs\out\build\x64-Debug\ALL_BUILD.vcxproj" (default target) (1) -> "C:\OpenZFS\openzfs\out\build\x64-Debug\cmd\zpool\zpool.vcxproj" (default target) (36) -> lld-link : error : duplicate symbol: strlcpy [C:\OpenZFS\openzfs\out\build\x64-Debug\cmd\zpool\zpool.vcxproj] lld-link : error : duplicate symbol: strlcat [C:\OpenZFS\openzfs\out\build\x64-Debug\cmd\zpool\zpool.vcxproj [...] ```

Now this was a real pain to figure out. For the longest time I thought it was something to do with the standard library and a wild goose chase ensued where I had to learn about how you exclude standard libraries, the whole thing with msvcrt and libcmt, how to pass linker flags in CMake, how to exclude standard library paths from includes and it was a whole thing and I'd probably be stuck here if not for obmarg's comment which suggested I build with /verbose:lib.

This revealed that the problem had in fact nothing to do with the standard library and everything to do within the codebase. 

[...]
    38>lld-link : error : duplicate symbol: strlcpy [C:\OpenZFS\openzfs\out\build\x64-Debug\cmd\zstream\zstreamdump.vcxproj]
         >>> defined at C:\OpenZFS\openzfs\lib\libspl\os\windows\posix.c:716
         >>>            libspl.lib(posix.obj)
         >>> defined at libspl.lib(strlcpy.obj)
[...]

The long and short of it is that there are two conflicting definitions, one platform specific and one generic and to disable the generic, some definitions need to be present, namely HAVE_STRLCPY and HAVE_STRLCAT.

So, we do exactly that...

Fourth build attempt #### Commands ```powershell cmake -B ".\out\build\x64-Debug" . ` -G "Visual Studio 17 2022" -T ClangCL -A x64 ` -DCMAKE_BUILD_TYPE=Debug ` -DCMAKE_C_STANDARD="90" ` -DCMAKE_C_FLAGS="-Wno-implicit-int -Wno-implicit-function-declaration -Wno-int-conversion -Wno-invalid-token-paste -DHAVE_STRLCPY -DHAVE_STRLCAT" ` -DCMAKE_CXX_FLAGS="-Wno-implicit-int -Wno-implicit-function-declaration -Wno-int-conversion -Wno-invalid-token-paste -DHAVE_STRLCPY -DHAVE_STRLCAT" ` -DCMAKE_C_COMPILER="C:/Program Files/LLVM/bin/clang.exe" ` -DCMAKE_CXX_COMPILER="C:/Program Files/LLVM/bin/clang++.exe" ` -DOPENSSL_ROOT_DIR="C:\OpenZFS\openssl" ` -DCRYPTO_STATIC_TEST="C:\OpenZFS\openssl\lib\VC\static\libcrypto64MTd.lib" ` -DLIB_EAY_DEBUG="C:\OpenZFS\openssl\lib\VC\static\libcrypto64MTd.lib" ` -DLIB_EAY_RELEASE="C:\OpenZFS\openssl\lib\VC\static\libcrypto64MT.lib" ` -DOPENSSL_INCLUDE_DIR="C:\OpenZFS\openssl\include" ` -DSSL_EAY_DEBUG="C:\OpenZFS\openssl\lib\VC\static\libssl64MTd.lib" ` -DSSL_EAY_RELEASE="C:\OpenZFS\openssl\lib\VC\static\libssl64MT.lib" cmake --build ".\out\build\x64-Debug" --verbose -j 8 ``` #### Build output ``` [...] "C:\OpenZFS\openzfs\out\build\x64-Debug\ALL_BUILD.vcxproj" (default target) (1) -> "C:\OpenZFS\openzfs\out\build\x64-Debug\module\os\windows\OpenZFS.vcxproj" (default target) (3) -> "C:\OpenZFS\openzfs\out\build\x64-Debug\module\zfs\zfskern.vcxproj" (default target) (26) -> (ClCompile target) -> C:\OpenZFS\openzfs\module\zfs\arc.c(7268,3): error : expected ';' after expression [C:\OpenZFS\openzfs\out\build\x64-Debug\module\zfs\zfskern.vcxproj] C:\OpenZFS\openzfs\module\zfs\arc.c(7268,10): error : called object type 'int' is not a function or function pointer [C:\OpenZFS\openzfs\out\build\x64-Debug\module\zfs\zfske rn.vcxproj] [...] "C:\OpenZFS\openzfs\out\build\x64-Debug\ALL_BUILD.vcxproj" (default target) (1) -> "C:\OpenZFS\openzfs\out\build\x64-Debug\cmd\zdb\zdb.vcxproj" (default target) (32) -> EXEC : SignTool error : No certificates were found that met all the given criteria. [C:\OpenZFS\openzfs\out\build\x64-Debug\cmd\zdb\zdb.vcxproj] C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(159,5): error MSB3073: The command "setlocal [C:\OpenZFS\openzf s\out\build\x64-Debug\cmd\zdb\zdb.vcxproj] C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(159,5): error MSB3073: cd C:\OpenZFS\openzfs\out\build\x64-Debug\ cmd\zdb [C:\OpenZFS\openzfs\out\build\x64-Debug\cmd\zdb\zdb.vcxproj] C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(159,5): error MSB3073: if %errorlevel% neq 0 goto :cmEnd [C:\Open ZFS\openzfs\out\build\x64-Debug\cmd\zdb\zdb.vcxproj] C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(159,5): error MSB3073: C: [C:\OpenZFS\openzfs\out\build\x64-Debug \cmd\zdb\zdb.vcxproj] C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(159,5): error MSB3073: if %errorlevel% neq 0 goto :cmEnd [C:\Open ZFS\openzfs\out\build\x64-Debug\cmd\zdb\zdb.vcxproj] C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(159,5): error MSB3073: "C:\Program Files (x86)\Windows Kits\10\bi n\10.0.19041.0\x64\signtool.exe" sign /v /as /fd sha256 /td sha256 /sha1 e96bb80ace0b559239c89a425ba0b58d5590fdb3 /tr http://timestamp.digicert.com C:/OpenZFS/openzfs/out/buil d/x64-Debug/cmd/zdb/Debug/zdb.exe [C:\OpenZFS\openzfs\out\build\x64-Debug\cmd\zdb\zdb.vcxproj] C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(159,5): error MSB3073: if %errorlevel% neq 0 goto :cmEnd [C:\Open ZFS\openzfs\out\build\x64-Debug\cmd\zdb\zdb.vcxproj] C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(159,5): error MSB3073: :cmEnd [C:\OpenZFS\openzfs\out\build\x64-D ebug\cmd\zdb\zdb.vcxproj] C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(159,5): error MSB3073: endlocal & call :cmErrorLevel %errorlevel% & goto :cmDone [C:\OpenZFS\openzfs\out\build\x64-Debug\cmd\zdb\zdb.vcxproj] C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(159,5): error MSB3073: :cmErrorLevel [C:\OpenZFS\openzfs\out\buil d\x64-Debug\cmd\zdb\zdb.vcxproj] C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(159,5): error MSB3073: exit /b %1 [C:\OpenZFS\openzfs\out\build\x 64-Debug\cmd\zdb\zdb.vcxproj] C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(159,5): error MSB3073: :cmDone [C:\OpenZFS\openzfs\out\build\x64- Debug\cmd\zdb\zdb.vcxproj] C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(159,5): error MSB3073: if %errorlevel% neq 0 goto :VCEnd [C:\Open ZFS\openzfs\out\build\x64-Debug\cmd\zdb\zdb.vcxproj] C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(159,5): error MSB3073: :VCEnd" exited with code 1. [C:\OpenZFS\op enzfs\out\build\x64-Debug\cmd\zdb\zdb.vcxproj] [...] ```

Okay, not sure I can do much about that but at least we've done what we can. But during the course of this whole thing there were two other things observed

Include any warning/errors/backtraces from the system logs

See above

Additional notes and comments

I'm not sure what the solution is, it's quite possible that I could've avoided this whole thing if I just RTFM but the README takes me to the official OpenZFS documentation and I tried to compile this one Thursday evening with little to no prior experience with building, just in general, forget drivers, on Windows, I'm much too used to *nix-isms so there may be some mea culpa.

Though, I assumed CMake will behave in as platform-independent manner as possible and has the necessary logic to work around these issues and since some tinkering was needed to get things mostly somewhat working-ish(TM), some improvements to a few CMakeLists.txt could make this whole process dramatically easier.

andrewc12 commented 2 years ago

Hi, firstly welcome.

Secondly, omg don't use my terrible script as a guide 🤪 I haven't read all of your issue so alot of this might be incorrect advice. The actual instructions are at https://github.com/openzfsonwindows/openzfs/blob/windows/module/os/windows/README.md

But if you want to try to do it from the terminal you should run this image

It corresponds to this line https://github.com/openzfsonwindows/openzfs/blob/windows/.github/workflows/windows-build-test.yml#L20 which is very easy to miss

andrewc12 commented 2 years ago

Also I use a manually extracted copy of SSL, the actuall version you should install is from here https://slproweb.com/products/Win32OpenSSL.html IMG_20220916_062000~2.jpg

lundman commented 2 years ago

Hello, yes the readme inside module/os/windows is the one to use. The top level readme is Upstream, but we could have a modified version of it until we merge - or at least some hints.

For my work flow, I have Visual Studio installed, and using the Visual Studio Installer, tick on that I want "clang", "cmake" and "spectre x64 libraries". Open VS and then "open folder" which picks up that it is CMake and gets set-up, pick X64-debug (always goes to ARM for some reason). and Build.

My todo does have updating to VS2022, WDK, SDK and clang. So I should get to that.

andrewc12 commented 2 years ago

I've got some of those changes in a branch in my tree

https://github.com/andrewc12/openzfs/blob/andrew_devvm01/module/os/windows/README.md

kwvg commented 2 years ago

It corresponds to this line https://github.com/openzfsonwindows/openzfs/blob/windows/.github/workflows/windows-build-test.yml#L20 which is very easy to miss

I'm aware, I read through index.js to make heads and tails of it and gave up pretty quickly as it brought more questions than answers. Maybe this is just me being used to the *nix ways of doing things but I'm not really a fan of having to launch a script that non-transparently sets up an environment, just so I can build.

I'd much rather know what's going on and have it documented than have to rely on a script to even get into a command shell capable of building. The script should be a convenience option, not the only option.

That's why I painstakingly, installed and configured the tools and set up the environment variables for the local user myself, it was a pain but at least it forced me to know where everything is stored, how Microsoft structures their directories and what is exactly needed to make everything click (the issue I filed is a very condensed version of the highlights, there were actually ~20 build attempts).

Open VS and then "open folder" which picks up that it is CMake and gets set-up, pick X64-debug (always goes to ARM for some reason). and Build.

I'm running this within a VM that doesn't have hardware accelerated graphics so I'd really not want to use Visual Studio IDE at all, I was planning to get a first build working and then continuing my work over SSH and then modify the VM to be essentially headless. This might be a dumb since this is a platform-specific driver we're talking about but being untethered to an IDE is what I'm used to.

https://github.com/openzfsonwindows/openzfs/blob/windows/module/os/windows/README.md

I did read the setup guide and the first step of downloading a Microsoft-provided image is what made me stop, as much as I appreciate the convenience, I like to know what it's actually setting up so I can reproduce it on a clean copy so that I have the option not to rely on provided images (even if it's not practical to exercise said option).

That being said, I don't remotely know enough to document them myself and it would probably steal time that could be spent in development or fixing bugs for you to do it, but it is something that's basically on my wishlist and I hope it happens sometime!

andrewc12 commented 2 years ago

May I suggest forking the repo and modifying the yaml file.

You could make it dump the environment variables before and after using the msvc environment script Which might help you find things.

andrewc12 commented 2 years ago

here's some stuff i pulled out of mine im assuming the stuff in C:\Program Files (x86)\Windows Kits is the sdk/wdk

EXTERNAL_INCLUDE=C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\ATLMFC\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\VS\include;C:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt

INCLUDE=C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\ATLMFC\include;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\VS\include;C:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt;C:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt

LIB=C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\ATLMFC\lib\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\lib\x64;C:\Program Files (x86)\Windows Kits\10\lib\10.0.22621.0\ucrt\x64;C:\Program Files (x86)\Windows Kits\10\\lib\10.0.22621.0\\um\x64

LIBPATH=C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\ATLMFC\lib\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\lib\x64;C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.33.31629\lib\x86\store\references;C:\Program Files (x86)\Windows Kits\10\UnionMetadata\10.0.22621.0;C:\Program Files (x86)\Windows Kits\10\References\10.0.22621.0;C:\Windows\Microsoft.NET\Framework64\v4.0.30319
lundman commented 2 years ago

I'm running this within a VM that doesn't have hardware accelerated graphics so I'd really not want to use Visual Studio IDE at all, I was planning to get a first build working and then continuing my work over SSH and then modify the VM to be essentially headless. Of course, I picked CMake over pure vxproj as the Upstream developers could not be expected to load VS to add a source file to the project, that wouldn't fly. I do run VS in a VM as well, as it is very handy when it comes to the debugger.

In appveyor, I run:

cmake -G "Ninja" -B./out/build/x64-Debug  -DOPENSSL_ROOT_DIR=C:/OpenSSL-v30-Win64 -DCRYPTO_STATIC_TEST=C:/OpenSSL-v30-Win64/lib/VC/libcrypto64MTd.lib -DLIB_EAY_DEBUG=C:/OpenSSL-v30-Win64/lib/VC/libcrypto64MTd.lib -DLIB_EAY_RELEASE=C:/OpenSSL-v30-Win64/lib/VC/libcrypto64MT.lib -DOPENSSL_INCLUDE_DIR=C:/OpenSSL-v30-Win64/include -DSSL_EAY_DEBUG=C:/OpenSSL-v30-Win64/lib/VC/libssl64MTd.lib -DSSL_EAY_RELEASE=C:/OpenSSL-v30-Win64/lib/VC/libssl64MT.lib

cmake --build ./out/build/x64-Debug

Microsoft-provided image is what made me stop, as much as I appreciate the convenience

It is all about convenience. If you want to setup own Windows, that's great. The guide was written as a "quickest way" to get to compiling, so that people wouldn't be turned away by "first, install 2 fresh copies of windows". But the free VM is a pain, as I have to reinstall it every 6 months. (90 day free license, 5 re-issues, or something).

andrewc12 commented 2 years ago

In appveyor, I run:

cmake -G "Ninja" -B./out/build/x64-Debug  -DOPENSSL_ROOT_DIR=C:/OpenSSL-v30-Win64 -DCRYPTO_STATIC_TEST=C:/OpenSSL-v30-Win64/lib/VC/libcrypto64MTd.lib -DLIB_EAY_DEBUG=C:/OpenSSL-v30-Win64/lib/VC/libcrypto64MTd.lib -DLIB_EAY_RELEASE=C:/OpenSSL-v30-Win64/lib/VC/libcrypto64MT.lib -DOPENSSL_INCLUDE_DIR=C:/OpenSSL-v30-Win64/include -DSSL_EAY_DEBUG=C:/OpenSSL-v30-Win64/lib/VC/libssl64MTd.lib -DSSL_EAY_RELEASE=C:/OpenSSL-v30-Win64/lib/VC/libssl64MT.lib

cmake --build ./out/build/x64-Debug

Did you always do that or is that you using the uglyness I came up with?

lundman commented 2 years ago

That was from you!

andrewc12 commented 2 years ago

ughh here i am making this project worse 😛

kwvg commented 2 years ago

Is there any reason why the entire codebase can't be compiled with MSVC wholesale? If it's just the flags that need to be carried over, then I can look into inserting equivalent flags within the CMakeLists.

lundman commented 2 years ago

Basically, all the assembler files, which use CPP, can not be used in Windows. They would have to be converted to Windows assembler, including swapping left/right-ness. and without CPP you'd have to expand many of them.

MSVC does not do #incude_next which is used to map the illumos header files, into the local-platform header files.

Since the goal is to include Windows port into the OpenZFS repository, as opposed to always having a separate repository, it is by far better to compile with clang - very few #ifdef are needed. If clang eventually gets "/DRIVER" flag, we won't need MSVC at all.

And minor: Upstream needs to be able to easily add a new source file in PRs, so devstudio project files is out.

kwvg commented 2 years ago

So, just to recap, using Clang wholesale is favorable to maintain upstream compatibility as using MSVC would incur significant development overhead, so dropping MSVC is relatively more desirable. Would that be correct?

Any thoughts on forcing CMake to check for MSVC and [preferred compiler] and then having the subproject containing driver.c use MSVC by force? I remember at one point being able to force CMake to only compile using a particular compiler but it was recommended against as that defeats the point of CMake being platform-agnostic but I don't see an alternative.

lundman commented 2 years ago

ZFSin was 100% MSVC project, but many changes needed, and no (few?) assembler at all. Changing to clang was a step forward, so much of the ZFS sources "just compiles". But you can not produce a working driver with it. Which is why we are at a hybrid situation. clang will compile all of ZFS into .libs. Then MSVC will compile driver.c and link with libs to produce the driver. (The .SYS file only, all .EXE is clang only) There is also a vcproj inside contrib/windows/ that I use only for deployment with VC debugger, for easy debugging. This still needs the libs to be compiled with CMake, then you can compile driver.c and deploy. I suspect that VS project file could probably be made to compile cmake automatically, then driver, but I didn't to spent more time on that. Eventually, I hope clang can do all, including /driver binary. I will still use VS myself, both for the cmake, and for the deployment with debugger.