Open toothache opened 3 years ago
VS2019 (vc142) is assumed, but the generated nuget package still contains v141 keyword
Correct and by-design. It is for backward compatibility to allow seamless upgrade in downstream. We only maintain for one version, most likely the latest, due to resource constraints. Ideally there should be auto-generated meta nuget for v141->v142, and rename everything to 142. Would you mind helping adding that?
Lack scripts to install VS build tools
Because I don't know a good way to solve all below together:
Suggestions wanted.
There're several Win10 SDK versions installed, but it seems that only 10.0.16299 is initialized;
What do you mean by initialized? You can only build against one winsdk version at a time. I remember this was added due to some downstream requirement of OS, probably WinServer Core or WinServer DC version and CDPx. Maybe this can be updated to a later version, haven't had change to investigate.
Follow version mapping here in case you need: https://developer.microsoft.com/en-us/windows/downloads/sdk-archive/
It has dependency to have curl built and installed
It should be installed.
If not, try ./setup.ps1 curl
.
Do you need to export some cmake ENVs to let CMake find it?
Export from where and find it in where?
They have circular dependency, but freetype doesn't support build w/o harfbuzz;
Yes, that's why the default setup.ps1
workflow build it twice.
Clone with -j50 doesn't reliably clone the repo and all the submodules;
This depends on GitHub behavior in different space/time, and need to be carefully tuned. Less concurrency also means I cannot fully utilize the bandwidth. Many networks (both ISP and website) even relies on multi-threading heavily to get reasonable performance.
Linux side I implemented a back-off strategy as part of the mirror-aware submodule cloning. Similar things can be done for Windows, but I'm not sure if we should duplicate this complex module or figure out a why to get bash. https://github.com/xkszltl/Roaster/blob/52428ec1d6074e5b0a4a1ac5b3e39021d3006d5f/pkgs/utils/git/submodule.sh#L55-L64
Build variant is not configured to release
Yes, dual-build is well supported there and there's no many dependencies.
latest_ver is override to latest and cause build failure.
This has been fixed with /DNOMINMAX
: https://github.com/xkszltl/Roaster/commit/e49f2d7cea16ddaba85c770a001c4f7c74fb65c6
Latest release has broken linkage so I have to use master.
FYI I did a Windows build during weekend. All library built successfully (on my system) except pytorch: https://github.com/pytorch/pytorch/issues/48895
I can successfully build libraries except for pytorch and onnx.
Will try to setup another clean machine to validate.
FYI I did a Windows build during weekend. All library built successfully (on my system) except pytorch: pytorch/pytorch#48895
Lack scripts to install VS build tools
Because I don't know a good way to solve all below together:
- Install when user don't have one.
- Complete user installation if it is insufficient (missing components).
- Avoid multiple installations, even in the case where we installed first and user want to install a normal VS later.
Suggestions wanted.
vswhere
provides reliable way to detect multiple installations. We can provide a default script to install vs_buildtool
. The script can helpful in the following cases:
There're several Win10 SDK versions installed, but it seems that only 10.0.16299 is initialized;
What do you mean by initialized? You can only build against one winsdk version at a time. I remember this was added due to some downstream requirement of OS, probably WinServer Core or WinServer DC version and CDPx. Maybe this can be updated to a later version, haven't had change to investigate. Follow version mapping here in case you need: https://developer.microsoft.com/en-us/windows/downloads/sdk-archive/
You explicitly initialized here: https://github.com/xkszltl/Roaster/blob/2205965ead563e6acc658107ebb254cec498b271/win/pkgs/env/toolchain.ps1#L185
Per my understanding, the Win10 SDK version doesn't matter too much since you don't rely on any new Windows APIs. In such case, Windows SDK should have stable ABI interfaces between different versions.
BTW, another suggestion is using vs command line to install SDKs. It's incrementally installed and you don't need to keep the SDK links either.
It has dependency to have curl built and installed
It should be installed. If not, try
./setup.ps1 curl
.
It's my mistake. I didn't notice that you'll use Invoke-WebRequest
if curl is not available in cmake.ps1
.
Do you need to export some cmake ENVs to let CMake find it?
Export from where and find it in where?
In my initial trials, CMake complained that it couldn't locate zlib or openssl (I manually executed those scripts one by one). I had to manually set some envs {LIB}_ROOT
according to CMake documentation. But later, this issue was gone.
They have circular dependency, but freetype doesn't support build w/o harfbuzz;
Yes, that's why the default
setup.ps1
workflow build it twice.
Yes, it worked as you described.
Clone with -j50 doesn't reliably clone the repo and all the submodules;
This depends on GitHub behavior in different space/time, and need to be carefully tuned. Less concurrency also means I cannot fully utilize the bandwidth. Many networks (both ISP and website) even relies on multi-threading heavily to get reasonable performance.
Linux side I implemented a back-off strategy as part of the mirror-aware submodule cloning. Similar things can be done for Windows, but I'm not sure if we should duplicate this complex module or figure out a why to get bash. https://github.com/xkszltl/Roaster/blob/52428ec1d6074e5b0a4a1ac5b3e39021d3006d5f/pkgs/utils/git/submodule.sh#L55-L64
Build variant is not configured to release
Yes, dual-build is well supported there and there's no many dependencies.
But we don't enable dual build in windows build. And the default build variant is debug
for boost.
latest_ver is override to latest and cause build failure.
This has been fixed with
/DNOMINMAX
: e49f2d7 Latest release has broken linkage so I have to use master.
vswhere provides reliable way to detect multiple installations. ...
Ideally I'd like to install standard (maybe community since that's free?) version of VS, so that people can add things later. Build tool version seems very CI-specific.
Is that possible to install standard VS via command line?
You explicitly initialized here: ...
WINVER and all those WinSDK macros for minimum targeting platform are set to SDK version by default, and won't work on older OS. I'm not aware of a global way to set it, but also want to avoid setting that per-project if possible.
But we don't enable dual build in windows build. And the default build variant is debug for boost.
I double checked, it is dual-build by default. We got both mt and mt-gd dlls.
Ideally I'd like to install standard (maybe community since that's free?) version of VS, so that people can add things later. Build tool version seems very CI-specific.
Is that possible to install standard VS via command line?
Yes. The script would be very similar with the vs_buildtools.ps1
in #7 . You just need to change the download url and may update the workloads to install.
https://docs.microsoft.com/en-us/visualstudio/install/workload-and-component-ids?view=vs-2019
WINVER and all those WinSDK macros for minimum targeting platform are set to SDK version by default, and won't work on older OS. I'm not aware of a global way to set it, but also want to avoid setting that per-project if possible.
Per-project setting is also something i want to avoid. But my question is we seems to install extra WinSDKs than what we really need.
We actually have two kinds of cmake generators, Ninja and VS2019.
winsdk_version
specifies the version of Windows SDK to use. Thus, for our case, it's 10.0.16299.0
.Therefore, at most two Windows SDK version will be used across windows build, but in winsdk.ps1
, four Windows SDKs are installed. It seems not necessary.
But we don't enable dual build in windows build. And the default build variant is debug for boost.
I double checked, it is dual-build by default. We got both mt and mt-gd dlls.
You're correct, it's dual build. I probably just took a glance at the build log and then thought it only build for debug variant.
For VS build, we don't specifically target a WinSDK version.
We do. It's the same as ninja build. vs generator should take what's given by the env var. Let me know if it failed to do so.
but in winsdk.ps1, four Windows SDKs are installed
Those 4 are selected based on popularity of Win10/Win2019 versions.
Snappy build failed again. Snappy is adding third party submodules in their repo.
Fixed with --recursive
in https://github.com/xkszltl/Roaster/commit/c54898a54ad8235c745b4dfed8c80b53a4629023
Please have a try.
Not actually. I'll provide error logs later.
2020-12-17T06:22:01.3378721Z ..\third_party\googletest\googlemock\src\gmock_main.cc(63): error C2491: 'main': definition of dllimport function not allowed
2020-12-17T06:22:01.3388588Z [13/46] Linking CXX executable snappy_test_tool.exe
2020-12-17T06:22:01.3394114Z FAILED: snappy_test_tool.exe
2020-12-17T06:22:01.3398712Z cmd.exe /C "cd . && "C:\Program Files\CMake\bin\cmake.exe" -E vs_link_exe --intdir=CMakeFiles\snappy_test_tool.dir --rc="C:\PROGRA~2\Windows Kits\10\bin\10.0.16299.0\x64\rc.exe" --mt="C:\PROGRA~2\Windows Kits\10\bin\10.0.16299.0\x64\mt.exe" --manifests -- "C:\PROGRA~2\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\bin\Hostx64\x64\link.exe" /nologo CMakeFiles\snappy_test_tool.dir\snappy_test_tool.cc.obj /out:snappy_test_tool.exe /implib:snappy_test_tool.lib /pdb:pdb\snappy_test_tool.pdb /version:0.0 /DEBUG:FASTLINK /LTCG:incremental /INCREMENTAL:NO /subsystem:console snappy_test_support.lib snappy.lib "C:\Program Files\gflags\lib\gflags.lib" shlwapi.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib && cd ."
2020-12-17T06:22:01.3403393Z LINK: command "C:\PROGRA~2\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\bin\Hostx64\x64\link.exe /nologo CMakeFiles\snappy_test_tool.dir\snappy_test_tool.cc.obj /out:snappy_test_tool.exe /implib:snappy_test_tool.lib /pdb:pdb\snappy_test_tool.pdb /version:0.0 /DEBUG:FASTLINK /LTCG:incremental /INCREMENTAL:NO /subsystem:console snappy_test_support.lib snappy.lib C:\Program Files\gflags\lib\gflags.lib shlwapi.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIFESTFILE:snappy_test_tool.exe.manifest" failed (exit code 1120) with the following output:
2020-12-17T06:22:01.3405412Z Creating library snappy_test_tool.lib and object snappy_test_tool.exp
2020-12-17T06:22:01.3406820Z snappy_test_tool.cc.obj : error LNK2001: unresolved external symbol "class file::OptionsStub const & __cdecl file::Defaults(void)" (?Defaults@file@@YAAEBVOptionsStub@1@XZ)
2020-12-17T06:22:01.3410428Z snappy_test_tool.cc.obj : error LNK2001: unresolved external symbol "public: __cdecl file::StatusStub::~StatusStub(void)" (??1StatusStub@file@@QEAA@XZ)
2020-12-17T06:22:01.3411848Z snappy_test_tool.cc.obj : error LNK2001: unresolved external symbol "public: bool __cdecl file::StatusStub::ok(void)" (?ok@StatusStub@file@@QEAA_NXZ)
2020-12-17T06:22:01.3417757Z snappy_test_tool.cc.obj : error LNK2001: unresolved external symbol "class file::StatusStub __cdecl file::GetContents(class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const &,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > *,class file::OptionsStub const &)" (?GetContents@file@@YA?AVStatusStub@1@AEBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@PEAV34@AEBVOptionsStub@1@@Z)
2020-12-17T06:22:01.3421431Z snappy_test_tool.cc.obj : error LNK2001: unresolved external symbol "class file::StatusStub __cdecl file::SetContents(class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const &,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const &,class file::OptionsStub const &)" (?SetContents@file@@YA?AVStatusStub@1@AEBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@0AEBVOptionsStub@1@@Z)
2020-12-17T06:22:01.3424734Z snappy_test_tool.cc.obj : error LNK2001: unresolved external symbol "class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > __cdecl snappy::StrFormat(char const *,...)" (?StrFormat@snappy@@YA?AV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@PEBDZZ)
2020-12-17T06:22:01.3426301Z snappy_test_tool.cc.obj : error LNK2001: unresolved external symbol "public: __cdecl snappy::LogMessage::~LogMessage(void)" (??1LogMessage@snappy@@QEAA@XZ)
2020-12-17T06:22:01.3428086Z snappy_test_tool.cc.obj : error LNK2001: unresolved external symbol "public: class snappy::LogMessage & __cdecl snappy::LogMessage::operator<<(class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const &)" (??6LogMessage@snappy@@QEAAAEAV01@AEBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@Z)
2020-12-17T06:22:01.3429751Z snappy_test_tool.cc.obj : error LNK2001: unresolved external symbol "public: class snappy::LogMessage & __cdecl snappy::LogMessage::operator<<(int)" (??6LogMessage@snappy@@QEAAAEAV01@H@Z)
2020-12-17T06:22:01.3432074Z snappy_test_tool.cc.obj : error LNK2001: unresolved external symbol "public: __cdecl snappy::LogMessageCrash::~LogMessageCrash(void)" (??1LogMessageCrash@snappy@@QEAA@XZ)
2020-12-17T06:22:01.3434104Z snappy_test_tool.exe : fatal error LNK1120: 10 unresolved externals
Confirmed. Seems to be a bug. Static lib works but we need dynamic libs. May due to some missing dllexport.
Thanks, this should fix the issue.
BTW, why don't we use a stable release branch to build snappy?
BTW, why don't we use a stable release branch to build snappy?
Because unit test of that one doesn't work. gtest linkage issue.
And I can't find the issues tab in snappy repo... =。=
Same here. Probably they closed it. See it you can find their internal channel (or abuse PR....)
Here're some issues observed when building roaster packages in Windows:
toolchain.ps1
:win/pkgs/env
folder is ignored by mistake;winsdk
:10.0.16299
is initialized;cmake
:curl
built and installed;zlib
andopenssl
:freetype
andharfbuzz
:freetype
doesn't support build w/o harfbuzz;boost
:-j50
doesn't reliably clone the repo and all the submodules;release
;snappy
:latest_ver
is override tolatest
and cause build failure.