search

aws / aws-lc

AWS-LC is a general-purpose cryptographic library maintained by the AWS Cryptography team for AWS and their customers. It іs based on code from the Google BoringSSL project and the OpenSSL project.
Other
288 stars 108 forks source link

Windows build failed if the build path location is too long #1733

Closed stormshield-gt closed 2 weeks ago

stormshield-gt commented 1 month ago

Problem:

On every Windows target, the project doesn't compile if the path to the build directory is too long, because of the Windows maximum path length limitation. To reproduce you have to build inside several nested folders, so the path will be greater than 260 characters.

The current workaround is to set the build directory closer to the root of the file system, but this is not easily discoverable and reliable enough.

Relevant details

AWS-LC commit: (47333e1) (through aws-lc-sys 0.20.0)

System information:

Build log ```text -- Selecting Windows SDK version 10.0.22621.0 to target Windows 10.0.19045. -- The C compiler identification is MSVC 19.40.33813.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - failed -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe - broken -- Configuring incomplete, errors occurred! --- stderr CMake Deprecation Warning at CMakeLists.txt:4 (cmake_minimum_required): Compatibility with CMake < 3.5 will be removed from a future version of CMake. Update the VERSION argument value or use a ... suffix to tell CMake that the project does not need compatibility with older versions. CMake Error at C:/Program Files/CMake/share/cmake-3.30/Modules/CMakeTestCCompiler.cmake:67 (message): The C compiler "C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/VC/Tools/MSVC/14.40.33807/bin/Hostx64/x64/cl.exe" is not able to compile a simple test program. It fails with the following output: Change Dir: 'C:/Users/stormshield/Documents/Git/my-project/build/target/target-x86_64-pc-windows-msvc/x86_64-pc-windows-msvc/debug/build/aws-lc-sys-7068532a0b4ccf5c/out/build/CMakeFiles/C MakeScratch/TryCompile-an6xjf' Run Build Command(s): "C:/Program Files (x86)/Microsoft Visual Studio/2022/BuildTools/MSBuild/Current/Bin/amd64/MSBuild.exe" cmTC_8400d.vcxproj /p:Configuration=Debug /p:Platform=x64 / p:VisualStudioVersion=17.0 /v:n MSBuild version 17.10.4+10fbfbf2e for .NET Framework Build started 8/1/2024 10:06:36 AM. Project "C:\Users\stormshield\Documents\Git\my-project\build\target\target-x86_64-pc-windows-msvc\x86_64-pc-windows-msvc\debug\build\aws-lc-sys-7068532a0b4ccf5c\out\build\CMakeFiles\CMake Scratch\TryCompile-an6xjf\cmTC_8400d.vcxproj" on node 1 (default targets). PrepareForBuild: Creating directory "cmTC_8400d.dir\Debug\". Structured output is enabled. The formatting of compiler diagnostics will reflect the error hierarchy. See https://aka.ms/cpp/structured-output for more details. Creating directory "C:\Users\stormshield\Documents\Git\my-project\build\target\target-x86_64-pc-windows-msvc\x86_64-pc-windows-msvc\debug\build\aws-lc-sys-7068532a0b4ccf5c\out\build\CMa keFiles\CMakeScratch\TryCompile-an6xjf\Debug\". Creating directory "cmTC_8400d.dir\Debug\cmTC_8400d.tlog\". InitializeBuildStatus: Creating "cmTC_8400d.dir\Debug\cmTC_8400d.tlog\unsuccessfulbuild" because "AlwaysCreate" was specified. Touching "cmTC_8400d.dir\Debug\cmTC_8400d.tlog\unsuccessfulbuild". ClCompile: C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.40.33807\bin\HostX64\x64\CL.exe /c /Zi /nologo /W1 /WX- /diagnostics:column /Od /Ob0 /D _MBCS /D "CMAK E_INTDIR=\"Debug\"" /RTC1 /MDd /GS /fp:precise /Zc:wchar_t /Zc:forScope /Zc:inline /Fo"cmTC_8400d.dir\Debug\\" /Fd"cmTC_8400d.dir\Debug\vc143.pdb" /external:W1 /Gd /TC /errorReport:queue -B repro "C:\Users\stormshield\Documents\Git\my-project\build\target\target-x86_64-pc-windows-msvc\x86_64-pc-windows-msvc\debug\build\aws-lc-sys-7068532a0b4ccf5c\out\build\CMakeFiles\CMakeScratch\ TryCompile-an6xjf\testCCompiler.c" testCCompiler.c Link: C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.40.33807\bin\HostX64\x64\link.exe /ERRORREPORT:QUEUE /OUT:"C:\Users\stormshield\Documents\Git\my-project\ build\target\target-x86_64-pc-windows-msvc\x86_64-pc-windows-msvc\debug\build\aws-lc-sys-7068532a0b4ccf5c\out\build\CMakeFiles\CMakeScratch\TryCompile-an6xjf\Debug\cmTC_8400d.exe" /INCREMENT AL /ILK:"cmTC_8400d.dir\Debug\cmTC_8400d.ilk" /NOLOGO kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIFEST UAC:"level='asInvoker' uiAccess='false'" /manifest:embed /DEBUG /PDB:"C:/Users/stormshield/Documents/Git/my-project/build/target/target-x86_64-pc-windows-msvc/x86_64-pc-windows-msvc/debug/build /aws-lc-sys-7068532a0b4ccf5c/out/build/CMakeFiles/CMakeScratch/TryCompile-an6xjf/Debug/cmTC_8400d.pdb" /SUBSYSTEM:CONSOLE /TLBID:1 /DYNAMICBASE /NXCOMPAT /IMPLIB:"C:/Users/stormshield/Docume nts/Git/my-project/build/target/target-x86_64-pc-windows-msvc/x86_64-pc-windows-msvc/debug/build/aws-lc-sys-7068532a0b4ccf5c/out/build/CMakeFiles/CMakeScratch/TryCompile-an6xjf/Debug/cmTC_8400d .lib" /MACHINE:X64 /machine:x64 cmTC_8400d.dir\Debug\testCCompiler.obj FileTracker : error FTK1011: could not create the new file tracking log file: C:\Users\stormshield\Documents\Git\my-project\build\target\target-x86_64-pc-windows-msvc\x86_64-pc-windows-ms vc\debug\build\aws-lc-sys-7068532a0b4ccf5c\out\build\CMakeFiles\CMakeScratch\TryCompile-an6xjf\cmTC_8400d.dir\Debug\cmTC_8400d.tlog\link-cvtres.read.1.tlog. The system cannot find the path s pecified. [C:\Users\stormshield\Documents\Git\my-project\build\target\target-x86_64-pc-windows-msvc\x86_64-pc-windows-msvc\debug\build\aws-lc-sys-7068532a0b4ccf5c\out\build\CMakeFiles\CMakeScra tch\TryCompile-an6xjf\cmTC_8400d.vcxproj] FileTracker : error FTK1011: could not create the new file tracking log file: C:\Users\stormshield\Documents\Git\my-project\build\target\target-x86_64-pc-windows-msvc\x86_64-pc-windows-ms vc\debug\build\aws-lc-sys-7068532a0b4ccf5c\out\build\CMakeFiles\CMakeScratch\TryCompile-an6xjf\cmTC_8400d.dir\Debug\cmTC_8400d.tlog\link-cvtres.write.1.tlog. The system cannot find the path specified. [C:\Users\stormshield\Documents\Git\my-project\build\target\target-x86_64-pc-windows-msvc\x86_64-pc-windows-msvc\debug\build\aws-lc-sys-7068532a0b4ccf5c\out\build\CMakeFiles\CMakeScr atch\TryCompile-an6xjf\cmTC_8400d.vcxproj] Done Building Project "C:\Users\stormshield\Documents\Git\my-project\build\target\target-x86_64-pc-windows-msvc\x86_64-pc-windows-msvc\debug\build\aws-lc-sys-7068532a0b4ccf5c\out\build\CM akeFiles\CMakeScratch\TryCompile-an6xjf\cmTC_8400d.vcxproj" (default targets) -- FAILED. Build FAILED. "C:\Users\stormshield\Documents\Git\my-project\build\target\target-x86_64-pc-windows-msvc\x86_64-pc-windows-msvc\debug\build\aws-lc-sys-7068532a0b4ccf5c\out\build\CMakeFiles\CMakeScratch\ TryCompile-an6xjf\cmTC_8400d.vcxproj" (default target) (1) -> (Link target) -> FileTracker : error FTK1011: could not create the new file tracking log file: C:\Users\stormshield\Documents\Git\my-project\build\target\target-x86_64-pc-windows-msvc\x86_64-pc-windows- msvc\debug\build\aws-lc-sys-7068532a0b4ccf5c\out\build\CMakeFiles\CMakeScratch\TryCompile-an6xjf\cmTC_8400d.dir\Debug\cmTC_8400d.tlog\link-cvtres.read.1.tlog. The system cannot find the path specified. [C:\Users\stormshield\Documents\Git\my-project\build\target\target-x86_64-pc-windows-msvc\x86_64-pc-windows-msvc\debug\build\aws-lc-sys-7068532a0b4ccf5c\out\build\CMakeFiles\CMakeSc ratch\TryCompile-an6xjf\cmTC_8400d.vcxproj] FileTracker : error FTK1011: could not create the new file tracking log file: C:\Users\stormshield\Documents\Git\my-project\build\target\target-x86_64-pc-windows-msvc\x86_64-pc-windows- msvc\debug\build\aws-lc-sys-7068532a0b4ccf5c\out\build\CMakeFiles\CMakeScratch\TryCompile-an6xjf\cmTC_8400d.dir\Debug\cmTC_8400d.tlog\link-cvtres.write.1.tlog. The system cannot find the pat h specified. [C:\Users\stormshield\Documents\Git\my-project\build\target\target-x86_64-pc-windows-msvc\x86_64-pc-windows-msvc\debug\build\aws-lc-sys-7068532a0b4ccf5c\out\build\CMakeFiles\CMakeS cratch\TryCompile-an6xjf\cmTC_8400d.vcxproj] 0 Warning(s) 2 Error(s) Time Elapsed 00:00:00.79 CMake will not be able to correctly generate this project. Call Stack (most recent call first): CMakeLists.txt:7 (enable_language) ```

From my understanding, they are several solutions to this problem:

Please let me know what is your thought on this.

stormshield-gt commented 1 month ago

If I force the CMake Generator to be Ninja instead of CMake's Visual Studio generator, it compiles but I get a linker error

stormshield-gt commented 1 month ago

The linker error was due to the fact that the CRT was embedded both by my rust internal library and by aws-lc. So compiling with ninja works for me. Sorry for the noise. I guess there is still the problem of max path with the CMake's Visual Studio generator, but the docs suggest that it's not well tested so it's fine. Maybe this should be reported to aws-lc-rs that use this generator by default on windows to not force the install of ninja ?

andrewhop commented 1 month ago

Thanks for the detailed write up.

I guess there is still the problem of max path with the CMake's Visual Studio generator, but the docs suggest that it's not well tested so it's fine.

Do you have a link for this documentation?

Maybe this should be reported to aws-lc-rs that use this generator by default on windows to not force the install of ninja ?

We updated our documentation in https://github.com/aws/aws-lc-rs/pull/465 and the aws-lc-rs build only requires Ninja for the FIPS build.

justsmth commented 1 month ago

I did some googling on this. The "MSBuild" (i.e., CMake generator) part of the problem has been resolved, but other tooling around MSVC might still fail when path names exceed 260 chars. This link appears to be the relevant top-level tracking issue for Visual Studio.

stormshield-gt commented 4 weeks ago

Do you have a link for this documentation?

I was referring to this part of the documentation that mention that the CMake's Visual Studio generator is not tested regularly: 

On Windows, CMake's Visual Studio generator may also work, but it not tested regularly and requires recent versions of CMake for assembly support.

I did some googling on this. The "MSBuild" (i.e., CMake generator) part of the problem has https://github.com/dotnet/msbuild/issues/53#issuecomment-459062618, but other tooling around MSVC might still fail when path names exceed 260 chars. This link appears to be the relevant top-level tracking issue for Visual Studio.

Thanks a lot for digging this up! So it's an upstream problem that we cannot fix here. Note that might happen more frequently with aws-lc-rs, because how nested build artifacts are by default in the Rust target directory, creating long paths.

The workaround of using Ninja, by setting the environment variable $env:CMAKE_GENERATOR="Ninja" and make sure we have not embedded the CRT twice is fine for us. Feel free to close the issue. Thanks a lot for your help

Edit: For completion, overriding the CMake generator used by aws-lc-rs is possible because the cmake crate used in the build script support this natively. That's why it's possible to do so even if there is no logic for it inside the build script of the aws-lc-sys crate.