search

nodejs / build

Better build and test infra for Node.
502 stars 165 forks source link

Compilation broken on Windows in the CI #3739

Open StefanStojanovic opened 1 month ago

StefanStojanovic commented 1 month ago

node-compile-windows and node-compile-windows-debug have been constantly failing since the weekend began. I'm currently investigating it and the reason seems to be the latest update of VS 2022 to 17.10.0 (everything was fine with 17.9.7). While I'm working on it, those jobs/machines will be down for some time, so just letting everyone know in advance. I'll write update here once I have something.

targos commented 1 month ago

I tried to compile locally out of curiosity. This is what I get:

  maglev.cc
C:\Users\mzasso\git\node\deps\v8\src\maglev\maglev-ir.cc(3855): fatal  error C1001: Internal compiler error. [C:\Users\
mzasso\git\node\tools\v8_gypfiles\v8_base_without_compiler.vcxproj]
  (compiler file 'D:\a\_work\1\s\src\vctools\Compiler\Utc\src\p2\main.c', line 242)
   To work around this problem, try simplifying or changing the program near the locations listed above.
  If possible please provide a repro here: https://developercommunity.visualstudio.com
  Please choose the Technical Support command on the Visual C++
   Help menu, or open the Technical Support help file for more information
    CL!RaiseException()+0x54
    CL!DllGetObjHandler()+0x31a0c
    CL!DllGetObjHandler()+0x448
    CL!DllGetObjHandler()+0x5c2690
    CL!DllGetObjHandler()+0x6cf368
    CL!DllGetObjHandler()+0x7c7a74
    CL!DllGetObjHandler()+0x7bce74
    CL!DllGetObjHandler()+0x6cd960
    CL!DllGetObjHandler()+0x394fbc
    CL!DllGetObjHandler()+0x114b0
    CL!DllGetObjHandler()+0x85d0
    CL!DllGetObjHandler()+0x18c88
    CL!DllGetObjHandler()+0x3fb918
    CL!strtol()+0xac
    CL!BaseThreadInitThunk()+0x30
    CL!RtlUserThreadStart()+0x3c
StefanStojanovic commented 1 month ago

I've managed to fix the compilation jobs. The problem was as I suspected the update to VS 17.10.0. Luckily, the VS community edition that we use offers rollback, so I took it back to 17.9.7 and pinned that version. There is also another possibility: to install VS2019 build tools with the exact version and then use VS only for the editor and add workloads, components, etc. from the build tools installation.

vcbuild.bat and vswhere_usability_wrapper.cmd are written so they only accept the latest version of build tools, so if you have VS 17.10.0, Node will not let you compile with eg v17.9.7 compiler. Since ClangCL is coming (hopefully) soon, I do not think this is worth fixing to support various versions of MSVC, but I'll keep it on my radar just in case.

The new problem I see now is that node-test-binary-windows-native-suites are constantly failing since yesterday (they were not running 1-2 days before that because of the compilation issue), so I'll dig into that.

StefanStojanovic commented 1 month ago

The fix was applied yesterday and everything seems to be back to normal, Some PRs will need to be rebased to pass the CI, but other than that, everything else is working normally.

I will keep this issue open for a few more days, in case something goes wrong. Planning to close it on Friday.

targos commented 1 month ago

What's the long term solution ? Is Microsoft aware of the MSVC bug?

VXACDev commented 1 month ago

Broken all over the place... The only "solution" with the current MSVC release is to #pragma optimize off/on around the problem function. REALLY not good-- this is impact MANY projects (including Unreal Engine)

swaroop-sridhar commented 1 month ago

Hi @targos @StefanStojanovic: I work for the MSVC optimizer team, and we want to look into the above failure and fix it.

We'd really appreciate if you can provide a repro of the failure (ex: a pre-processed Cpp file and cl command line used) so that we can investigate the compiler error. Please file an issue with the repro at: https://developercommunity.visualstudio.com/cpp/report. Thanks. This will help to resolve the issue and unblock the VS-toolset update.

Thanks.

VXACDev commented 1 month ago

I pasted how to repro.. it’s a public repository “Vahalla Game” on but.. I just got an msvc update about 2 hrs ago, and it’s working properly again, so looks like someone fixed it… but double check me on that. The issue was in “RenderSky.cpp” in the draw() function. Mind you this isn’t my code, I use it for a demo/unit test for compatibility. Happy to provide more info if you need it.

Sent from Proton Mail for iOS

On Wed, May 29, 2024 at 4:16 PM, Swaroop Sridhar @.***(mailto:On Wed, May 29, 2024 at 4:16 PM, Swaroop Sridhar < wrote:

Hi @.(https://github.com/targos) @.(https://github.com/StefanStojanovic): I work for the MSVC optimizer team, and we want to look into the above failure and fix it.

We'd really appreciate if you can provide a repro of the failure (ex: a pre-processed Cpp file and cl command line used) so that we can investigate the compiler error. Please file an issue with the repro at: https://developercommunity.visualstudio.com/cpp/report. Thanks. This will help to resolve the issue and unblock the VS-toolset update.

Thanks.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

StefanStojanovic commented 1 month ago

What's the long term solution ? Is Microsoft aware of the MSVC bug?

I've brought this up in the conversation with my contact at Microsoft. He told me, he'll look into it and get back to me. I've also noticed there is a new version of VS v17.10.1 (still not in Chocolatey, but will be there in a few days probably). I'll try it locally to see if it compiles correctly.

swaroop-sridhar commented 1 month ago

Hi @StefanStojanovic it is still useful if you can file a developer community ticket with the repro at: https://developercommunity.visualstudio.com/cpp/report
This way, the issue will be officially tracked and resolved.

Thanks.

targos commented 1 month ago

I tried the new version and it fails as well. I'll need some time to find a way to generate a preprocessed file to submit to the community forums

targos commented 1 month ago

I don't really know what to do. As soon as I add /P to the MSVC arguments, compilation fails saying that it cannot find v8_pch.h.

VXACDev commented 1 month ago

There's a 2nd problem with these fixes [Compiler Version Mismatch::

There's a side-effect to the last "Fix" that sort-of fixed the optimization having an internal error-- Now, libraries built won't link on older (as in 1 week older) VS 2022, meaning you will need to deploy LIBs (if you deploy libs to customer) for 2022 prior to Mid-May 2022, AND Libs AFTER Mid-May 2022.

The unit-test is quite easy.. just build any project with MSVC Prior to the last 2 updates, and create and link to a LIB made in the latest MSVC, and you'll get an error saying the LIB and OBJs have a compiler-version mismatch.

Forcing customers to "upgrade" their existing compiler is not acceptable either (production machines simply won't do that on a whim).

Microsoft Visual Studio Professional 2022 Version 17.10.1 VisualStudio.17.Release/17.10.1+34928.147 Microsoft .NET Framework Version 4.8.09032

Installed Version: Professional

Visual C++ 2022 00483-10050-07952-AA543 Microsoft Visual C++ 2022

ASP.NET and Web Tools 17.10.338.1105 ASP.NET and Web Tools

Azure App Service Tools v3.0.0 17.10.338.1105 Azure App Service Tools v3.0.0

C# Tools 4.10.0-3.24270.2+e8f775c1d8a73dee7ad02408712d714251e708ea C# components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Extensibility Message Bus 1.4.39 @.***) Provides common messaging-based MEF services for loosely coupled Visual Studio extension components communication and integration.

GitHub Copilot 0.2.889.30432 GitHub Copilot is an AI pair programmer that helps you write code faster and with less work.

Linux Core Dump Debugging 1.0.9.34902 Enables debugging of Linux core dumps.

Microsoft JVM Debugger 1.0 Provides support for connecting the Visual Studio debugger to JDWP compatible Java Virtual Machines

NuGet Package Manager 6.10.0 NuGet Package Manager in Visual Studio. For more information about NuGet, visit https://docs.nuget.org/

Razor (ASP.NET Core) 17.10.3.2427201+4f57d1de251e654812adde201c0265a8ca7ca31d Provides languages services for ASP.NET Core Razor.

Test Adapter for Boost.Test 1.0 Enables Visual Studio's testing tools with unit tests written for Boost.Test. The use terms and Third Party Notices are available in the extension installation directory.

Test Adapter for Google Test 1.0 Enables Visual Studio's testing tools with unit tests written for Google Test. The use terms and Third Party Notices are available in the extension installation directory.

TypeScript Tools 17.0.30327.2001 TypeScript Tools for Microsoft Visual Studio

Visual Basic Tools 4.10.0-3.24270.2+e8f775c1d8a73dee7ad02408712d714251e708ea Visual Basic components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used.

Visual C++ for Linux Development 1.0.9.34902 Visual C++ for Linux Development

Visual F# Tools 17.10.0-beta.24228.1+dd749058c91585e9b5dae62b0f8df892429ee28f Microsoft Visual F# Tools

Visual Studio IntelliCode 2.2 AI-assisted development for Visual Studio.

VisualStudio.DeviceLog 1.0 Information about my package

VisualStudio.Mac 1.0 Mac Extension for Visual Studio

VSPackage Extension 1.0 VSPackage Visual Studio Extension Detailed Info

Xamarin 17.10.0.110 @.***) Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android.

Xamarin Designer 17.10.3.10 @.***) Visual Studio extension to enable Xamarin Designer tools in Visual Studio.

Xamarin.Android SDK 13.2.2.0 (d17-5/45b0e14) Xamarin.Android Reference Assemblies and MSBuild support. Mono: d9a6e87 Java.Interop: @. SQLite: @. Xamarin.Android Tools: @.***

Sent with Proton Mail secure email.

On Friday, May 31st, 2024 at 3:57 AM, Michaël Zasso @.***> wrote:

I don't really know what to do. As soon as I add /P to the MSVC arguments, compilation fails saying that it cannot find v8_pch.h.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

swaroop-sridhar commented 1 month ago

Hi @targos, the failure about a missing header usually indicates a missing include path in the compilation step. Do you get the same error with /E build? If you cannot generate the pre-compiled source, can you please share the instructions to build from the repro? (ex: After cloning nodejs repo, the build steps enough to follow, and the cl invocation that produces the compiler error)? I'll take a look using the same steps. Thanks.

targos commented 1 month ago

From the main branch of the nodejs/node repo:

.\vcbuild.bat debug

After some time, it errors.

Then:

cd tools\v8_gypfiles
"C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\bin\HostX64\x64\CL.exe" /c /I..\..\deps\v8 /I..\..\deps\v8\include /I"..\..\out\Debug\obj\global_intermediate\generate-bytecode-output-root" /I..\..\out\Debug\obj\global_intermediate /I"..\..\deps\icu-small\source\i18n" /I"..\..\deps\icu-small\source\common" /I"..\..\deps\v8\third_party\abseil-cpp" /Z7 /nologo /W3 /WX- /diagnostics:column /MP /O2 /Ob2 /Oi /Oy- /D _GLIBCXX_USE_CXX11_ABI=1 /D NODE_OPENSSL_CONF_NAME=nodejs_conf /D NODE_OPENSSL_HAS_QUIC /D ICU_NO_USER_DATA_OVERRIDE /D V8_GYP_BUILD /D V8_TYPED_ARRAY_MAX_SIZE_IN_HEAP=64 /D V8_ENABLE_CHECKS /D WIN32 /D _CRT_SECURE_NO_DEPRECATE /D _CRT_NONSTDC_NO_DEPRECATE /D _HAS_EXCEPTIONS=0 /D BUILDING_V8_SHARED=1 /D BUILDING_UV_SHARED=1 /D NOMINMAX /D OPENSSL_NO_PINSHARED /D OPENSSL_THREADS /D V8_TARGET_ARCH_X64 /D _WIN32_WINNT=0x0602 /D _SILENCE_ALL_CXX20_DEPRECATION_WARNINGS /D V8_HAVE_TARGET_OS /D V8_TARGET_OS_WIN /D "V8_EMBEDDER_STRING=\"-node.13\"" /D ENABLE_DISASSEMBLER /D V8_PROMISE_INTERNAL_FIELD_COUNT=1 /D V8_SHORT_BUILTIN_CALLS /D OBJECT_PRINT /D V8_INTL_SUPPORT /D V8_ATOMIC_OBJECT_FIELD_WRITES /D V8_ENABLE_LAZY_SOURCE_POSITIONS /D V8_USE_SIPHASH /D V8_SHARED_RO_HEAP /D NDEBUG /D V8_WIN64_UNWINDING_INFO /D V8_ENABLE_REGEXP_INTERPRETER_THREADED_DISPATCH /D V8_USE_ZLIB /D V8_ENABLE_SPARKPLUG /D V8_ENABLE_MAGLEV /D V8_ENABLE_TURBOFAN /D V8_ENABLE_SYSTEM_INSTRUMENTATION /D V8_ENABLE_ETW_STACK_WALKING /D V8_ENABLE_WEBASSEMBLY /D V8_ENABLE_JAVASCRIPT_PROMISE_HOOKS /D V8_ENABLE_CONTINUATION_PRESERVED_EMBEDDER_DATA /D V8_ALLOCATION_FOLDING /D V8_ALLOCATION_SITE_TRACKING /D V8_ADVANCED_BIGINT_ALGORITHMS /D UCONFIG_NO_SERVICE=1 /D U_ENABLE_DYLOAD=0 /D U_STATIC_IMPLEMENTATION=1 /D U_HAVE_STD_STRING=1 /D UCONFIG_NO_BREAK_ITERATION=0 /D DEBUG /D _DEBUG /D V8_TRACE_MAPS /D V8_ENABLE_ALLOCATION_TIMEOUT /D V8_ENABLE_FORCE_SLOW_PATH /D ENABLE_HANDLE_ZAPPING /D _UNICODE /D UNICODE /GF /Gm- /MTd /GS /fp:precise /Zc:wchar_t /Zc:forScope /Zc:inline /Yu"v8_pch.h" /Fp"..\..\out\Debug\obj\v8_compiler\v8_compiler.pch" /Fo"..\..\out\Debug\obj\v8_compiler\\deps\v8\src\compiler\backend\instruction-selector.obj" /Fd"..\..\out\Debug\v8_compiler.pdb" /external:W3 /Gd /TP /wd4351 /wd4355 /wd4800 /wd4251 /wd4275 /wd4244 /wd4267 /wd4129 /wd4245 /wd4324 /wd4506 /wd4661 /wd4701 /wd4702 /wd4703 /wd4709 /wd4714 /wd4715 /wd4718 /wd4723 /wd4724 /FIv8_pch.h /FC /errorReport:prompt /utf-8 /Zc:__cplusplus -std:c++20 /Zm2000 "..\..\deps\v8\src\compiler\backend\instruction-selector.cc"

Error:

instruction-selector.cc
D:\Git\nodejs\node\deps\v8\src\compiler\turboshaft\opmasks.h(150,75): error C2100: vous ne pouvez pas déréférencer un opérande de type 'v8::internal::compiler::turboshaft::WordRepresentation'
D:\Git\nodejs\node\deps\v8\src\compiler\turboshaft\opmasks.h(150,75): note: le contexte d’instanciation du modèle (le plus ancien) est
D:\Git\nodejs\node\deps\v8\src\compiler\turboshaft\opmasks.h(150,20): note: voir la référence à l'instanciation alias modèle 'v8::internal::compiler::turboshaft::Opmask::MaskBuilder<v8::internal::compiler::turboshaft::WordBinopOp,v8::internal::compiler::turboshaft::Opmask::OpMaskField<v8::internal::compiler::turboshaft::WordBinopOp::Kind,4>,v8::internal::compiler::turboshaft::Opmask::OpMaskField<v8::internal::compiler::turboshaft::Opmask::UnwrapRepresentation<v8::internal::compiler::turboshaft::WordRepresentation>::type,5>>::For<v8::internal::compiler::turboshaft::WordBinopOp::Kind::kAdd,v8::internal::compiler::turboshaft::WordRepresentation{v8::internal::compiler::turboshaft::RegisterRepresentation{v8::internal::compiler::turboshaft::MaybeRegisterRepresentation{v8::internal::compiler::turboshaft::MaybeRegisterRepresentation::Enum:v8::internal::compiler::turboshaft::MaybeRegisterRepresentation::Enum::kWord32}}}>' en cours de compilation
D:\Git\nodejs\node\deps\v8\src\compiler\turboshaft\opmasks.h(136,15): fatal error C1907: impossible de récupérer à partir des erreurs précédentes ; arrêt de la compilation
D:\Git\nodejs\node\deps\v8\src\compiler\turboshaft\representations.h(457,49): note: durant l'évaluation de la fonction constexpr 'v8::internal::compiler::turboshaft::WordRepresentation::value'
D:\Git\nodejs\node\deps\v8\src\compiler\turboshaft\opmasks.h(150,75): note: durant l'évaluation de la fonction constexpr 'v8::internal::compiler::turboshaft::WordRepresentation::operator v8::internal::compiler::turboshaft::WordRepresentation::Enum'
ERREUR INTERNE DU COMPILATEUR dans 'C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.40.33807\bin\HostX64\x64\cl.exe'
    Choisissez la commande Support technique du menu ? (Aide) de Visual C++
  ou ouvrez le fichier d'aide du Support technique pour plus d'informations
targos commented 1 month ago

Ok, I was able to make a standalone repro using the output from /E: repro.zip

Opened https://developercommunity.visualstudio.com/t/Internal-compiler-error-with-MSVC-1440/10673166

targos commented 1 month ago

The report I made is now Fixed - Pending Release.

targos commented 1 month ago

VS 17.10.2 was released, but it doesn't include an updated MSVC.

liudonghua123 commented 3 weeks ago

I encountered the similar problems when build node v22.3.0 recently, see the detailed ci logs here.

And at the same time I build v20.15.0, it is ok, the ci environment is the same (windows-latest in github actions).

The github action workflow file is windows-latest node-build.yml.

targos commented 3 weeks ago

Still an issue with VS 17.10.3.

StefanStojanovic commented 3 weeks ago

I encountered the similar problems when build node v22.3.0 recently, see the detailed ci logs here.

And at the same time I build v20.15.0, it is ok, the ci environment is the same (windows-latest in github actions).

The github action workflow file is windows-latest node-build.yml.

This is most likely due to the V8 version we have in those 2 versions.

cristianadam commented 1 week ago

Qt WebEngine has a fix at https://codereview.qt-project.org/c/qt/qtwebengine-chromium/+/573472

Flarna commented 5 days ago

Still get INTERNAL COMPILER ERROR with 17.10.4

targos commented 5 days ago

😞 My report at https://developercommunity.visualstudio.com/t/Internal-compiler-error-with-MSVC-1440/10673166 is still marked "Pending Release"

liudonghua123 commented 5 days ago

Sorry for the disturbance in this issue, I didn't notice that such many notification appeared here.

I tried a lot of methods including install old version of vs 2022 build tools. But none of them works for me currently. I found it's difficult to specify which vs 2022 to use for building node explicitly.

Does anyone make it work on github action. The official windows build action has disabled manually.

I also find some docs about vs 2022 17.10 updates. See https://learn.microsoft.com/zh-cn/cpp/overview/cpp-conformance-improvements?view=msvc-170#improvements_1710, https://devblogs.microsoft.com/cppblog/whats-new-for-cpp-developers-in-visual-studio-2022-17-10/, https://devblogs.microsoft.com/visualstudio/visual-studio-2022-17-10-now-available/.

Maybe some modification need to make on the upstream v8 project to fix it.

StefanStojanovic commented 2 hours ago

Maybe some modification need to make on the upstream v8 project to fix it.

V8 is dropping MSVC soon and they'll most likely not accept any MSVC specific patch at this time.