"make llvm-build" fails to launch compiler on windows

[dankegel]

On either win10 or win11, with a fresh checkout of dev, following the instructions on https://tinygo.org/docs/guides/build/, there were three problems.

0) confusion about shell Fix: specify that all steps after choco are done in Git for Windows shell The target audience probably already guesses this, but it can't hurt to be clear.

1) cmake not found Workaround: change instructions from

choco install --confirm git golang mingw make cmake ninja python

to

choco install --confirm git golang mingw make ninja python
choco install --confirm cmake --installargs 'ADD_CMAKE_TO_PATH=System'

Again, target audience will know to put CMake on PATH, but automating it can't hurt.

2) ninja passes unix-like environment variable settings on commandline, Now, this one is more of a headscratcher, but patching llvm-source/llvm/CMakeLists.txt like so gets past the problem:

-      set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE ${CCACHE_PROGRAM})
+      set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE "env ${CCACHE_PROGRAM}")

[dankegel]

Unfortunately, there is a fourth problem:

[9/1335] Building CXX object tools/llvm-config/CMakeFiles/llvm-config.dir/llvm-config.cpp.obj
FAILED: tools/llvm-config/CMakeFiles/llvm-config.dir/llvm-config.cpp.obj
env CCACHE_CPP2=yes CCACHE_HASHDIR=yes C:/ProgramData/chocolatey/bin/ccache.exe C:\ProgramData\chocolatey\bin\c++.exe -DCMAKE_CFG_INTDIR=\".\" -DGTEST_HAS_RTTI=0 -D_FILE_OFFSET_BITS=64 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -IC:/Users/dank/src/tinygo/llvm-build/tools/llvm-config -IC:/Users/dank/src/tinygo/llvm-project/llvm/tools/llvm-config -IC:/Users/dank/src/tinygo/llvm-build/include -IC:/Users/dank/src/tinygo/llvm-project/llvm/include -Wa,-mbig-obj -Werror=date-time -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-class-memaccess -Wno-redundant-move -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wno-comment  -O2 -DNDEBUG  -fno-exceptions -fno-rtti -std=c++14 -MD -MT tools/llvm-config/CMakeFiles/llvm-config.dir/llvm-config.cpp.obj -MF tools\llvm-config\CMakeFiles\llvm-config.dir\llvm-config.cpp.obj.d -o tools/llvm-config/CMakeFiles/llvm-config.dir/llvm-config.cpp.obj -c C:/Users/dank/src/tinygo/llvm-project/llvm/tools/llvm-config/llvm-config.cpp
c++.exe: fatal error: no input files
compilation terminated.

Evidently there is something special about how llvm-config is built. Oddly, running that command by hand succeeds, but it doesn't make ninja happy; rerunning ninja tries to rebuild that file and fails again.

You can reproduce the problem in isolation with

$ ninja tools/llvm-config/CMakeFiles/llvm-config.dir/llvm-config.cpp.obj

procmon reports the command being run is

env CCACHE_CPP2=yes CCACHE_HASHDIR=yes C:/ProgramData/chocolatey/bin/ccache.exe C:\ProgramData\chocolatey\bin\c++.exe -DCMAKE_CFG_INTDIR=\".\" -DGTEST_HAS_RTTI=0 -D_FILE_OFFSET_BITS=64 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -IC:/Users/dank/src/tinygo/llvm-build/tools/llvm-config -IC:/Users/dank/src/tinygo/llvm-project/llvm/tools/llvm-config -IC:/Users/dank/src/tinygo/llvm-build/include -IC:/Users/dank/src/tinygo/llvm-project/llvm/include -Wa,-mbig-obj -Werror=date-time -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-class-memaccess -Wno-redundant-move -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wno-comment  -O2 -DNDEBUG  -fno-exceptions -fno-rtti -std=c++14 -MD -MT tools/llvm-config/CMakeFiles/llvm-config.dir/llvm-config.cpp.obj -MF tools\llvm-config\CMakeFiles\llvm-config.dir\llvm-config.cpp.obj.d -o tools/llvm-config/CMakeFiles/llvm-config.dir/llvm-config.cpp.obj -c C:/Users/dank/src/tinygo/llvm-project/llvm/tools/llvm-config/llvm-config.cpp

but it also reports that env then runs

"C:\ProgramData\chocolatey\bin\c++.exe" "-DCMAKE_CFG_INTDIR=\.\" -DGTEST_HAS_RTTI=0 -D_FILE_OFFSET_BITS=64 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -IC:/Users/dank/src/tinygo/llvm-build/tools/llvm-config -IC:/Users/dank/src/tinygo/llvm-project/llvm/tools/llvm-config -IC:/Users/dank/src/tinygo/llvm-build/include -IC:/Users/dank/src/tinygo/llvm-project/llvm/include -Wa,-mbig-obj -Werror=date-time -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-class-memaccess -Wno-redundant-move -Wno-noexcept-type -Wdelete-non-virtual-dtor -Wno-comment  -O2 -DNDEBUG  -fno-exceptions -fno-rtti -std=c++14 -MD -MT tools/llvm-config/CMakeFiles/llvm-config.dir/llvm-config.cpp.obj -MF tools\llvm-config\CMakeFiles\llvm-config.dir\llvm-config.cpp.obj.d -o tools/llvm-config/CMakeFiles/llvm-config.dir/llvm-config.cpp.obj -c C:/Users/dank/src/tinygo/llvm-project/llvm/tools/llvm-config/llvm-config.cpp"

Note the trailing doublequote! (Also note the moved quote around CMAKE_CFG_INTDIR.)

Whatever moved the quote around in CMAKE_CFG_INTDIR probably broke it, as they are there for the benefit of the c compiler, not the shell. The extra trailing doublequote, if real, might explain why c++ thought it had no arguments.

I suspect opaque mingw env commandline shenanigans...

[dankegel]

Commenting out CMAKE_CFG_INTDIR in llvm-config/CMakeLists.txt lets the build complete, though with a bad value for some variables in llvm-config, probably. (This could be made to work correctly by disabling ccache for just that file, maybe.)

A more correct workaround than any of the above is probably to configure llvm without ccache, and the simplest way to do that is to uninstall ccache. (Which may explain why this worked in CI.)

So, the build instructions on Windows might just want to have the following weasel words for now:

Because ccache and mingw and our branch of llvm do not quite get along together, rename all copies of ccache.exe on PATH to ccache-disabled.exe before building. (You can use "where ccache" in a git-for-windows window to find them.)

[aykevl]

I don't have any trouble building LLVM on Windows. I don't have ccache installed, so I guess I never encountered this particular bug.

I think these suggestions are a good idea. For ccache, I think it's possible to disable it on Windows. We detect it in the Makefile and can easily disable this detection on Windows. Doing things right by default is much better than explaining it in the docs. PRs welcome :)

[dankegel]

Hit this again on a fresh machine... both the choco install cmake problem and the strange ccache-causes-wrong-c++-compiler problem.

Ick ick ick. If I run into it on a third machine, I will try to remember to fix the ccache thing. In the meantime I should add a note to the doc...

[dankegel]

i7-11700 took only 23 minutes to build llvm 14. Not shabby. It took longer to find this bug report again :-)