Closed ronawho closed 2 years ago
FYI @mppf and @daviditen
For the ordering, I just ran into this (on linux) and tracked it down to this loop where the break
only breaks out of the inner. I suspect that is the same bug you're seeing
The ordering issue is resolved with #19278.
The problem with not finding headers seems like it's happening somewhere inside llvm. We're generating commands that look like:
<internal clang cc> -cc1 -triple x86_64-apple-macosx11.0.0 ... -include sys_basic.h ... -I $CHPL_HOME/runtime/include ...
So it should be finding sys_basic.h in that include directory. The error message states that it couldn't find "./sys_basic.h" though, so it seems to be appending "./" to the file it's looking for. The function that puts out the error message is openFile
and is called through this call chain:
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 2.1
* frame #0: 0x00000001050988ea libLLVM.dylib`llvm::sys::fs::openFile(llvm::Twine const&, int&, llvm::sys::fs::CreationDisposition, llvm::sys::fs::FileAccess, llvm::sys::fs::OpenFlags, unsigned int)
frame #1: 0x00000001050966ea libLLVM.dylib`llvm::sys::fs::openFileForRead(llvm::Twine const&, int&, llvm::sys::fs::OpenFlags, llvm::SmallVectorImpl<char>*) + 60
frame #2: 0x0000000105098af2 libLLVM.dylib`llvm::sys::fs::openNativeFileForRead(llvm::Twine const&, llvm::sys::fs::OpenFlags, llvm::SmallVectorImpl<char>*) + 29
frame #3: 0x0000000105071d13 libLLVM.dylib`(anonymous namespace)::RealFileSystem::openFileForRead(llvm::Twine const&) + 127
frame #4: 0x000000010506a8f8 libLLVM.dylib`llvm::vfs::FileSystem::getBufferForFile(llvm::Twine const&, long long, bool, bool) + 60
frame #5: 0x0000000100de6eae chpl`clang::driver::RocmInstallationDetector::detectHIPRuntime() + 644
frame #6: 0x0000000100de6987 chpl`clang::driver::RocmInstallationDetector::RocmInstallationDetector(clang::driver::Driver const&, llvm::Triple const&, llvm::opt::ArgList const&, bool, bool) + 1835
frame #7: 0x0000000100e1f017 chpl`clang::driver::toolchains::Darwin::Darwin(clang::driver::Driver const&, llvm::Triple const&, llvm::opt::ArgList const&) + 125
frame #8: 0x0000000100e1f874 chpl`clang::driver::toolchains::DarwinClang::DarwinClang(clang::driver::Driver const&, llvm::Triple const&, llvm::opt::ArgList const&) + 14
frame #9: 0x0000000100dab7c5 chpl`clang::driver::Driver::getToolChain(llvm::opt::ArgList const&, llvm::Triple const&) const + 141
frame #10: 0x0000000100dae93a chpl`clang::driver::Driver::BuildCompilation(llvm::ArrayRef<char const*>) + 3000
frame #11: 0x00000001002dce18 chpl`setupClang(info=0x000000011b39d830, mainFile="$CHPL_HOME/runtime/etc/rtmain.c") at clangUtil.cpp:1639:17
frame #12: 0x00000001002e2c64 chpl`runClang(just_parse_filename=0x0000000000000000) at clangUtil.cpp:2627:3
frame #13: 0x00000001001fc3f0 chpl`codegenPartTwo() at codegen.cpp:2522:5
frame #14: 0x00000001001fbf69 chpl`codegen() at codegen.cpp:2739:3
frame #15: 0x0000000100369830 chpl`runPass(tracker=0x00007ffeefbff230, passIndex=40, isChpldoc=false) at runpasses.cpp:208:3
frame #16: 0x00000001003696ad chpl`runPasses(tracker=0x00007ffeefbff230, isChpldoc=false) at runpasses.cpp:170:5
frame #17: 0x000000010035d342 chpl`main(argc=3, argv=0x00007ffeefbff2a0) at driver.cpp:1831:3
Looking at the arguments just before calling into llvm in frame 11 the argument looks fine. [8] is where it should be finding the header and [34] and [35] are the header hitting the error:
(lldb) p clangInfo->driverArgsCStrings
(std::vector<const char *, std::allocator<const char *> >) $0 = size=44 {
[0] = 0x000000010a620360 "<chapel clang driver invocation>"
[1] = 0x000000010a6202f0 "-I/Users/diten/chapel8/modules/standard"
[2] = 0x000000010a667b70 "-I/Users/diten/chapel8/modules/packages"
[3] = 0x0000000111f61f90 "-I/Users/diten/chapel8/runtime/include/localeModels/flat"
[4] = 0x0000000111f16870 "-I/Users/diten/chapel8/runtime/include/localeModels"
[5] = 0x0000000111fb6640 "-I/Users/diten/chapel8/runtime/include/comm/none"
[6] = 0x000000010a667b00 "-I/Users/diten/chapel8/runtime/include/comm"
[7] = 0x000000010a612910 "-I/Users/diten/chapel8/runtime/include/tasks/qthreads"
[8] = 0x000000010a661d90 "-I/Users/diten/chapel8/runtime/include"
[9] = 0x000000010a662030 "-I/Users/diten/chapel8/runtime/include/qio"
[10] = 0x000000010a6cdbf0 "-I/Users/diten/chapel8/runtime/include/atomics/cstdlib"
[11] = 0x000000010a6e02a0 "-I/Users/diten/chapel8/runtime/include/mem/jemalloc"
[12] = 0x000000010a6d2f20 "-I/Users/diten/chapel8/runtime/include/mem/jemalloc"
[13] = 0x000000010a661fc0 "-I/Users/diten/chapel8/third-party/utf8-decoder"
[14] = 0x000000010a6d2f60 "-DCHPL_JEMALLOC_PREFIX=chpl_je_"
[15] = 0x000000010a6e0160 "-I/Users/diten/chapel8/third-party/hwloc/install/darwin-x86_64-none-llvm-none-flat/include"
[16] = 0x000000011b393e30 "-I/Users/diten/chapel8/third-party/qthread/install/darwin-x86_64-none-llvm-none-flat-jemalloc-bundled/include"
[17] = 0x000000010a6d2fc0 "-I/Users/diten/chapel8/third-party/jemalloc/install/target/darwin-x86_64-none-llvm-none/include"
[18] = 0x000000010a81bdb1 "-I."
[19] = 0x000000010a6e0200 "-I/var/folders/9s/sbzsd5091kj4xklbxfd28zkw000vft/T//chpl-diten.deleteme-KQoZBz"
[20] = 0x000000010a81bde1 "-Wall"
[21] = 0x000000010a81bdf9 "-Werror"
[22] = 0x000000010a81be11 "-Wpointer-arith"
[23] = 0x000000010a81be29 "-Wwrite-strings"
[24] = 0x000000010a81be41 "-Wno-strict-aliasing"
[25] = 0x000000010a81be59 "-Wmissing-declarations"
[26] = 0x000000010a81be71 "-Wmissing-prototypes"
[27] = 0x000000010a81be89 "-Wstrict-prototypes"
[28] = 0x000000010a6e02e0 "-Wmissing-format-attribute"
[29] = 0x000000010a81beb9 "-Wno-unused-function"
[30] = 0x000000010a661d10 "-I/Users/diten/chapel8/modules/internal"
[31] = 0x000000010a81bee9 "-DCHPL_GEN_CODE"
[32] = 0x000000010a81bf01 "-pthread"
[33] = 0x000000010a81bf19 "-I/usr/local/include"
[34] = 0x000000010a81bf31 "-include"
[35] = 0x000000010a81bf49 "sys_basic.h"
[36] = 0x000000010a81bf61 "-include"
[37] = 0x000000010a81bf79 "ctype.h"
[38] = 0x000000010a81bf91 "-include"
[39] = 0x000000010a81bfa9 "wctype.h"
[40] = 0x000000010a81bfc1 "-include"
[41] = 0x000000010a6cdc30 "llvm/chapel_libc_wrapper.h"
[42] = 0x000000010a81bff1 "-c"
[43] = 0x000000010a65dd90 "/Users/diten/chapel8/runtime/etc/rtmain.c"
@daviditen - it would be pretty easy for us to give the full path to sys_basic.h.
Could you try changing this code in clangUtil.cpp to do that manually to see if that solves the issue?
// Always include sys_basic because it might change the
// behaviour of macros!
clangOtherArgs.push_back("-include");
clangOtherArgs.push_back("sys_basic.h");
If I give the full path to sys_basic.h
it goes on to the next header it wants to find and errors about that, but with a different include path tacked on - this time [1] from above.
In file included from <built-in>:1:
/Users/diten/chapel8/runtime/include/sys_basic.h:70:10: fatal error: cannot open file '/Users/diten/chapel8/modules/standard/sys/types.h': No such file or directory
#include <sys/types.h>
^
1 error generated.
error: error running clang during code generation
I am not seeing this problem on my system -- here is the corresponding line in case that helps.
Can you check if, on the failing system, we see arguments along the lines of -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/local/include
?
See also
Does enabling this for system
in addition to bundled
address the problem?
My full <internal clang cc>
line is:
<internal clang cc> -cc1 -triple x86_64-apple-macosx11.0.0 -Wundef-prefix=TARGET_OS_ -Werror=undef-prefix -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -emit-obj -mrelax-all --mrelax-relocations -disable-free -disable-llvm-verifier -discard-value-names -main-file-name rtmain.c -mrelocation-model pic -pic-level 2 -mframe-pointer=all -fno-rounding-math -munwind-tables -fcompatibility-qualified-id-block-type-checking -fvisibility-inlines-hidden-static-local-var -target-cpu penryn -tune-cpu generic -debugger-tuning=lldb -target-linker-version 711 -resource-dir /usr/local/Cellar/llvm@12/12.0.1_1/lib/clang/12.0.1 -include /Users/diten/chapel8/runtime/include/sys_basic.h -include ctype.h -include wctype.h -include llvm/chapel_libc_wrapper.h -I /Users/diten/chapel8/modules/standard -I /Users/diten/chapel8/modules/packages -I /Users/diten/chapel8/runtime/include/localeModels/flat -I /Users/diten/chapel8/runtime/include/localeModels -I /Users/diten/chapel8/runtime/include/comm/none -I /Users/diten/chapel8/runtime/include/comm -I /Users/diten/chapel8/runtime/include/tasks/qthreads -I /Users/diten/chapel8/runtime/include -I /Users/diten/chapel8/runtime/include/qio -I /Users/diten/chapel8/runtime/include/atomics/cstdlib -I /Users/diten/chapel8/runtime/include/mem/jemalloc -I /Users/diten/chapel8/runtime/include/mem/jemalloc -I /Users/diten/chapel8/third-party/utf8-decoder -D CHPL_JEMALLOC_PREFIX=chpl_je_ -I /Users/diten/chapel8/third-party/hwloc/install/darwin-x86_64-none-llvm-none-flat/include -I /Users/diten/chapel8/third-party/qthread/install/darwin-x86_64-none-llvm-none-flat-jemalloc-bundled/include -I /Users/diten/chapel8/third-party/jemalloc/install/target/darwin-x86_64-none-llvm-none/include -I . -I /var/folders/9s/sbzsd5091kj4xklbxfd28zkw000vft/T//chpl-diten.deleteme-O4P70b -I /Users/diten/chapel8/modules/internal -D CHPL_GEN_CODE -I /usr/local/include -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk/usr/local/include -internal-isystem /usr/local/Cellar/llvm@12/12.0.1_1/lib/clang/12.0.1/include -internal-externc-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk/usr/include -Wall -Werror -Wpointer-arith -Wwrite-strings -Wno-strict-aliasing -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wmissing-format-attribute -Wno-unused-function -fconst-strings -fdebug-compilation-dir /Users/diten/chapel8/test/release/examples -ferror-limit 19 -pthread -stack-protector 1 -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fmax-type-align=16 -o rtmain.o -x c /Users/diten/chapel8/runtime/etc/rtmain.c
I see -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk/usr/local/include
in there. I see that mine has MacOSX11.sdk but I also have MacOSX12.sdk sitting next to it. Should it be detecting/using the newest version?
Changing the line in chpl_llvm.py to:
if (target_platform == "darwin" and (llvm_val == "bundled" or llvm_val == "system")):
didn't change the behavior.
@daviditen - are you able to compile simple C programs with the clang
provided in CHPL_TARGET_CC
? (Edit - I tried that, and I can)
I am doing the brew install llvm@12
on my system to see if I see the same problems. I do see the same problems.
Has anybody checked if this problem also occurs with the bundled LLVM?
I'm pretty sure the bundled LLVM is fine. I'm rebuilding it right now to double check though.
My build of the bundled LLVM 12 works OK on my system. Which makes it seem even weirder. What about the bundled version is different? Does the homebrew clang have some special weirdness? Or are we invoking them differently?
Confirmed that bundled is also OK on my system.
I do see some differences in the arguments to <internal clang cc>
.
Bundled has -target-sdk-version=12.1
but system is missing a -target-sdk-version
option.
System has -disable-llvm-verifier -discard-value-names
but bundled is missing these.
Bundled has:
-resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/13.0.0
-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
-internal-isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/local/include
-internal-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/13.0.0/include
-internal-externc-isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include
System has:
-resource-dir /usr/local/Cellar/llvm@12/12.0.1_1/lib/clang/12.0.1
-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk
-internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk/usr/local/include
-internal-isystem /usr/local/Cellar/llvm@12/12.0.1_1/lib/clang/12.0.1/include
-internal-externc-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk/usr/include
Though I think we still want to get to the bottom of this, I was also curious whether @ronawho had better or different luck with llvm@13 now that it's been enabled.
In the OP I saw issues for system/brew llvm13 and I reproduced those just now. Switching to a bundled llvm 13 build did work for me though.
In the OP I saw issues for system/brew llvm13
My bad, I was just going off of the issue title.
In the OP I saw issues for system/brew llvm13
My bad, I was just going off of the issue title.
Don't think you were offended, but I didn't mean that as RTFM, just noting that I saw the issue before and continue to see it now (when I opened this I don't think we had official support for 13, so I think it made sense to recheck)
I thought, since it works in LLVM 11, but not in LLVM 12, there must be some kind of LLVM commit that changed the behavior. So I have built a local version of LLVM 11 (configured in a manner I think is similar to the Homebrew LLVM 11) and installed it and did not see the problem. But, then, when I build a local version of LLVM 12 and installed it I didn't see the problem either.
So, that means to me that there must be something different about how Homebrew is configuring llvm@12 vs llvm@11 vs my test build that is related to the problem. I'm going to try some variations on my experiment to check a few things I might have messed up.
So far, I haven't been able to reproduce the issue with my own install, but I have with building the Homebrew formula (with brew install --build-from-source llvm@12
).
I am trying one more thing to reduce differences in my cmake but after that will investigate if the patches that Homebrew is applying make any difference. You can see the formula with brew edit llvm@12
and these are the patches applied on mac os x:
# Fix crash in clangd when built with GCC <6. Remove in LLVM 13
# https://github.com/clangd/clangd/issues/800
# https://github.com/Homebrew/homebrew-core/issues/84365
patch do
url "https://github.com/llvm/llvm-project/commit/ec1fb9533305e9bd69294ede7e5e7d9befbb2225.patch?full_index=1"
sha256 "b80a5718420c789588f3392366ac15485e43bea8e81adb14424c3cad4afa7315"
end
# Fix parallel builds. Remove in LLVM 13.
# https://reviews.llvm.org/D106305
# https://lists.llvm.org/pipermail/llvm-dev/2021-July/151665.html
patch do
url "https://github.com/llvm/llvm-project/commit/b31080c596246bc26d2493cfd5e07f053cf9541c.patch?full_index=1"
sha256 "b4576303404e68100dc396d2414d6740c5bfd0162979d22152a688d1e7307379"
end
I couldn't reproduce it with my own build as similar as possible to Homebrew in cmake invocation and applying the same patches. Now I think it has something to do with log entries in the homebrew build along these lines:
clang++ called with: -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/tmp/llvmA12-20220310-93596-19c20g8/llvm-project-12.0.1.src/llvm/build/utils/TableGen -I/tmp/llvmA12-20220310-93596-19c20g8/llvm-project-12.0.1.src/llvm/utils/TableGen -I/tmp/llvmA12-20220310-93596-19c20g8/llvm-project-12.0.1.src/llvm/build/include -I/tmp/llvmA12-20220310-93596-19c20g8/llvm-project-12.0.1.src/llvm/include -stdlib=libc++ -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-class-memaccess -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -O2 -g -DNDEBUG -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -std=c++14 -MD -MT utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o -MF CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o.d -o CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o -c /tmp/llvmA12-20220310-93596-19c20g8/llvm-project-12.0.1.src/llvm/utils/TableGen/AsmMatcherEmitter.cpp
superenv removed: -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wimplicit-fallthrough -Wcovered-switch-default -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -O2 -g -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk
superenv added: -pipe -w -Os -march=nehalem -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk --sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -isystem/usr/local/include -isystem/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/readline/include -I/usr/local/opt/sqlite/include
superenv executed: clang++ -pipe -w -Os -march=nehalem -stdlib=libc++ -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/tmp/llvmA12-20220310-93596-19c20g8/llvm-project-12.0.1.src/llvm/build/utils/TableGen -I/tmp/llvmA12-20220310-93596-19c20g8/llvm-project-12.0.1.src/llvm/utils/TableGen -I/tmp/llvmA12-20220310-93596-19c20g8/llvm-project-12.0.1.src/llvm/build/include -I/tmp/llvmA12-20220310-93596-19c20g8/llvm-project-12.0.1.src/llvm/include -stdlib=libc++ -fPIC -fvisibility-inlines-hidden -Wno-unused-parameter -Wno-long-long -Wno-class-memaccess -Wno-noexcept-type -DNDEBUG -std=c++14 -MD -MT utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o -MF CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o.d -o CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o -c /tmp/llvmA12-20220310-93596-19c20g8/llvm-project-12.0.1.src/llvm/utils/TableGen/AsmMatcherEmitter.cpp -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk --sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -isystem/usr/local/include -isystem/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/readline/include -I/usr/local/opt/sqlite/include
Note that, superenv
, whatever it is, is adding -isysroot
--sysroot
and -isystem
arguments. My current theory is that these are affecting the behavior, somehow.
Here are the corresponding log lines from an llvm@11 build
clang++ called with: -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/tmp/llvmA11-20220311-63465-hmz8yy/llvm-project-11.1.0.src/llvm/build/utils/TableGen -I/tmp/llvmA11-20220311-63465-hmz8yy/llvm-project-11.1.0.src/llvm/utils/TableGen -I/tmp/llvmA11-20220311-63465-hmz8yy/llvm-project-11.1.0.src/llvm/build/include -I/tmp/llvmA11-20220311-63465-hmz8yy/llvm-project-11.1.0.src/llvm/include -stdlib=libc++ -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-class-memaccess -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -O2 -g -DNDEBUG -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -std=c++14 -MD -MT utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o -MF CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o.d -o CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o -c /tmp/llvmA11-20220311-63465-hmz8yy/llvm-project-11.1.0.src/llvm/utils/TableGen/AsmMatcherEmitter.cpp
superenv removed: -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wimplicit-fallthrough -Wcovered-switch-default -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -O2 -g -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk
superenv added: -pipe -w -Os -march=nehalem -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk --sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -isystem/usr/local/include -isystem/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/readline/include -I/usr/local/opt/sqlite/include
superenv executed: clang++ -pipe -w -Os -march=nehalem -stdlib=libc++ -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/tmp/llvmA11-20220311-63465-hmz8yy/llvm-project-11.1.0.src/llvm/build/utils/TableGen -I/tmp/llvmA11-20220311-63465-hmz8yy/llvm-project-11.1.0.src/llvm/utils/TableGen -I/tmp/llvmA11-20220311-63465-hmz8yy/llvm-project-11.1.0.src/llvm/build/include -I/tmp/llvmA11-20220311-63465-hmz8yy/llvm-project-11.1.0.src/llvm/include -stdlib=libc++ -fPIC -fvisibility-inlines-hidden -Wno-unused-parameter -Wno-long-long -Wno-class-memaccess -Wno-noexcept-type -DNDEBUG -std=c++14 -MD -MT utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o -MF CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o.d -o CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o -c /tmp/llvmA11-20220311-63465-hmz8yy/llvm-project-11.1.0.src/llvm/utils/TableGen/AsmMatcherEmitter.cpp -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk --sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -isystem/usr/local/include -isystem/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/readline/include -I/usr/local/opt/sqlite/include
superenv is implemented in as compiler wrappers in /usr/local/Homebrew/Library/Homebrew/shims/mac/super and I'm trying to edit the script to allow -g
(so at least I can debug clang built this way). Also, if we put env :std
in the formula (near the top, I guess?) it will disable superenv.
I now have a version showing the error and one that doesn't that I can debug. But, I haven't gotten very far.
In order to get to this point, I needed to get the Homebrew build to include -DCMAKE_BUILD_TYPE=RelWithDebInfo
. (brew edit llvm@12
brew install --build-from-source llvm@12
)
Another thing that tripped me up for a while is that lldb sets SDKROOT
which changes the behavior of this code.
Here is a better stack trace for the failure error
It looks like getFileAndSuggestModule is trying to open "./sys_basic.h"
but then this fails and the error code is compared against llvm::errc::no_such_file_or_directory
. This comparison succeeds on a non-homebrew build of clang but fails with the homebrew build. What is weird is that EC.message()
is "No such file or directory" in both cases. This feels like a case where we have different headers vs libraries but I don't know if that is the cause.
I see this
otool -L /usr/local/opt/llvm@12/lib/libLLVM.dylib
...
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)
otool -L bin/darwin-x86_64/chpl
...
/usr/local/opt/llvm@12/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
This is bad because we statically link with the LLVM libraries, so that means that the various .a files are built with one libc++ headers and then the binary tries to call different functions in the libc++ lib. This is related to #18727 and #2702.
I think this is the cause of the problem but I'm not sure how to fix it. @daviditen I am timing out but maybe using these otool commands to check -- maybe you can figure out a way we can link chpl
against the system libc++ instead of the homebrew installed one that comes in the LLVM package.
I have a start at something to fix it with this patch
diff --git a/util/chplenv/chpl_llvm.py b/util/chplenv/chpl_llvm.py
index debc8316cc..e27a77b1cf 100755
--- a/util/chplenv/chpl_llvm.py
+++ b/util/chplenv/chpl_llvm.py
@@ -557,6 +561,16 @@ def get_host_compile_args():
llvm_config = get_llvm_config()
if llvm_val == 'system':
+ # On Mac OS X with Homebrew, apply a workaround for issue #19217.
+ # This avoids finding headers in the libc++ installed by llvm@12 e.g.
+ host_platform = chpl_platform.get('host')
+ if host_platform == "darwin":
+ homebrew_prefix = chpl_platform.get_homebrew_prefix()
+ if homebrew_prefix:
+ system.append("-isysroot")
+ system.append("/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk")
+ system.append("-I" + "/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include")
+
# Ubuntu 16.04 needed -fno-rtti for LLVM 3.7
# tested on that system after installing
# llvm-3.7-dev llvm-3.7 clang-3.7 libclang-3.7-dev libedit-dev
@@ -618,6 +632,17 @@ def get_host_link_args():
if llvm_val == 'system':
+ # On Mac OS X with Homebrew, apply a workaround for issue #19217.
+ # This avoids linking with the libc++ installed by llvm@12 e.g.
+ host_platform = chpl_platform.get('host')
+ if host_platform == "darwin":
+ homebrew_prefix = chpl_platform.get_homebrew_prefix()
+ if homebrew_prefix:
+ # -isysroot is just about includes
+ # and -Wl,-syslibroot seems to have no effect
+ system.append("-L" + "/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/lib")
+
+
# Decide whether to try to link statically or dynamically.
# Future work: consider using 'llvm-config --shared-mode'
# to make this choice.
What I need to do next is to compute the Mac OS X SDKROOT that Homebrew clang wants to use, somehow.
I've created PR #19445 to address this problem
I have opened a Homebrew issue https://github.com/Homebrew/homebrew-core/issues/96915 to report this problem to Homebrew.
I've opened an LLVM project issue https://github.com/llvm/llvm-project/issues/54438 to report this problem there.
While looking into https://github.com/chapel-lang/chapel/issues/17825 I wanted to try out
llvm@12
, but ran into a few issues (note that I first ran into this on aarch64 mac, but I can reproduce on x86 as well.)First is that if
llvm@11
andllvm@12
are both installedllvm@11
is preferred, but I don't think this is intentional. Reversing the order ofllvm_versions
findsllvm@12
, so seems like either that or the code we use to iterate to find homebrew llvm versions needs to be reversed.When actually using
llvm@12
(and 13) I get errors about not being able to finder headers when compiling any program:Here's my chplenv: