llvm@12 issues on mac

[ronawho]

While looking into https://github.com/chapel-lang/chapel/issues/17825 I wanted to try out llvm@12, but ran into a few issues (note that I first ran into this on aarch64 mac, but I can reproduce on x86 as well.)

First is that if llvm@11 and llvm@12 are both installed llvm@11 is preferred, but I don't think this is intentional. Reversing the order of llvm_versions finds llvm@12, so seems like either that or the code we use to iterate to find homebrew llvm versions needs to be reversed.

When actually using llvm@12 (and 13) I get errors about not being able to finder headers when compiling any program:

chpl examples/hello.chpl
<built-in>:1:10: fatal error: cannot open file './sys_basic.h': No such file or directory
#include "sys_basic.h"
         ^
1 error generated.
error: error running clang during code generation

Here's my chplenv:

./util/printchplenv --all --anonymize
CHPL_HOST_PLATFORM: darwin
CHPL_HOST_COMPILER: clang
  CHPL_HOST_CC: clang
  CHPL_HOST_CXX: clang++
CHPL_HOST_ARCH: arm64
CHPL_TARGET_PLATFORM: darwin
CHPL_TARGET_COMPILER: llvm
  CHPL_TARGET_CC: /opt/homebrew/Cellar/llvm@12/12.0.1_1/bin/clang
  CHPL_TARGET_CXX: /opt/homebrew/Cellar/llvm@12/12.0.1_1/bin/clang++
CHPL_TARGET_ARCH: arm64
CHPL_TARGET_CPU: native
CHPL_LOCALE_MODEL: flat
  CHPL_GPU_CODEGEN: none
CHPL_COMM: none
CHPL_TASKS: fifo
CHPL_LAUNCHER: none
CHPL_TIMERS: generic
CHPL_UNWIND: none
CHPL_HOST_MEM: cstdlib
CHPL_MEM: jemalloc
CHPL_ATOMICS: cstdlib
CHPL_GMP: bundled
CHPL_HWLOC: none
CHPL_RE2: bundled
CHPL_LLVM: system
  CHPL_LLVM_CONFIG: /opt/homebrew/opt/llvm@12/bin/llvm-config
CHPL_AUX_FILESYS: none
CHPL_LIB_PIC: none
CHPL_SANITIZE: none
CHPL_SANITIZE_EXE: none

[ronawho]

FYI @mppf and @daviditen

[aconsroe-hpe]

For the ordering, I just ran into this (on linux) and tracked it down to this loop where the break only breaks out of the inner. I suspect that is the same bug you're seeing

https://github.com/chapel-lang/chapel/blob/86d1584a9b6721850720289961e227ecf7fb77f9/util/chplenv/chpl_llvm.py#L144-L149

[daviditen]

The ordering issue is resolved with #19278.

The problem with not finding headers seems like it's happening somewhere inside llvm. We're generating commands that look like:

<internal clang cc> -cc1 -triple x86_64-apple-macosx11.0.0 ... -include sys_basic.h ... -I $CHPL_HOME/runtime/include ...

So it should be finding sys_basic.h in that include directory. The error message states that it couldn't find "./sys_basic.h" though, so it seems to be appending "./" to the file it's looking for. The function that puts out the error message is openFile and is called through this call chain:

* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 2.1
  * frame #0: 0x00000001050988ea libLLVM.dylib`llvm::sys::fs::openFile(llvm::Twine const&, int&, llvm::sys::fs::CreationDisposition, llvm::sys::fs::FileAccess, llvm::sys::fs::OpenFlags, unsigned int)
    frame #1: 0x00000001050966ea libLLVM.dylib`llvm::sys::fs::openFileForRead(llvm::Twine const&, int&, llvm::sys::fs::OpenFlags, llvm::SmallVectorImpl<char>*) + 60
    frame #2: 0x0000000105098af2 libLLVM.dylib`llvm::sys::fs::openNativeFileForRead(llvm::Twine const&, llvm::sys::fs::OpenFlags, llvm::SmallVectorImpl<char>*) + 29
    frame #3: 0x0000000105071d13 libLLVM.dylib`(anonymous namespace)::RealFileSystem::openFileForRead(llvm::Twine const&) + 127
    frame #4: 0x000000010506a8f8 libLLVM.dylib`llvm::vfs::FileSystem::getBufferForFile(llvm::Twine const&, long long, bool, bool) + 60
    frame #5: 0x0000000100de6eae chpl`clang::driver::RocmInstallationDetector::detectHIPRuntime() + 644
    frame #6: 0x0000000100de6987 chpl`clang::driver::RocmInstallationDetector::RocmInstallationDetector(clang::driver::Driver const&, llvm::Triple const&, llvm::opt::ArgList const&, bool, bool) + 1835
    frame #7: 0x0000000100e1f017 chpl`clang::driver::toolchains::Darwin::Darwin(clang::driver::Driver const&, llvm::Triple const&, llvm::opt::ArgList const&) + 125
    frame #8: 0x0000000100e1f874 chpl`clang::driver::toolchains::DarwinClang::DarwinClang(clang::driver::Driver const&, llvm::Triple const&, llvm::opt::ArgList const&) + 14
    frame #9: 0x0000000100dab7c5 chpl`clang::driver::Driver::getToolChain(llvm::opt::ArgList const&, llvm::Triple const&) const + 141
    frame #10: 0x0000000100dae93a chpl`clang::driver::Driver::BuildCompilation(llvm::ArrayRef<char const*>) + 3000
    frame #11: 0x00000001002dce18 chpl`setupClang(info=0x000000011b39d830, mainFile="$CHPL_HOME/runtime/etc/rtmain.c") at clangUtil.cpp:1639:17
    frame #12: 0x00000001002e2c64 chpl`runClang(just_parse_filename=0x0000000000000000) at clangUtil.cpp:2627:3
    frame #13: 0x00000001001fc3f0 chpl`codegenPartTwo() at codegen.cpp:2522:5
    frame #14: 0x00000001001fbf69 chpl`codegen() at codegen.cpp:2739:3
    frame #15: 0x0000000100369830 chpl`runPass(tracker=0x00007ffeefbff230, passIndex=40, isChpldoc=false) at runpasses.cpp:208:3
    frame #16: 0x00000001003696ad chpl`runPasses(tracker=0x00007ffeefbff230, isChpldoc=false) at runpasses.cpp:170:5
    frame #17: 0x000000010035d342 chpl`main(argc=3, argv=0x00007ffeefbff2a0) at driver.cpp:1831:3

Looking at the arguments just before calling into llvm in frame 11 the argument looks fine. [8] is where it should be finding the header and [34] and [35] are the header hitting the error:

(lldb) p clangInfo->driverArgsCStrings
(std::vector<const char *, std::allocator<const char *> >) $0 = size=44 {
  [0] = 0x000000010a620360 "<chapel clang driver invocation>"
  [1] = 0x000000010a6202f0 "-I/Users/diten/chapel8/modules/standard"
  [2] = 0x000000010a667b70 "-I/Users/diten/chapel8/modules/packages"
  [3] = 0x0000000111f61f90 "-I/Users/diten/chapel8/runtime/include/localeModels/flat"
  [4] = 0x0000000111f16870 "-I/Users/diten/chapel8/runtime/include/localeModels"
  [5] = 0x0000000111fb6640 "-I/Users/diten/chapel8/runtime/include/comm/none"
  [6] = 0x000000010a667b00 "-I/Users/diten/chapel8/runtime/include/comm"
  [7] = 0x000000010a612910 "-I/Users/diten/chapel8/runtime/include/tasks/qthreads"
  [8] = 0x000000010a661d90 "-I/Users/diten/chapel8/runtime/include"
  [9] = 0x000000010a662030 "-I/Users/diten/chapel8/runtime/include/qio"
  [10] = 0x000000010a6cdbf0 "-I/Users/diten/chapel8/runtime/include/atomics/cstdlib"
  [11] = 0x000000010a6e02a0 "-I/Users/diten/chapel8/runtime/include/mem/jemalloc"
  [12] = 0x000000010a6d2f20 "-I/Users/diten/chapel8/runtime/include/mem/jemalloc"
  [13] = 0x000000010a661fc0 "-I/Users/diten/chapel8/third-party/utf8-decoder"
  [14] = 0x000000010a6d2f60 "-DCHPL_JEMALLOC_PREFIX=chpl_je_"
  [15] = 0x000000010a6e0160 "-I/Users/diten/chapel8/third-party/hwloc/install/darwin-x86_64-none-llvm-none-flat/include"
  [16] = 0x000000011b393e30 "-I/Users/diten/chapel8/third-party/qthread/install/darwin-x86_64-none-llvm-none-flat-jemalloc-bundled/include"
  [17] = 0x000000010a6d2fc0 "-I/Users/diten/chapel8/third-party/jemalloc/install/target/darwin-x86_64-none-llvm-none/include"
  [18] = 0x000000010a81bdb1 "-I."
  [19] = 0x000000010a6e0200 "-I/var/folders/9s/sbzsd5091kj4xklbxfd28zkw000vft/T//chpl-diten.deleteme-KQoZBz"
  [20] = 0x000000010a81bde1 "-Wall"
  [21] = 0x000000010a81bdf9 "-Werror"
  [22] = 0x000000010a81be11 "-Wpointer-arith"
  [23] = 0x000000010a81be29 "-Wwrite-strings"
  [24] = 0x000000010a81be41 "-Wno-strict-aliasing"
  [25] = 0x000000010a81be59 "-Wmissing-declarations"
  [26] = 0x000000010a81be71 "-Wmissing-prototypes"
  [27] = 0x000000010a81be89 "-Wstrict-prototypes"
  [28] = 0x000000010a6e02e0 "-Wmissing-format-attribute"
  [29] = 0x000000010a81beb9 "-Wno-unused-function"
  [30] = 0x000000010a661d10 "-I/Users/diten/chapel8/modules/internal"
  [31] = 0x000000010a81bee9 "-DCHPL_GEN_CODE"
  [32] = 0x000000010a81bf01 "-pthread"
  [33] = 0x000000010a81bf19 "-I/usr/local/include"
  [34] = 0x000000010a81bf31 "-include"
  [35] = 0x000000010a81bf49 "sys_basic.h"
  [36] = 0x000000010a81bf61 "-include"
  [37] = 0x000000010a81bf79 "ctype.h"
  [38] = 0x000000010a81bf91 "-include"
  [39] = 0x000000010a81bfa9 "wctype.h"
  [40] = 0x000000010a81bfc1 "-include"
  [41] = 0x000000010a6cdc30 "llvm/chapel_libc_wrapper.h"
  [42] = 0x000000010a81bff1 "-c"
  [43] = 0x000000010a65dd90 "/Users/diten/chapel8/runtime/etc/rtmain.c"

[mppf]

@daviditen - it would be pretty easy for us to give the full path to sys_basic.h.

Could you try changing this code in clangUtil.cpp to do that manually to see if that solves the issue?

  // Always include sys_basic because it might change the
  // behaviour of macros!
  clangOtherArgs.push_back("-include");
  clangOtherArgs.push_back("sys_basic.h");

[daviditen]

If I give the full path to sys_basic.h it goes on to the next header it wants to find and errors about that, but with a different include path tacked on - this time [1] from above.

In file included from <built-in>:1:
/Users/diten/chapel8/runtime/include/sys_basic.h:70:10: fatal error: cannot open file '/Users/diten/chapel8/modules/standard/sys/types.h': No such file or directory
#include <sys/types.h>
         ^
1 error generated.
error: error running clang during code generation

[mppf]

I am not seeing this problem on my system -- here is the corresponding line in case that helps.

``` -cc1 -triple x86_64-apple-macosx12.0.0 -Wundef-prefix=TARGET_OS_ -Werror=undef-prefix -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name rtmain.c -mrelocation-model pic -pic-level 2 -mframe-pointer=all -fno-rounding-math -munwind-tables -fcompatibility-qualified-id-block-type-checking -target-cpu penryn -debugger-tuning=lldb -target-linker-version 711 -resource-dir /usr/local/Cellar/llvm@11/11.1.0_4/lib/clang/11.1.0 -include sys_basic.h -include ctype.h -include wctype.h -include llvm/chapel_libc_wrapper.h -I /Users/mferguson/w/main/modules/standard -I /Users/mferguson/w/main/modules/packages -I /Users/mferguson/w/main/runtime/include/localeModels/flat -I /Users/mferguson/w/main/runtime/include/localeModels -I /Users/mferguson/w/main/runtime/include/comm/none -I /Users/mferguson/w/main/runtime/include/comm -I /Users/mferguson/w/main/runtime/include/tasks/qthreads -I /Users/mferguson/w/main/runtime/include -I /Users/mferguson/w/main/runtime/include/qio -I /Users/mferguson/w/main/runtime/include/atomics/cstdlib -I /Users/mferguson/w/main/runtime/include/mem/jemalloc -I /Users/mferguson/w/main/runtime/include/mem/jemalloc -I /Users/mferguson/w/main/third-party/utf8-decoder -D CHPL_JEMALLOC_PREFIX=chpl_je_ -I /Users/mferguson/w/main/third-party/gmp/install/darwin-x86_64-native-llvm-none/include -I /Users/mferguson/w/main/third-party/hwloc/install/darwin-x86_64-native-llvm-none-flat/include -I /Users/mferguson/w/main/third-party/qthread/install/darwin-x86_64-native-llvm-none-flat-jemalloc-bundled/include -I /Users/mferguson/w/main/third-party/jemalloc/install/target/darwin-x86_64-native-llvm-none/include -I /Users/mferguson/w/main/third-party/re2/install/darwin-x86_64-native-llvm-none/include -I . -I /var/folders/b6/1lwlz22n5d53878_x52vp3kr0007pg/T//chpl-mferguson.deleteme-6KrqdI -I /Users/mferguson/w/main/modules/internal -D CHPL_GEN_CODE -I /usr/local/include -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/local/include -internal-isystem /usr/local/Cellar/llvm@11/11.1.0_4/lib/clang/11.1.0/include -internal-externc-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include -Wall -Werror -Wpointer-arith -Wwrite-strings -Wno-strict-aliasing -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wmissing-format-attribute -Wno-unused-function -fconst-strings -fdebug-compilation-dir /Users/mferguson/w/main -ferror-limit 19 -pthread -stack-protector 1 -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fmax-type-align=16 -o rtmain.o -x c /Users/mferguson/w/main/runtime/etc/rtmain.c ```

Can you check if, on the failing system, we see arguments along the lines of -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/local/include ?

[mppf]

See also

https://github.com/chapel-lang/chapel/blob/b1287c975dcf12b8cfd2f37b559532c6ccd04479/util/chplenv/chpl_llvm.py#L362-L383

Does enabling this for system in addition to bundled address the problem?

[daviditen]

My full <internal clang cc> line is:

<internal clang cc> -cc1 -triple x86_64-apple-macosx11.0.0 -Wundef-prefix=TARGET_OS_ -Werror=undef-prefix -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -emit-obj -mrelax-all --mrelax-relocations -disable-free -disable-llvm-verifier -discard-value-names -main-file-name rtmain.c -mrelocation-model pic -pic-level 2 -mframe-pointer=all -fno-rounding-math -munwind-tables -fcompatibility-qualified-id-block-type-checking -fvisibility-inlines-hidden-static-local-var -target-cpu penryn -tune-cpu generic -debugger-tuning=lldb -target-linker-version 711 -resource-dir /usr/local/Cellar/llvm@12/12.0.1_1/lib/clang/12.0.1 -include /Users/diten/chapel8/runtime/include/sys_basic.h -include ctype.h -include wctype.h -include llvm/chapel_libc_wrapper.h -I /Users/diten/chapel8/modules/standard -I /Users/diten/chapel8/modules/packages -I /Users/diten/chapel8/runtime/include/localeModels/flat -I /Users/diten/chapel8/runtime/include/localeModels -I /Users/diten/chapel8/runtime/include/comm/none -I /Users/diten/chapel8/runtime/include/comm -I /Users/diten/chapel8/runtime/include/tasks/qthreads -I /Users/diten/chapel8/runtime/include -I /Users/diten/chapel8/runtime/include/qio -I /Users/diten/chapel8/runtime/include/atomics/cstdlib -I /Users/diten/chapel8/runtime/include/mem/jemalloc -I /Users/diten/chapel8/runtime/include/mem/jemalloc -I /Users/diten/chapel8/third-party/utf8-decoder -D CHPL_JEMALLOC_PREFIX=chpl_je_ -I /Users/diten/chapel8/third-party/hwloc/install/darwin-x86_64-none-llvm-none-flat/include -I /Users/diten/chapel8/third-party/qthread/install/darwin-x86_64-none-llvm-none-flat-jemalloc-bundled/include -I /Users/diten/chapel8/third-party/jemalloc/install/target/darwin-x86_64-none-llvm-none/include -I . -I /var/folders/9s/sbzsd5091kj4xklbxfd28zkw000vft/T//chpl-diten.deleteme-O4P70b -I /Users/diten/chapel8/modules/internal -D CHPL_GEN_CODE -I /usr/local/include -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk/usr/local/include -internal-isystem /usr/local/Cellar/llvm@12/12.0.1_1/lib/clang/12.0.1/include -internal-externc-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk/usr/include -Wall -Werror -Wpointer-arith -Wwrite-strings -Wno-strict-aliasing -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wmissing-format-attribute -Wno-unused-function -fconst-strings -fdebug-compilation-dir /Users/diten/chapel8/test/release/examples -ferror-limit 19 -pthread -stack-protector 1 -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fmax-type-align=16 -o rtmain.o -x c /Users/diten/chapel8/runtime/etc/rtmain.c

I see -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk/usr/local/include in there. I see that mine has MacOSX11.sdk but I also have MacOSX12.sdk sitting next to it. Should it be detecting/using the newest version?

Changing the line in chpl_llvm.py to: if (target_platform == "darwin" and (llvm_val == "bundled" or llvm_val == "system")):

didn't change the behavior.

[mppf]

@daviditen - are you able to compile simple C programs with the clang provided in CHPL_TARGET_CC ? (Edit - I tried that, and I can)

I am doing the brew install llvm@12 on my system to see if I see the same problems. I do see the same problems.

[mppf]

Has anybody checked if this problem also occurs with the bundled LLVM?

[daviditen]

I'm pretty sure the bundled LLVM is fine. I'm rebuilding it right now to double check though.

[mppf]

My build of the bundled LLVM 12 works OK on my system. Which makes it seem even weirder. What about the bundled version is different? Does the homebrew clang have some special weirdness? Or are we invoking them differently?

[daviditen]

Confirmed that bundled is also OK on my system.

I do see some differences in the arguments to <internal clang cc>.

Bundled has -target-sdk-version=12.1 but system is missing a -target-sdk-version option. System has -disable-llvm-verifier -discard-value-names but bundled is missing these.

Bundled has:

-resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/13.0.0
-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
-internal-isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/local/include
-internal-isystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/13.0.0/include
-internal-externc-isystem /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include

System has:

-resource-dir /usr/local/Cellar/llvm@12/12.0.1_1/lib/clang/12.0.1
-isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk
-internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk/usr/local/include
-internal-isystem /usr/local/Cellar/llvm@12/12.0.1_1/lib/clang/12.0.1/include
-internal-externc-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX11.sdk/usr/include

[bradcray]

Though I think we still want to get to the bottom of this, I was also curious whether @ronawho had better or different luck with llvm@13 now that it's been enabled.

[ronawho]

In the OP I saw issues for system/brew llvm13 and I reproduced those just now. Switching to a bundled llvm 13 build did work for me though.

[bradcray]

In the OP I saw issues for system/brew llvm13

My bad, I was just going off of the issue title.

[ronawho]

In the OP I saw issues for system/brew llvm13

My bad, I was just going off of the issue title.

Don't think you were offended, but I didn't mean that as RTFM, just noting that I saw the issue before and continue to see it now (when I opened this I don't think we had official support for 13, so I think it made sense to recheck)

[mppf]

I thought, since it works in LLVM 11, but not in LLVM 12, there must be some kind of LLVM commit that changed the behavior. So I have built a local version of LLVM 11 (configured in a manner I think is similar to the Homebrew LLVM 11) and installed it and did not see the problem. But, then, when I build a local version of LLVM 12 and installed it I didn't see the problem either.

So, that means to me that there must be something different about how Homebrew is configuring llvm@12 vs llvm@11 vs my test build that is related to the problem. I'm going to try some variations on my experiment to check a few things I might have messed up.

[mppf]

So far, I haven't been able to reproduce the issue with my own install, but I have with building the Homebrew formula (with brew install --build-from-source llvm@12).

I am trying one more thing to reduce differences in my cmake but after that will investigate if the patches that Homebrew is applying make any difference. You can see the formula with brew edit llvm@12 and these are the patches applied on mac os x:

  # Fix crash in clangd when built with GCC <6. Remove in LLVM 13
  # https://github.com/clangd/clangd/issues/800
  # https://github.com/Homebrew/homebrew-core/issues/84365
  patch do
    url "https://github.com/llvm/llvm-project/commit/ec1fb9533305e9bd69294ede7e5e7d9befbb2225.patch?full_index=1"
    sha256 "b80a5718420c789588f3392366ac15485e43bea8e81adb14424c3cad4afa7315"
  end

  # Fix parallel builds. Remove in LLVM 13.
  # https://reviews.llvm.org/D106305
  # https://lists.llvm.org/pipermail/llvm-dev/2021-July/151665.html
  patch do
    url "https://github.com/llvm/llvm-project/commit/b31080c596246bc26d2493cfd5e07f053cf9541c.patch?full_index=1"
    sha256 "b4576303404e68100dc396d2414d6740c5bfd0162979d22152a688d1e7307379"
  end

[mppf]

I couldn't reproduce it with my own build as similar as possible to Homebrew in cmake invocation and applying the same patches. Now I think it has something to do with log entries in the homebrew build along these lines:

clang++ called with: -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/tmp/llvmA12-20220310-93596-19c20g8/llvm-project-12.0.1.src/llvm/build/utils/TableGen -I/tmp/llvmA12-20220310-93596-19c20g8/llvm-project-12.0.1.src/llvm/utils/TableGen -I/tmp/llvmA12-20220310-93596-19c20g8/llvm-project-12.0.1.src/llvm/build/include -I/tmp/llvmA12-20220310-93596-19c20g8/llvm-project-12.0.1.src/llvm/include -stdlib=libc++ -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-class-memaccess -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -O2 -g -DNDEBUG -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -std=c++14 -MD -MT utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o -MF CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o.d -o CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o -c /tmp/llvmA12-20220310-93596-19c20g8/llvm-project-12.0.1.src/llvm/utils/TableGen/AsmMatcherEmitter.cpp
superenv removed:  -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wimplicit-fallthrough -Wcovered-switch-default -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -O2 -g -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk
superenv added:    -pipe -w -Os -march=nehalem -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk --sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -isystem/usr/local/include -isystem/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/readline/include -I/usr/local/opt/sqlite/include
superenv executed: clang++ -pipe -w -Os -march=nehalem -stdlib=libc++ -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/tmp/llvmA12-20220310-93596-19c20g8/llvm-project-12.0.1.src/llvm/build/utils/TableGen -I/tmp/llvmA12-20220310-93596-19c20g8/llvm-project-12.0.1.src/llvm/utils/TableGen -I/tmp/llvmA12-20220310-93596-19c20g8/llvm-project-12.0.1.src/llvm/build/include -I/tmp/llvmA12-20220310-93596-19c20g8/llvm-project-12.0.1.src/llvm/include -stdlib=libc++ -fPIC -fvisibility-inlines-hidden -Wno-unused-parameter -Wno-long-long -Wno-class-memaccess -Wno-noexcept-type -DNDEBUG -std=c++14 -MD -MT utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o -MF CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o.d -o CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o -c /tmp/llvmA12-20220310-93596-19c20g8/llvm-project-12.0.1.src/llvm/utils/TableGen/AsmMatcherEmitter.cpp -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk --sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -isystem/usr/local/include -isystem/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/readline/include -I/usr/local/opt/sqlite/include

Note that, superenv, whatever it is, is adding -isysroot --sysroot and -isystem arguments. My current theory is that these are affecting the behavior, somehow.

Here are the corresponding log lines from an llvm@11 build

clang++ called with: -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/tmp/llvmA11-20220311-63465-hmz8yy/llvm-project-11.1.0.src/llvm/build/utils/TableGen -I/tmp/llvmA11-20220311-63465-hmz8yy/llvm-project-11.1.0.src/llvm/utils/TableGen -I/tmp/llvmA11-20220311-63465-hmz8yy/llvm-project-11.1.0.src/llvm/build/include -I/tmp/llvmA11-20220311-63465-hmz8yy/llvm-project-11.1.0.src/llvm/include -stdlib=libc++ -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-class-memaccess -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -O2 -g -DNDEBUG -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -std=c++14 -MD -MT utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o -MF CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o.d -o CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o -c /tmp/llvmA11-20220311-63465-hmz8yy/llvm-project-11.1.0.src/llvm/utils/TableGen/AsmMatcherEmitter.cpp
superenv removed:  -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wimplicit-fallthrough -Wcovered-switch-default -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -O2 -g -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk
superenv added:    -pipe -w -Os -march=nehalem -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk --sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -isystem/usr/local/include -isystem/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/readline/include -I/usr/local/opt/sqlite/include
superenv executed: clang++ -pipe -w -Os -march=nehalem -stdlib=libc++ -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/tmp/llvmA11-20220311-63465-hmz8yy/llvm-project-11.1.0.src/llvm/build/utils/TableGen -I/tmp/llvmA11-20220311-63465-hmz8yy/llvm-project-11.1.0.src/llvm/utils/TableGen -I/tmp/llvmA11-20220311-63465-hmz8yy/llvm-project-11.1.0.src/llvm/build/include -I/tmp/llvmA11-20220311-63465-hmz8yy/llvm-project-11.1.0.src/llvm/include -stdlib=libc++ -fPIC -fvisibility-inlines-hidden -Wno-unused-parameter -Wno-long-long -Wno-class-memaccess -Wno-noexcept-type -DNDEBUG -std=c++14 -MD -MT utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o -MF CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o.d -o CMakeFiles/obj.llvm-tblgen.dir/AsmMatcherEmitter.cpp.o -c /tmp/llvmA11-20220311-63465-hmz8yy/llvm-project-11.1.0.src/llvm/utils/TableGen/AsmMatcherEmitter.cpp -isysroot/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk --sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -isystem/usr/local/include -isystem/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/readline/include -I/usr/local/opt/sqlite/include

superenv is implemented in as compiler wrappers in /usr/local/Homebrew/Library/Homebrew/shims/mac/super and I'm trying to edit the script to allow -g (so at least I can debug clang built this way). Also, if we put env :std in the formula (near the top, I guess?) it will disable superenv.

[mppf]

I now have a version showing the error and one that doesn't that I can debug. But, I haven't gotten very far.

In order to get to this point, I needed to get the Homebrew build to include -DCMAKE_BUILD_TYPE=RelWithDebInfo. (brew edit llvm@12 brew install --build-from-source llvm@12)

Another thing that tripped me up for a while is that lldb sets SDKROOT which changes the behavior of this code.

[mppf]

Here is a better stack trace for the failure error

``` * thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 2.564 frame #0: 0x00000001019a45c9 chpl`clang::HeaderSearch::getFileAndSuggestModule(llvm::StringRef, clang::SourceLocation, clang::DirectoryEntry const*, bool, clang::Module*, clang::ModuleMap::KnownHeader*) [inlined] clang::DiagnosticsEngine::Report(this=0x000000010706fa00, Loc=(ID = 12657), DiagID=6) at Diagnostic.h:1473:14 [opt] frame #1: 0x00000001019a45c9 chpl`clang::HeaderSearch::getFileAndSuggestModule(this=0x0000000107055a00, FileName=(Data = "./sys_basic.h", Length = 13), IncludeLoc=(ID = 12657), Dir=, IsSystemHeaderDir=, RequestingModule=0x0000000000000000, SuggestedModule=0x00007ff7bfefd350) at HeaderSearch.cpp:359:13 [opt] * frame #2: 0x00000001019a577d chpl`clang::HeaderSearch::LookupFile(this=, Filename=(Data = "sys_basic.h\"\n#include \"ctype.h\"\n#include \"wctype.h\"\n#include \"llvm/chapel_libc_wrapper.h\"\n", Length = 11), IncludeLoc=(ID = 12657), isAngled=false, FromDir=0x0000000000000000, CurDir=0x00007ff7bfefd288, Includers=ArrayRef > @ 0x00007ff7bfefcc68, SearchPath=0x00007ff7bfefd9e8, RelativePath=0x00007ff7bfefd5d0, RequestingModule=0x0000000000000000, SuggestedModule=0x00007ff7bfefd350, IsMapped=0x00007ff7bfefd35b, IsFrameworkFound=0x00007ff7bfefd35a, SkipCache=, BuildSystemModule=) at HeaderSearch.cpp:822:39 [opt] frame #3: 0x00000001019cdc23 chpl`clang::Preprocessor::LookupFile(this=0x0000000107056a18, FilenameLoc=(ID = 12657), Filename=(Data = "sys_basic.h\"\n#include \"ctype.h\"\n#include \"wctype.h\"\n#include \"llvm/chapel_libc_wrapper.h\"\n", Length = 11), isAngled=false, FromDir=0x0000000000000000, FromFile=0x0000000000000000, CurDir=0x00007ff7bfefd288, SearchPath=0x00007ff7bfefd9e8, RelativePath=0x00007ff7bfefd5d0, SuggestedModule=0x00007ff7bfefd350, IsMapped=0x00007ff7bfefd35b, IsFrameworkFound=0x00007ff7bfefd35a, SkipCache=) at PPDirectives.cpp:801:42 [opt] frame #4: 0x00000001019d3869 chpl`clang::Preprocessor::LookupHeaderIncludeOrImport(this=0x0000000107056a18, CurDir=0x00007ff7bfefd288, Filename=0x00007ff7bfefd2e8, FilenameLoc=(ID = 12657), FilenameRange=CharSourceRange @ 0x00007ff7bfefcf20, FilenameTok=0x00007ff7bfefdf60, IsFrameworkFound=0x00007ff7bfefd35a, IsImportDecl=, IsMapped=0x00007ff7bfefd35b, LookupFrom=0x0000000000000000, LookupFromFile=0x0000000000000000, LookupFilename=0x00007ff7bfefd2c0, RelativePath=0x00007ff7bfefd5d0, SearchPath=0x00007ff7bfefd9e8, SuggestedModule=0x00007ff7bfefd350, isAngled=) at PPDirectives.cpp:1733:33 [opt] frame #5: 0x00000001019d10c8 chpl`clang::Preprocessor::HandleHeaderIncludeOrImport(this=0x0000000107056a18, HashLoc=(ID = 12648), IncludeTok=, FilenameTok=0x00007ff7bfefdf60, EndLoc=(ID = 12670), LookupFrom=, LookupFromFile=0x0000000000000000) at PPDirectives.cpp:1930:33 [opt] frame #6: 0x00000001019ce6e7 chpl`clang::Preprocessor::HandleIncludeDirective(this=0x0000000107056a18, HashLoc=(ID = 12648), IncludeTok=0x0000000107820a10, LookupFrom=0x0000000000000000, LookupFromFile=) at PPDirectives.cpp:1701:17 [opt] frame #7: 0x00000001019ceaed chpl`clang::Preprocessor::HandleDirective(this=0x0000000107056a18, Result=0x0000000107820a10) at PPDirectives.cpp:1004:14 [opt] frame #8: 0x00000001019b092e chpl`clang::Lexer::LexTokenInternal(this=0x0000000105e30e00, Result=0x0000000107820a10, TokAtPhysicalStartOfLine=) at Lexer.cpp:3973:7 [opt] frame #9: 0x00000001019f1664 chpl`clang::Preprocessor::Lex(this=0x0000000107056a18, Result=0x0000000107820a10) at Preprocessor.cpp:894:33 [opt] frame #10: 0x000000010112d410 chpl`clang::ParseAST(S=0x000000010785d600, PrintStats=, SkipFunctionBodies=) at ParseAST.cpp:155:7 [opt] frame #11: 0x0000000100c88d99 chpl`clang::FrontendAction::Execute(this=0x0000600016701bc0) at FrontendAction.cpp:949:8 [opt] frame #12: 0x0000000100c41166 chpl`clang::CompilerInstance::ExecuteAction(this=0x0000000105c5f280, Act=0x0000600016701bc0) at CompilerInstance.cpp:949:33 [opt] frame #13: 0x00000001002e28ec chpl`runClang(just_parse_filename=0x0000000000000000) at clangUtil.cpp:2672:28 frame #14: 0x00000001001fb220 chpl`codegenPartTwo() at codegen.cpp:2527:5 frame #15: 0x00000001001fad99 chpl`codegen() at codegen.cpp:2744:3 frame #16: 0x0000000100369500 chpl`runPass(tracker=0x00007ff7bfeff5d0, passIndex=40, isChpldoc=false) at runpasses.cpp:208:3 frame #17: 0x000000010036937d chpl`runPasses(tracker=0x00007ff7bfeff5d0, isChpldoc=false) at runpasses.cpp:170:5 frame #18: 0x000000010035d00f chpl`main(argc=2, argv=0x00007ff7bfeff760) at driver.cpp:1843:3 frame #19: 0x00000001056cd4fe dyld`start + 462 ```

It looks like getFileAndSuggestModule is trying to open "./sys_basic.h" but then this fails and the error code is compared against llvm::errc::no_such_file_or_directory. This comparison succeeds on a non-homebrew build of clang but fails with the homebrew build. What is weird is that EC.message() is "No such file or directory" in both cases. This feels like a case where we have different headers vs libraries but I don't know if that is the cause.

[mppf]

I see this

otool -L /usr/local/opt/llvm@12/lib/libLLVM.dylib 
  ...
  /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)

otool -L bin/darwin-x86_64/chpl 
   ...
   /usr/local/opt/llvm@12/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)

This is bad because we statically link with the LLVM libraries, so that means that the various .a files are built with one libc++ headers and then the binary tries to call different functions in the libc++ lib. This is related to #18727 and #2702.

I think this is the cause of the problem but I'm not sure how to fix it. @daviditen I am timing out but maybe using these otool commands to check -- maybe you can figure out a way we can link chpl against the system libc++ instead of the homebrew installed one that comes in the LLVM package.

[mppf]

I have a start at something to fix it with this patch

diff --git a/util/chplenv/chpl_llvm.py b/util/chplenv/chpl_llvm.py
index debc8316cc..e27a77b1cf 100755
--- a/util/chplenv/chpl_llvm.py
+++ b/util/chplenv/chpl_llvm.py
@@ -557,6 +561,16 @@ def get_host_compile_args():
     llvm_config = get_llvm_config()

     if llvm_val == 'system':
+        # On Mac OS X with Homebrew, apply a workaround for issue #19217.
+        # This avoids finding headers in the libc++ installed by llvm@12 e.g.
+        host_platform = chpl_platform.get('host')
+        if host_platform == "darwin":
+            homebrew_prefix = chpl_platform.get_homebrew_prefix()
+            if homebrew_prefix:
+                system.append("-isysroot")
+                system.append("/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk")
+                system.append("-I" + "/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include")
+
         # Ubuntu 16.04 needed -fno-rtti for LLVM 3.7
         # tested on that system after installing
         #   llvm-3.7-dev llvm-3.7 clang-3.7 libclang-3.7-dev libedit-dev
@@ -618,6 +632,17 @@ def get_host_link_args():

     if llvm_val == 'system':
+        # On Mac OS X with Homebrew, apply a workaround for issue #19217.
+        # This avoids linking with the libc++ installed by llvm@12 e.g.
+        host_platform = chpl_platform.get('host')
+        if host_platform == "darwin":
+            homebrew_prefix = chpl_platform.get_homebrew_prefix()
+            if homebrew_prefix:
+              # -isysroot is just about includes
+              # and -Wl,-syslibroot seems to have no effect
+              system.append("-L" + "/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/lib")
+
+
         # Decide whether to try to link statically or dynamically.
         # Future work: consider using 'llvm-config --shared-mode'
         # to make this choice.

What I need to do next is to compute the Mac OS X SDKROOT that Homebrew clang wants to use, somehow.

[mppf]

I've created PR #19445 to address this problem

[mppf]

I have opened a Homebrew issue https://github.com/Homebrew/homebrew-core/issues/96915 to report this problem to Homebrew.

[mppf]

I've opened an LLVM project issue https://github.com/llvm/llvm-project/issues/54438 to report this problem there.