Problem when linking to clang and getting the Homebrew libc++

`brew gist-logs <formula>` link OR `brew config` AND `brew doctor` output

% brew gist-logs llvm@13  
Error: No logs.

% brew config
HOMEBREW_VERSION: 3.4.1-67-gb31d8e9
ORIGIN: https://github.com/Homebrew/brew
HEAD: b31d8e929b09562f431f926d1a158753a1df5877
Last commit: 18 hours ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 5731c186dbbe0c0804f7706767029d57a0813a7a
Core tap last commit: 50 minutes ago
Core tap branch: master
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_CORE_GIT_REMOTE: https://github.com/Homebrew/homebrew-core
HOMEBREW_MAKE_JOBS: 12
Homebrew Ruby: 2.6.8 => /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.8/bin/ruby
CPU: dodeca-core 64-bit kabylake
Clang: 13.1.6 build 1316
Git: 2.35.1 => /usr/local/bin/git
Curl: 7.77.0 => /usr/bin/curl
macOS: 12.2.1-x86_64
CLT: 13.2.0.0.1.1638488800
Xcode: 13.3

% brew doctor
Your system is ready to brew.

Verification

[X] I ran brew update and am still able to reproduce my issue.
[X] I have resolved all warnings from brew doctor and that did not fix my problem.

What were you trying to do (and why)?

I'm one of the developers for the upstream Chapel project. Chapel was working with llvm@11 but we were running into strange problems with llvm@12 and llvm@13 (see https://github.com/chapel-lang/chapel/issues/19217 ).

It took me a long time but I eventually figured out that the problem was due to mixing libc++ versions. The Chapel compiler, chpl was linking with the libc++ included in the Homebrew LLVM package; but the LLVM and Clang libraries were linking with a libc++ provided by the system. This will occur with any program that is linking with llvm or clang libraries using the usual commands to add a -L path for /usr/local/Cellar/llvm/13.0.1_1/lib, including using the output of /usr/local/opt/llvm@13/bin/llvm-config --ldflags.

I have included a small reproducer example in the step-by-step section below.

What happened (include all command output)?

With the test program provided below, I get this output

% ./a.out
EC.message is No such file or directory
EC == no_such_file_or_directory is 0

What did you expect to happen?

I expected this output from the test program provided below (without the workaround):

% ./a.out
EC.message is No such file or directory
EC == no_such_file_or_directory is 1

Step-by-step reproduction instructions (by running `brew` commands)

brew install llvm@13

Then, I have saved the following to pretend-in-llvm.cpp and within it are the commands to reproduce this problem.


/*

# first, imagine we are compiling something within LLVM
# this should use the same SDK that Homebrew used when compiling LLVM/clang
# other flags come from `/usr/local/opt/llvm@13/bin/llvm-config --cxxflags`
clang++ -c pretend-in-llvm.cpp -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -c -fno-rtti -I/usr/local/Cellar/llvm/13.0.1_1/include -std=c++14 -stdlib=libc++ -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/usr/local/include

# now, link an application with the above
# supposing it is in one of the LLVM/Clang libraries
# other flags come from `/usr/local/opt/llvm@13/bin/llvm-config --ldflags`
clang++ pretend-in-llvm.o -o a.out -lclangFrontend -lclangSerialization -lclangDriver -lclangCodeGen -lclangParse -lclangSema -lclangAnalysis -lclangEdit -lclangASTMatchers -lclangAST -lclangLex -lclangBasic -Wl,-rpath,/usr/local/Cellar/llvm/13.0.1_1/lib -L/usr/local/Cellar/llvm/13.0.1_1/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lLLVM-13 -L/usr/local/lib

./a.out
#  EC.message is No such file or directory
#  EC == no_such_file_or_directory is 0

# We can see the problem with otool -L :
otool -L ./a.out
#  ...
#  /usr/local/opt/llvm/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)

otool -L //usr/local/opt/llvm/lib/libLLVM.dylib
#  ...
#  /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)

# (in other words, a.out is using the libc++ installed by llvm@13
#  but LLVM itself is linked with a system libc++)

# Adjusting the link line like this works around the problem:
clang++ pretend-in-llvm.o -o a.out -L/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/lib -lclangFrontend -lclangSerialization -lclangDriver -lclangCodeGen -lclangParse -lclangSema -lclangAnalysis -lclangEdit -lclangASTMatchers -lclangAST -lclangLex -lclangBasic -Wl,-rpath,/usr/local/Cellar/llvm/13.0.1_1/lib -L/usr/local/Cellar/llvm/13.0.1_1/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lLLVM-13 -L/usr/local/lib

./a.out
#  EC.message is No such file or directory
#  EC == no_such_file_or_directory is 1

*/

#include "llvm/Support/Errc.h"
#include "clang/Basic/FileManager.h"

#include <iostream>

int main() {
  clang::FileManager fileMgr((clang::FileSystemOptions()));

  auto file = fileMgr.getFileRef("./nonexistant.h", /*OpenFile=*/true);

  std::error_code EC = llvm::errorToErrorCode(file.takeError());

  std::cout << "EC.message is " << EC.message() << "\n";

  bool not_no_such = (EC != llvm::errc::no_such_file_or_directory);
  bool is_no_such = !not_no_such;

  std::cout << "EC == no_such_file_or_directory is " << is_no_such << "\n";

  return 0;
}

@carlocab - that's great, thanks. I see that #106925 went in but I'm not understanding what happened to fix the problem. Could you summarize? Thanks. Also I am understanding that we should be able to remove the workaround now from Chapel, so please speak up if I'm misunderstanding.

Regarding the ncurses linkage, ncurses itself is not a direct dependency for Chapel. And, when I check a local build or the Homebrew chpl on an Intel Mac with otool -L (which has different format from what you show above), I'm not seeing it. I see that the Homebrew LLVM 14 libLLVM.dylib itself depends on ncurses, though. Maybe whatever produced the output you showed above is including the transitive dependency? (If that is the case, perhaps the ncurses dependency was dropped for M1 when LLVM 14 went from llvm to llvm@14 in #106925, for some reason).

I can confirm that the reproducer in this issue is no longer misbehaving for me with the updated LLVM 14 or 15 Homebrew packages.

``` laptop ~ % clang++ -c pretend-in-llvm.cpp -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -c -fno-rtti -I/usr/local/Cellar/llvm@14/14.0.6/include -std=c++14 -stdlib=libc++ -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/usr/local/include laptop ~ % clang++ pretend-in-llvm.o -o a.out -lclang-cpp -Wl,-rpath,/usr/local/Cellar/llvm@14/14.0.6/lib -L/usr/local/Cellar/llvm@14/14.0.6/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lLLVM-14 -L/usr/local/lib laptop ~ % ./a.out EC.message is No such file or directory EC == no_such_file_or_directory is 1 laptop ~ % clang++ -c pretend-in-llvm.cpp -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -c -fno-rtti -I/usr/local/Cellar/llvm/15.0.0/include -std=c++14 -stdlib=libc++ -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/usr/local/include laptop ~ % clang++ pretend-in-llvm.o -o a.out -lclang-cpp -Wl,-rpath,/usr/local/Cellar/llvm/15.0.0/lib -L/usr/local/Cellar/llvm/15.0.0/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lLLVM-15 -L/usr/local/lib laptop ~ % ./a.out EC.message is No such file or directory EC == no_such_file_or_directory is 1 ```

I see that #106925 went in but I'm not understanding what happened to fix the problem. Could you summarize?

That's this one: https://github.com/Homebrew/homebrew-core/commit/d3999fcadc162f4f55b3237d5636ef3e6355738a (plus a little extra here, since I am bad at atomic commits)

brew info llvm will also give you another hint.

Also I am understanding that we should be able to remove the workaround now from Chapel, so please speak up if I'm misunderstanding.

I think so; if by workaround you mean what you have in place to try to avoid linkage with both LLVM libc++ and Apple libc++.

Regarding the ncurses linkage, ncurses itself is not a direct dependency for Chapel. And, when I check a local build or the Homebrew chpl on an Intel Mac with otool -L (which has different format from what you show above), I'm not seeing it. I see that the Homebrew LLVM 14 libLLVM.dylib itself depends on ncurses, though. Maybe whatever produced the output you showed above is including the transitive dependency? (If that is the case, perhaps the ncurses dependency was dropped for M1 when LLVM 14 went from llvm to llvm@14 in #106925, for some reason).

Yea, maybe, but all our builds of libLLVM (14, 15, ARM, Intel) link with ncurses, so not really sure what's going on there.

See, for example, the output of brew linkage on Monterey ARM, and Monterey Intel for both LLVM 14 and 15.

I see that #106925 went in but I'm not understanding what happened to fix the problem. Could you summarize?

That's this one: d3999fc (plus a little extra here, since I am bad at atomic commits)

brew info llvm will also give you another hint.

OK, so the LLVM libc++ is being installed into a different directory from the LLVM install (e.g. /usr/local/opt/llvm/lib/c++ vs just /usr/local/opt/llvm/lib).

Homebrew / homebrew-core