search

Homebrew / homebrew-core

🍻 Default formulae for the missing package manager for macOS (or Linux)
https://brew.sh
BSD 2-Clause "Simplified" License
13.57k stars 12.32k forks source link

Problem when linking to clang and getting the Homebrew libc++ #96915

Closed mppf closed 1 year ago

mppf commented 2 years ago

brew gist-logs <formula> link OR brew config AND brew doctor output

% brew gist-logs llvm@13  
Error: No logs.
% brew config
HOMEBREW_VERSION: 3.4.1-67-gb31d8e9
ORIGIN: https://github.com/Homebrew/brew
HEAD: b31d8e929b09562f431f926d1a158753a1df5877
Last commit: 18 hours ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 5731c186dbbe0c0804f7706767029d57a0813a7a
Core tap last commit: 50 minutes ago
Core tap branch: master
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_CORE_GIT_REMOTE: https://github.com/Homebrew/homebrew-core
HOMEBREW_MAKE_JOBS: 12
Homebrew Ruby: 2.6.8 => /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.8/bin/ruby
CPU: dodeca-core 64-bit kabylake
Clang: 13.1.6 build 1316
Git: 2.35.1 => /usr/local/bin/git
Curl: 7.77.0 => /usr/bin/curl
macOS: 12.2.1-x86_64
CLT: 13.2.0.0.1.1638488800
Xcode: 13.3
% brew doctor
Your system is ready to brew.

Verification

What were you trying to do (and why)?

I'm one of the developers for the upstream Chapel project. Chapel was working with llvm@11 but we were running into strange problems with llvm@12 and llvm@13 (see https://github.com/chapel-lang/chapel/issues/19217 ).

It took me a long time but I eventually figured out that the problem was due to mixing libc++ versions. The Chapel compiler, chpl was linking with the libc++ included in the Homebrew LLVM package; but the LLVM and Clang libraries were linking with a libc++ provided by the system. This will occur with any program that is linking with llvm or clang libraries using the usual commands to add a -L path for /usr/local/Cellar/llvm/13.0.1_1/lib, including using the output of /usr/local/opt/llvm@13/bin/llvm-config --ldflags.

I have included a small reproducer example in the step-by-step section below.

What happened (include all command output)?

With the test program provided below, I get this output

% ./a.out
EC.message is No such file or directory
EC == no_such_file_or_directory is 0

What did you expect to happen?

I expected this output from the test program provided below (without the workaround):

% ./a.out
EC.message is No such file or directory
EC == no_such_file_or_directory is 1

Step-by-step reproduction instructions (by running brew commands)

brew install llvm@13

Then, I have saved the following to pretend-in-llvm.cpp and within it are the commands to reproduce this problem.


/*

# first, imagine we are compiling something within LLVM
# this should use the same SDK that Homebrew used when compiling LLVM/clang
# other flags come from `/usr/local/opt/llvm@13/bin/llvm-config --cxxflags`
clang++ -c pretend-in-llvm.cpp -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -c -fno-rtti -I/usr/local/Cellar/llvm/13.0.1_1/include -std=c++14 -stdlib=libc++ -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/usr/local/include

# now, link an application with the above
# supposing it is in one of the LLVM/Clang libraries
# other flags come from `/usr/local/opt/llvm@13/bin/llvm-config --ldflags`
clang++ pretend-in-llvm.o -o a.out -lclangFrontend -lclangSerialization -lclangDriver -lclangCodeGen -lclangParse -lclangSema -lclangAnalysis -lclangEdit -lclangASTMatchers -lclangAST -lclangLex -lclangBasic -Wl,-rpath,/usr/local/Cellar/llvm/13.0.1_1/lib -L/usr/local/Cellar/llvm/13.0.1_1/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lLLVM-13 -L/usr/local/lib

./a.out
#  EC.message is No such file or directory
#  EC == no_such_file_or_directory is 0

# We can see the problem with otool -L :
otool -L ./a.out
#  ...
#  /usr/local/opt/llvm/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)

otool -L //usr/local/opt/llvm/lib/libLLVM.dylib
#  ...
#  /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)

# (in other words, a.out is using the libc++ installed by llvm@13
#  but LLVM itself is linked with a system libc++)

# Adjusting the link line like this works around the problem:
clang++ pretend-in-llvm.o -o a.out -L/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/lib -lclangFrontend -lclangSerialization -lclangDriver -lclangCodeGen -lclangParse -lclangSema -lclangAnalysis -lclangEdit -lclangASTMatchers -lclangAST -lclangLex -lclangBasic -Wl,-rpath,/usr/local/Cellar/llvm/13.0.1_1/lib -L/usr/local/Cellar/llvm/13.0.1_1/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lLLVM-13 -L/usr/local/lib

./a.out
#  EC.message is No such file or directory
#  EC == no_such_file_or_directory is 1

*/

#include "llvm/Support/Errc.h"
#include "clang/Basic/FileManager.h"

#include <iostream>

int main() {
  clang::FileManager fileMgr((clang::FileSystemOptions()));

  auto file = fileMgr.getFileRef("./nonexistant.h", /*OpenFile=*/true);

  std::error_code EC = llvm::errorToErrorCode(file.takeError());

  std::cout << "EC.message is " << EC.message() << "\n";

  bool not_no_such = (EC != llvm::errc::no_such_file_or_directory);
  bool is_no_such = !not_no_such;

  std::cout << "EC == no_such_file_or_directory is " << is_no_such << "\n";

  return 0;
}
carlocab commented 2 years ago

I'm not sure what to make of this problem, but I'm pretty sure your problem isn't due to mixed libc++ linkage, for (at least) four reasons:

  1. You have this linking behaviour with llvm@11 as well: libLLVM links with /usr/lib/libc++.1.dylib but linking with libLLVM will typically lead to linkage with a Homebrew-provided libc++ as well.
  2. Many other formulae link with both libLLVM and a Homebrew-provided libc++ with no issues.
  3. It is actually impossible not to link at least indirectly with /usr/lib/libc++.1.dylib. This is because every binary must link with /usr/lib/libSystem.B.dylib, and libSystem actually has a library dependency on /usr/lib/libc++.1.dylib. This means, in particular, that any Homebrew libc++ itself depends on /usr/lib/libc++.1.dylib.
  4. The two-level namespace on macOS often makes loading libraries of different versions in the same process unproblematic, unlike what you get when you try the same on Linux.

My guess is that you've run into a different LLVM bug.

mppf commented 2 years ago

You have this linking behaviour with llvm@11 as well: libLLVM links with /usr/lib/libc++.1.dylib but linking with libLLVM will typically lead to linkage with a Homebrew-provided libc++ as well.

Yes, my hypothesis here is that the Mac OS X libc++ happens to be compatible with the one from llvm@11 for the versions used here.

I'm not sure what to make of this problem, but I'm pretty sure your problem isn't due to mixed libc++ linkage, for (at least) four reasons:

Did you try the reproducer? I think the otool -L output is a pretty strong sign that the libc++ versions are different. The fact that the problem goes away with the -L path adjustment (and also makes the libc++ versions the same) seems pretty conclusive to me that, at the very least, mixing libc++ versions is part of the problem. Of course there might be another solution.

Many other formulae link with both libLLVM and a Homebrew-provided libc++ with no issues.

Yes and some things appeared to work in our project. I boiled down to a reproducer above a pattern that was not working. Edit -- it might also matter if static or dynamic linking with clang/llvm libraries is used.

My guess is that you've run into a different LLVM bug.

Note that the clang from llvm@13 is not involved in compiling my example at all.

carlocab commented 2 years ago

Yes, my hypothesis here is that the Mac OS X libc++ happens to be compatible with the one from llvm@11 for the versions used here.

If this hypothesis were true, then I would expect to see

  1. this behaviour to differ across different versions of macOS. Does it? Admittedly, Apple aren't terribly transparent about system libc++ versioning, but given that they update the rest of the toolchain on newer macOS versions, it would stand to reason that they update libc++ too.
  2. problems with other formulae that make use of LLVM 13. There are none that I'm aware of so far.

I think the otool -L output is a pretty strong sign that the libc++ versions are different.

Yes, I agree with you that the libc++ versions differ. However,

The fact that the problem goes away with the -L path adjustment (and also makes the libc++ versions the same) seems pretty conclusive to me that, at the very least, mixing libc++ versions is part of the problem.

I'm still not convinced.

Note that the clang from llvm@13 is not involved in compiling my example at all.

Sure, but I don't think this is relevant here. At the very least, this isn't relevant to any claims I've previously made.

mppf commented 2 years ago

I'm still not convinced.

Well, feel free to investigate further. I've already spent about a week on this and the workaround I show here is the best I could come up with.

Bo98 commented 2 years ago

Note that the clang from llvm@13 is not involved in compiling my example at all.

Does the behaviour change if the clang from llvm@13 is used?

mppf commented 2 years ago

Note that the clang from llvm@13 is not involved in compiling my example at all.

Does the behaviour change if the clang from llvm@13 is used?

Not for me, anyway

``` % /usr/local/Cellar/llvm/13.0.1_1/bin/clang++ -c pretend-in-llvm.cpp -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -c -fno-rtti -I/usr/local/Cellar/llvm/13.0.1_1/include -std=c++14 -stdlib=libc++ -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/usr/local/include % vim pretend-in-llvm.cpp % /usr/local/Cellar/llvm/13.0.1_1/bin/clang++ pretend-in-llvm.o -o a.out -lclangFrontend -lclangSerialization -lclangDriver -lclangCodeGen -lclangParse -lclangSema -lclangAnalysis -lclangEdit -lclangASTMatchers -lclangAST -lclangLex -lclangBasic -Wl,-rpath,/usr/local/Cellar/llvm/13.0.1_1/lib -L/usr/local/Cellar/llvm/13.0.1_1/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lLLVM-13 -L/usr/local/lib % ./a.out EC.message is No such file or directory EC == no_such_file_or_directory is 0 ```
carlocab commented 2 years ago

Do you mind reporting this upstream (i.e. LLVM)? I think this is an upstream bug.

Linking libLLVM with /usr/lib/libc++.1.dylib is very difficult to avoid. LLVM's build system requires that you build libLLVM (or, at least, its constituent static libraries) before you build libc++. Thus, any standard build of LLVM will have libLLVM linked against /usr/lib/libc++.1.dylib.

If your hypothesis is correct, then it is impossible to link against libLLVM and non-Apple libc++ simultaneously in a standard shared library build, and fixing it would require building LLVM at least twice: once to build a libc++ to link libLLVM against, and another to build the libLLVM you eventually install. (You would still load /usr/lib/libc++.1.dylib indirectly anyway, though, so this might not be a complete fix.)

mppf commented 2 years ago

@carlocab - the reason that I have posted it here and not upstream is that I was unable to reproduce the issue when not doing a Homebrew build. (I.e. if I build and install LLVM somewhere myself, manually, outside of homebrew, everything works OK). AFAIK that is due to the -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk arguments that the Homebrew build is using. (AFAIK, this argument comes both from something in the cmake from the Homebrew formula and from superenv).

Additionally I wanted to ask if it was clear to you why building with headers from one version of a library and then linking with a different version of the same library is a problem, in general. I could explain that more / find a reference to link here, if not. We could certainly ask upstream if they expect that to work for libc++ specifically - but I'm not so sure they can even answer the question of whether it is expected to work between Apple's libc++ and the open source one. I think we would have to involve somebody from Apple to answer that. Apple's libc++ could have any number of changes that break compatibility with the open source version.

Linking libLLVM with /usr/lib/libc++.1.dylib is very difficult to avoid. LLVM's build system requires that you build libLLVM (or, at least, its constituent static libraries) before you build libc++. Thus, any standard build of LLVM will have libLLVM linked against /usr/lib/libc++.1.dylib.

If your hypothesis is correct ... fixing it would require building LLVM at least twice: once to build a libc++ to link libLLVM against, and another to build the libLLVM you eventually install. (You would still load /usr/lib/libc++.1.dylib indirectly anyway, though, so this might not be a complete fix.)

Right, that is one way to fix it. Including that, I can think of several ways that Homebrew could fix it:

Relevant to the last option: Why does the LLVM formula install libc++ at all? Does it need to continue to do that?

Do you already know the answer? Why does homebrew install libc++ with LLVM by default?

mppf commented 2 years ago

I am almost certainly confused -DLLVM_ENABLE_LIBCXX vs -DLLVM_ENABLE_RUNTIMES including libcxx. Also https://github.com/Homebrew/linuxbrew-core/issues/13332 seems related.

carlocab commented 2 years ago

@carlocab - the reason that I have posted it here and not upstream is that I was unable to reproduce the issue when not doing a Homebrew build. (I.e. if I build and install LLVM somewhere myself, manually, outside of homebrew, everything works OK). AFAIK that is due to the -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk arguments that the Homebrew build is using. (AFAIK, this argument comes both from something in the cmake from the Homebrew formula and from superenv).

Which version of LLVM did you build? How did you configure it?

Additionally I wanted to ask if it was clear to you why building with headers from one version of a library and then linking with a different version of the same library is a problem, in general. I could explain that more / find a reference to link here, if not.

I'm aware that, in general, this is a recipe for ABI breakage. It just seems unlikely that this is what's happening here.

We know that this doesn't occur with LLVM 11 but does with LLVM 13, and you've suggested that Apple libc++ is ABI compatible with LLVM 11 libc++. This means that LLVM 13 has broken the libc++ ABI, and hasn't bumped the library version accordingly. This strikes me as extremely unlikely, even setting aside the fact that the lead libc++ developer works at Apple. Still, supposing this were the case: this would then really need to be reported upstream.

We also know that this problem does not go away with Homebrew LLVM 13 clang++. This is important because using Homebrew clang++ will make use of Homebrew libc++ headers when compiling your reproduction. (You can check by passing -H to your invocation of clang++.) Thus, the problem can be reproduced when using headers that match the linked library exactly.

We can also try the reverse exercise: use Homebrew LLVM 13 clang++ (and therefore LLVM 13 libc++ headers) but insist on linkage against Apple libc++. This configuration leads to the correct result:

❯ ./a.out
EC.message is No such file or directory
EC == no_such_file_or_directory is 1

If there was indeed ABI breakage in libc++ that is exposed by your example, then it's relatively likely that we should get the wrong result here. But that's not what we're seeing.

Right, that is one way to fix it. Including that, I can think of several ways that Homebrew could fix it:

  • as you said, it could build LLVM multiple times in order to create a clang that links with the built libc++. This solution makes the least sense to me of these options. However, I do not see why this would be a show-stopper. To me it looks like the formula already builds clang multiple times. https://github.com/Homebrew/homebrew-core/blob/14af399c3d901c51bdfc932ff594f116318c9033/Formula/llvm.rb#L178-L179
  • package libc++ separately and install it to a separate directory. (Ubuntu does not do this; while libc++-13-dev and llvm-13-dev are separate packages, they both install libraries to /usr/lib/llvm-13/lib. But the issue is not a big deal on Ubuntu since by default the compilers will use the gnu libstdc++). This would address the problem because people would have to opt in to linking with the non-system libc++, somehow.
  • just don't provide a package that installs a different libc++ from the system one. AFAIK this is the approach FreeBSD uses - and that is relevant because FreeBSD is another system that uses this libc++ as the system C++ library.

At this stage, all of these solutions are show-stoppers because they're not the right solutions. They are predicated on a diagnosis that doesn't seem to be correct.

Relevant to the last option: Why does the LLVM formula install libc++ at all?

A few reasons:

  1. Upstream ship it with their binary builds of LLVM.
  2. Users have asked for it.
  3. We aim to provide a feature-complete build of LLVM.

Those are just the one off the top of my head. There's likely more.

Does it need to continue to do that?

Yes, unless we have compelling reasons to do otherwise. I don't think we have any so far.

mppf commented 2 years ago

Which version of LLVM did you build? How did you configure it?

I built LLVM 12 with the Homebrew patches applied. I tried to make it as similar as possible to the Homebrew build but was having problems with the lldb build so disabled that. Here is the cmake command I used to configure it:

``` cmake ../llvm-project/llvm \ -DCMAKE_INSTALL_PREFIX=/usr/local/Cellar/llvm@12/12.0.1_1 \ -DCMAKE_INSTALL_LIBDIR=lib \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_FIND_FRAMEWORK=LAST \ -DCMAKE_VERBOSE_MAKEFILE=ON \ -Wno-dev \ -DBUILD_TESTING=OFF \ -DCMAKE_OSX_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk \ -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;lld;mlir;polly;openmp" \ -DLLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi;libunwind" \ -DLLVM_POLLY_LINK_INTO_TOOLS=ON \ -DLLVM_BUILD_EXTERNAL_COMPILER_RT=ON \ -DLLVM_LINK_LLVM_DYLIB=ON \ -DLLVM_ENABLE_EH=ON \ -DLLVM_ENABLE_FFI=ON \ -DLLVM_ENABLE_RTTI=ON \ -DLLVM_INCLUDE_DOCS=OFF \ -DLLVM_INCLUDE_TESTS=OFF \ -DLLVM_INSTALL_UTILS=ON \ -DLLVM_ENABLE_Z3_SOLVER=OFF \ -DLLVM_OPTIMIZED_TABLEGEN=ON \ -DLLVM_TARGETS_TO_BUILD=all \ -DLLDB_USE_SYSTEM_DEBUGSERVER=ON \ -DLLDB_ENABLE_PYTHON=ON \ -DLLDB_ENABLE_LUA=OFF \ -DLLDB_ENABLE_LZMA=ON \ -DLLDB_PYTHON_RELATIVE_PATH=libexec/python3.9/site-packages \ -DLIBOMP_INSTALL_ALIASES=OFF \ -DCLANG_PYTHON_BINDINGS_VERSIONS=3.9 \ -DLLVM_CREATE_XCODE_TOOLCHAIN=ON \ -DPACKAGE_VENDOR=Homebrew \ -DBUG_REPORT_URL=https://github.com/Homebrew/homebrew-core/issues \ -DCLANG_VENDOR_UTI=org.homebrew.clang \ -DFFI_INCLUDE_DIR=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include/ffi \ -DFFI_LIBRARY_DIR=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/lib \ -DLLVM_BUILD_LLVM_C_DYLIB=ON \ -DLLVM_ENABLE_LIBCXX=ON \ -DRUNTIMES_CMAKE_ARGS=-DCMAKE_INSTALL_RPATH=@loader_path/../lib \ -DDEFAULT_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk \ ```

We know that this doesn't occur with LLVM 11 but does with LLVM 13, and you've suggested that Apple libc++ is ABI compatible with LLVM 11 libc++.

I don't know that any version of Apple's libc++ is ABI compatible with any version of LLVM's libc++. It could be that there were minor problems that we never ran in to.

This means that LLVM 13 has broken the libc++ ABI, and hasn't bumped the library version accordingly. This strikes me as extremely unlikely, even setting aside the fact that the lead libc++ developer works at Apple. Still, supposing this were the case: this would then really need to be reported upstream.

Do you know how to query the library version? The one reported by otool -L I showed above is always 1 for the Homebrew-built libc++ but the Mac OS X libc++ has something that seems like a real version number (1200.3.0):

  /usr/local/opt/llvm/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
  /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)

Is this the same version number you are talking about?

We also know that this problem does not go away with Homebrew LLVM 13 clang++. This is important because using Homebrew clang++ will make use of Homebrew libc++ headers when compiling your reproduction. (You can check by passing -H to your invocation of clang++.) Thus, the problem can be reproduced when using headers that match the linked library exactly.

That is interesting. However I don't think it necessarily rules out header vs library version differences. In particular, the reproducer uses some of clang in a library (and links statically with it). That library is compiled as part of the Homebrew package and so it does not use the Homebrew libc++ today. So, in other words, to do that experiment properly, you would need a version of the Homebrew LLVM 13 clang libraries was compiled with the Homebrew LLVM 13 libc++ at include-time. I bet that will make the reproducer no longer show a problem. Is that an experiment you are able to try?

Efecan2 commented 2 years ago

Duplicate of #

bradcray commented 2 years ago

@Efecan2 :

Duplicate of #

Of what?

carlocab commented 2 years ago

I don't know that any version of Apple's libc++ is ABI compatible with any version of LLVM's libc++.

This seems to backpedal from your earlier claim that

Yes, my hypothesis here is that the Mac OS X libc++ happens to be compatible with the one from llvm@11 for the versions used here.

It could be that there were minor problems that we never ran in to.

It could, but this is still predicated on one of Apple or upstream LLVM breaking the ABI (both unlikely). In any case, any minor problems, if they exist, appear to be orthogonal to the issue here. They haven't been exposed previously, and are not exposed by your reproducer in LLVM 11.

Do you know how to query the library version? The one reported by otool -L I showed above is always 1 for the Homebrew-built libc++ but the Mac OS X libc++ has something that seems like a real version number (1200.3.0):

  /usr/local/opt/llvm/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
  /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)

Is this the same version number you are talking about?

What is important is the compatibility version, which is the same.

That is interesting. However I don't think it necessarily rules out header vs library version differences. In particular, the reproducer uses some of clang in a library (and links statically with it). That library is compiled as part of the Homebrew package and so it does not use the Homebrew libc++ today. So, in other words, to do that experiment properly, you would need a version of the Homebrew LLVM 13 clang libraries was compiled with the Homebrew LLVM 13 libc++ at include-time. I bet that will make the reproducer no longer show a problem. Is that an experiment you are able to try?

You would bet wrong.

❯ ./a.out
EC.message is No such file or directory
EC == no_such_file_or_directory is 0

This is built from LLVM 13.0.1 (llvm/llvm-project@75e33f71c2dae584b13a7d1186ae0a038ba98838). I invoked CMake using

Details

``` cmake -S llvm -B testbuild \ -DCMAKE_INSTALL_PREFIX=/tmp/llvm \ -DCMAKE_BUILD_TYPE=Release \ -DCMAKE_OSX_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk \ -DLLVM_ENABLE_PROJECTS=clang \ -DLLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi" \ -DLLVM_BUILD_EXTERNAL_COMPILER_RT=ON \ -DLLVM_LINK_LLVM_DYLIB=ON \ -DLLVM_ENABLE_EH=ON \ -DLLVM_ENABLE_FFI=ON \ -DLLVM_ENABLE_RTTI=ON \ -DLLVM_INCLUDE_DOCS=OFF \ -DLLVM_INCLUDE_TESTS=OFF \ -DLLVM_ENABLE_Z3_SOLVER=OFF \ -DLLVM_OPTIMIZED_TABLEGEN=ON \ -DLLVM_TARGETS_TO_BUILD=Native \ -DCLANG_PYTHON_BINDINGS_VERSIONS="3.10;3.9;3.8" \ -DFFI_INCLUDE_DIR=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include/ffi \ -DFFI_LIBRARY_DIR=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/lib \ -DLLVM_BUILD_LLVM_C_DYLIB=ON \ -DLLVM_ENABLE_LIBCXX=ON \ -DRUNTIMES_CMAKE_ARGS=-DCMAKE_INSTALL_RPATH="@loader_path/../lib;/opt/homebrew/opt/llvm/lib" \ -DDEFAULT_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk \ -DCMAKE_C_COMPILER=/opt/homebrew/opt/llvm/bin/clang \ -DCMAKE_CXX_COMPILER=/opt/homebrew/opt/llvm/bin/clang++ \ -DCMAKE_PREFIX_PATH=/opt/homebrew \ -DCMAKE_INSTALL_RPATH="@loader_path/../lib;/opt/homebrew/opt/llvm/lib" ```

The object file was compiled with

clang++ -c pretend-in-llvm.cpp -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -c -fno-rtti -I/tmp/llvm/include -std=c++14 -stdlib=libc++ -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS

and linked with

clang++ pretend-in-llvm.o -o a.out -lclangFrontend -lclangSerialization -lclangDriver -lclangCodeGen -lclangParse -lclangSema -lclangAnalysis -lclangEdit -lclangASTMatchers -lclangAST -lclangLex -lclangBasic -Wl,-rpath,/tmp/llvm/lib -L/tmp/llvm/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lLLVM -L/tmp/llvm/lib

We can check that this links with the libc++ that we want:

❯ otool -L ./a.out
./a.out:
    @rpath/libLLVM.dylib (compatibility version 1.0.0, current version 13.0.1)
    @rpath/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

❯ otool -l ./a.out | rg -A2 LC_RPATH
          cmd LC_RPATH
      cmdsize 32
         path /tmp/llvm/lib (offset 12)

It also loads our just-built libc++:

❯ DYLD_PRINT_LIBRARIES=1 ./a.out 1> /dev/null 2>&1 | rg 'libc[^A-Za-z]'
dyld[43551]: <355E121E-260B-3931-9825-E8033D109D3F> /private/tmp/llvm/lib/libc++.1.0.dylib
dyld[43551]: <B6B73FA9-4438-3B3D-A3B7-3B8CFA594B16> /opt/homebrew/Cellar/llvm/13.0.1_1/lib/libc++.1.0.dylib
dyld[43551]: <8F8963C1-403D-3282-AE5E-4F6E0E85F5A6> /usr/lib/libc++abi.dylib
dyld[43551]: <0EB3A986-B1BC-3F18-B7CC-51E2E5E00DC3> /usr/lib/libc++.1.dylib
dyld[43551]: <0D566764-7528-3C4B-9526-1C8DA0A9F59B> /opt/homebrew/Cellar/llvm/13.0.1_1/lib/libc++abi.1.0.dylib

This does still load Apple libc++, but I've already explained previously how there is no way to avoid this: every process you run loads Apple libc++. What is important is that it loads our just-built libc++ first.

We can also check that libLLVM has the right linkage:

❯ otool -L /tmp/llvm/lib/libLLVM.dylib
/tmp/llvm/lib/libLLVM.dylib:
    @rpath/libLLVM.dylib (compatibility version 1.0.0, current version 13.0.1)
    /usr/lib/libffi.dylib (compatibility version 1.0.0, current version 27.0.0)
    /usr/lib/libedit.3.dylib (compatibility version 2.0.0, current version 3.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
    /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
    /opt/homebrew/opt/ncurses/lib/libncursesw.6.dylib (compatibility version 6.0.0, current version 6.0.0)
    /usr/lib/libxml2.2.dylib (compatibility version 10.0.0, current version 10.9.0)
    /opt/homebrew/opt/llvm/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)

So, even if we compile libLLVM using Homebrew libc++ (headers included), we still get the wrong output with your reproducer.

To be sure, I did check that I could reproduce the same results as your initial example when using Homebrew LLVM (i.e. not one that I just built locally) and linking with Homebrew libc++ vs Apple libc++.

carlocab commented 2 years ago

To try to remove any doubt about mixing libc++ headers and libraries, I used my just-built clang to compile your example:

❯ /tmp/llvm/bin/clang++ -c pretend-in-llvm.cpp -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -c -fno-rtti -I/tmp/llvm/include -std=c++14 -stdlib=libc++ -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS

❯ /tmp/llvm/bin/clang++ pretend-in-llvm.o -o a.out -lclangFrontend -lclangSerialization -lclangDriver -lclangCodeGen -lclangParse -lclangSema -lclangAnalysis -lclangEdit -lclangASTMatchers -lclangAST -lclangLex -lclangBasic -Wl,-rpath,/tmp/llvm/lib -L/tmp/llvm/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lLLVM -L/tmp/llvm/lib

This should guarantee that I am using my just-installed libc++ headers and my just-built libc++.

Here's what I get:

❯ ./a.out
EC.message is No such file or directory
EC == no_such_file_or_directory is 0
mppf commented 2 years ago

OK, at this point you have more information than I do that would be useful to LLVM developers in an upstream bug report. In particular, the bug report to give them is ideally one that does not use Homebrew at all.

Wait, no, the cmake you did uses the Homebrew clang in order to get it to build with that libc++. So it is still not a demonstrator independent of Homebrew.

bradcray commented 2 years ago

Just noting that for those who are only following along by email notifications (as I have been) that Michael's latest comment has been edited on the GitHub issue to include an additional observation.

mppf commented 2 years ago

I've been able to create some scripts to reproduce that we can use as a bug report to LLVM. These scripts don't use Homebrew at all. (I haven't tried it on a system without Homebrew installed at all but I didn't have any llvm package installed).

runtest.sh ``` #!/bin/bash # assumes ../llvm-project is checked out # bug report uses 75e33f71c2dae584b13a7d1186ae0a038ba98838 llvmorg-13.0.1 # # will make ../llvm-build1 and ../llvm-build2 # will install to /tmp/llvm1 and /tmp/llvm2 ./cmake1.sh ./build1.sh ./cmake2.sh ./build2.sh ./repro.sh ``` cmake1.sh ``` #!/bin/bash mkdir -p ../llvm-build1 cd ../llvm-build1 cmake ../llvm-project/llvm \ -DCMAKE_INSTALL_PREFIX=/tmp/llvm1 \ -DCMAKE_BUILD_TYPE=RelWithDebInfo \ -DCMAKE_OSX_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk \ -DLLVM_ENABLE_PROJECTS=clang \ -DLLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi" \ -DLLVM_BUILD_EXTERNAL_COMPILER_RT=ON \ -DLLVM_LINK_LLVM_DYLIB=ON \ -DLLVM_ENABLE_EH=ON \ -DLLVM_ENABLE_FFI=ON \ -DLLVM_ENABLE_RTTI=ON \ -DLLVM_INCLUDE_DOCS=OFF \ -DLLVM_INCLUDE_TESTS=OFF \ -DLLVM_ENABLE_Z3_SOLVER=OFF \ -DLLVM_OPTIMIZED_TABLEGEN=ON \ -DLLVM_TARGETS_TO_BUILD=Native \ -DCLANG_PYTHON_BINDINGS_VERSIONS="3.10;3.9;3.8" \ -DFFI_INCLUDE_DIR=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include/ffi \ -DFFI_LIBRARY_DIR=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/lib \ -DLLVM_BUILD_LLVM_C_DYLIB=ON \ -DLLVM_ENABLE_LIBCXX=ON \ -DDEFAULT_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk \ ``` build1.sh ``` #!/bin/bash export CMAKE_BUILD_PARALLEL_LEVEL=5 cd ../llvm-build1 cmake --build . $CMAKE_PARBUILD --target install-llvm-headers cmake --build . $CMAKE_PARBUILD --target install-cmake-exports cmake --build . $CMAKE_PARBUILD --target LLVMSupport cmake --build . $CMAKE_PARBUILD --target install-LLVMSupport cmake --build . $CMAKE_PARBUILD --target llvm-config #cmake --build . --target install-llvm-config #cmake --build . --target clang #cmake --build . --target install-clang cmake --build . $CMAKE_PARBUILD --target install ``` cmake2.sh ``` #!/bin/bash mkdir -p ../llvm-build2 cd ../llvm-build2 # carlocab's reproducer cmake ../llvm-project/llvm \ -DCMAKE_INSTALL_PREFIX=/tmp/llvm2 \ -DCMAKE_BUILD_TYPE=RelWithDebInfo \ -DCMAKE_OSX_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk \ -DLLVM_ENABLE_PROJECTS=clang \ -DLLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi" \ -DLLVM_BUILD_EXTERNAL_COMPILER_RT=ON \ -DLLVM_LINK_LLVM_DYLIB=ON \ -DLLVM_ENABLE_EH=ON \ -DLLVM_ENABLE_FFI=ON \ -DLLVM_ENABLE_RTTI=ON \ -DLLVM_INCLUDE_DOCS=OFF \ -DLLVM_INCLUDE_TESTS=OFF \ -DLLVM_ENABLE_Z3_SOLVER=OFF \ -DLLVM_OPTIMIZED_TABLEGEN=ON \ -DLLVM_TARGETS_TO_BUILD=Native \ -DCLANG_PYTHON_BINDINGS_VERSIONS="3.10;3.9;3.8" \ -DFFI_INCLUDE_DIR=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/include/ffi \ -DFFI_LIBRARY_DIR=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/lib \ -DLLVM_BUILD_LLVM_C_DYLIB=ON \ -DLLVM_ENABLE_LIBCXX=ON \ -DRUNTIMES_CMAKE_ARGS=-DCMAKE_INSTALL_RPATH="@loader_path/../lib;/tmp/llvm1/lib" \ -DDEFAULT_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk \ -DCMAKE_C_COMPILER=/tmp/llvm1/bin/clang \ -DCMAKE_CXX_COMPILER=/tmp/llvm1/bin/clang++ \ -DCMAKE_PREFIX_PATH=/tmp/llvm1 \ -DCMAKE_INSTALL_RPATH="@loader_path/../lib;/tmp/llvm1/lib" exit 0 ``` build2.sh ``` #!/bin/bash export CMAKE_BUILD_PARALLEL_LEVEL=5 cd ../llvm-build2 cmake --build . $CMAKE_PARBUILD --target install-llvm-headers cmake --build . $CMAKE_PARBUILD --target install-cmake-exports cmake --build . $CMAKE_PARBUILD --target LLVMSupport cmake --build . $CMAKE_PARBUILD --target install-LLVMSupport cmake --build . $CMAKE_PARBUILD --target llvm-config #cmake --build . --target install-llvm-config #cmake --build . --target clang #cmake --build . --target install-clang cmake --build . $CMAKE_PARBUILD --target install ``` repro.sh ``` #!/bin/bash /tmp/llvm2/bin/clang++ -c repro.cpp -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -c -fno-rtti -I/tmp/llvm2/include -std=c++14 -stdlib=libc++ -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/usr/local/include /tmp/llvm2/bin/clang++ repro.o -o a.out -lclangFrontend -lclangSerialization -lclangDriver -lclangCodeGen -lclangParse -lclangSema -lclangAnalysis -lclangEdit -lclangASTMatchers -lclangAST -lclangLex -lclangBasic -Wl,-rpath,/tmp/llvm2/lib -L/tmp/llvm2/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lLLVM -L/usr/local/lib ./a.out # This adjustment to the link line allows the test to pass, # for some reason. #/tmp/llvm2/bin/clang++ repro.o -o a.out -L/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/lib -lclangFrontend -lclangSerialization -lclangDriver -lclangCodeGen -lclangParse -lclangSema -lclangAnalysis -lclangEdit -lclangASTMatchers -lclangAST -lclangLex -lclangBasic -Wl,-rpath,/tmp/llvm2/lib -L/tmp/llvm2/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lLLVM -L/usr/local/lib ``` repro.cpp ``` #include "llvm/Support/Errc.h" #include "clang/Basic/FileManager.h" #include int main() { clang::FileManager fileMgr((clang::FileSystemOptions())); auto file = fileMgr.getFileRef("./nonexistant.h", /*OpenFile=*/true); std::error_code EC = llvm::errorToErrorCode(file.takeError()); std::cout << "EC.message is " << EC.message() << "\n"; bool not_no_such = (EC != llvm::errc::no_such_file_or_directory); bool is_no_such = !not_no_such; std::cout << "EC == no_such_file_or_directory is " << is_no_such << "\n"; if (!is_no_such) { std::cout << "FAIL\n"; } return 0; } ```
mppf commented 2 years ago

I have created this llvm project issue: https://github.com/llvm/llvm-project/issues/54438

kencu commented 2 years ago

NB:

% otool -L a.out
a.out:
    /usr/local/opt/llvm/lib/libLLVM.dylib (compatibility version 1.0.0, current version 13.0.1)
    /usr/local/opt/llvm/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.100.3)

% ./a.out
EC.message is No such file or directory
EC == no_such_file_or_directory is 0

% install_name_tool -change /usr/lib/libc++.1.dylib  /usr/local/opt/llvm/lib/libc++.1.dylib /usr/local/opt/llvm/lib/libLLVM.dylib

% ./a.out
EC.message is No such file or directory
EC == no_such_file_or_directory is 1

% install_name_tool -change /usr/local/opt/llvm/lib/libc++.1.dylib /usr/lib/libc++.1.dylib /usr/local/opt/llvm/lib/libLLVM.dylib

% ./a.out
EC.message is No such file or directory
EC == no_such_file_or_directory is 0

Mixing libc++ versions is unsupported upstream.

carlocab commented 2 years ago

@kencu As I mentioned in the discussion above, I don't think this is a consequence of mixing libc++ versions.

You will get the erroneous behaviour if you build libc++ first and then link libLLVM against your just-built libc++.

kencu commented 2 years ago

If you see what I did with install_name_tool, you can make the error come and go at will just by by forcing mixing or not mixing of libc++ versions…

kencu commented 2 years ago

Of course, it’s possible some other factor could be involved too…toolchains are like that!

carlocab commented 2 years ago

Yes, I can see that.

However, that does not imply that the bug is caused by mixing libc++ versions. There being a bug in LLVM 13 libc++ is consistent with the behaviour from install_name_tool that you've demonstrated. Your example does not rule that out.

In particular:

You will get the erroneous behaviour if you build libc++ first and then link libLLVM against your just-built libc++.

This means that you also get the bug if you don't mix libc++ versions by linking only with LLVM 13 libc++. Unless you are saying that there is still mixed libc++ linkage here?

mppf commented 2 years ago

@carlocab - have you been able to review the discussion in https://groups.google.com/g/llvm-dev/c/6ZZ0FjiueDY/m/zVwhWzoOBQAJ as suggested in https://github.com/llvm/llvm-project/issues/54438#issuecomment-1087974558 ? From that discussion, it looks to me like it is not reasonable to expect the clang libraries to link with a custom-built libc++ on Mac OS X. Quoting from the discussion:

is it possible to ship a custom libc++ in a .app archive for older versions of macOS in order to use C++17?

Generally, I would say this is a tricky thing to do. As you mention, the problem is that if you link against any other library that does link against the system-provided libc++.dylib, you'll have ODR violations because some symbols will be found in both libraries. As a result, I believe you could also end up in a situation where you end up using globals from one library when the globals from the other library has been initialized. So basically, nothing good happens.

If you can ensure that your application doesn't pull in the system libc++.dylib (even transitively), then this would in theory be safe to do. Due to the brittleness of the solution, my immediate reaction would be to not recommend that.

So, to me, it sounds like supporting that kind of configuration requires a lot of effort (possibly, heroic effort) on the part of the application and/or library developers. Which I don't think the clang project would want to do.

If you have remaining doubts, would you be able to ask the LLVM developers for more clarification? (And one way to do so, if we need more people to see the issue, is to post about it to the LLVM Discourse). Thanks.

Bo98 commented 2 years ago

The solution towards this is a bit complicated.

A partial install of libc++ is needed for at least the headers, on systems using a SDK earlier than 11.3. Since LLVM 12, Clang first looks for C++ headers beside it in the toolchain and then if it doesn't exist it looks into the SDK - though we have backported this particular change all the way back to LLVM 8 however, since it's needed in order to actually be able to compile anything with newer SDKs. So for the macOS 11.3 SDK and later, we can omit libc++ headers entirely but since we technically support as far back as 10.10 so we will not do that.

But that's fine because newer headers and an older system library is a supported use case. It has to be because this is exactly what Xcode does half the time.

Linking to /usr/lib/libc++.dylib is already the default behaviour unless that has changed recently (there were actually upstream proposals to make it default to LLVM's libc++ but this hasn't happened yet). The problem comes when you are adding llvm-config --libs flags because unfortunately libc++.dylib is in the same directory as libLLVM and co.

You're right though that /usr/lib/libc++.dylib should always be preferred.

So what do we do about it? Well, to cover a few ideas, we could:

kencu commented 2 years ago

llvm upstream has been clear for years that they will not support mixing libc++ versions.

It works sometimes, and fails sometimes, randomly, like this example. I don't believe it is a bug. As the system libc++ and the lllvm libc++ diverge in age, it gets worse.

-stdlib=llvm-libc++ will not work without hacking clang's build. It's not too hard, we have done it to support using gcc11's libstdc++ with clang on macOS, but it's not trivial either.

Of course you should always use the current clang headers.

So ship libc++ if you want, it works sometimes, just tell people that is is likely fragile, they have to test, linking in the static libc++ might work better, they should not count on it working, etc.

Bo98 commented 2 years ago

llvm upstream has been clear for years that they will not support mixing libc++ versions.

At no point in my comment did I debate against that. I'm proposing solutions that make that mixing no longer happen.

kencu commented 2 years ago

At no point in my comment did I debate against that. I'm proposing solutions that make that mixing no longer happen.

Others said it was a bug.

It is very very very hard to avoid mixing libc++ versions if you are going to try to use a newer llvm libc++ , because everything in the system is linked against the system libc++.dylib.

Best of luck!

bradcray commented 2 years ago

@Bo98, @kencu, @carlocab: Thanks for all the thoughts and data here so far.

I wanted to give this issue a bump by asking some high-level questions:

Thanks

SMillerDev commented 2 years ago

Is there an obvious owner of this issue from the homebrew team going forward?

We don't have owners of issues. Whoever wants something fixed can fix it.

Is there a suggested best practice workaround for projects like ours that rely on the LLVM homebrew formula to compile and link things across multiple Mac OS X versions?

This won't be possible with Homebrew AFAIK since formula only ever build for a single OS.

As far as I can tell, our current best option would be to stop using homebrew LLVM and have our formula build and use the one that we bundle with Chapel, but that would have the obvious downsides of wasted build times, larger bottle sizes, and not following the spirit of homebrew.

Not to mention being against the requirements for formulae: https://docs.brew.sh/Acceptable-Formulae#stuff-that-requires-vendored-versions-of-homebrew-formulae

carlocab commented 2 years ago

Apologies for being MIA here -- my day job has been quite busy as of late.

I've yet to catch up with the newer posts here since I last commented (will do so soon), but here's a possible (but slightly hacky) solution that at least has the benefit of being less invasive, and should seem invisible to most users.

We can try changing the install name of

/usr/lib/libc++.1.dylib

in libLLVM.dylib to

@rpath/libc++.1.dylib

The idea is to make it so that when you link with LLVM libc++, so does libLLVM. Conversely, if you link with system libc++, you also have libLLVM using the system libc++.

This is actually inspired by @kencu's observation above. I'll need to think a bit more about whether this is a reasonable fix, but at the very least this will fix the reproduction case we have here.

kencu commented 2 years ago

I have a feeling that for any serious work using a newer libc++ on older sytems, you would have to copy what Google does and link your application to libc++.a statically. Then, at least, you just have to make certain you don't purposefully share any objects with the system frameworks (and I would not pretend I know how to ensure this, but presumably some of you do know how), and the chances of random error would then seem to be minimized. Perhaps I will see if I can rework our example here to do that.

As I understand it, Chrome sets "-nostdlibs" and then adds in the link to libc++.a manually, and everything "just works".

Nothing in Chrome seems to link to libc++.dylib, anyway.

% otool -L "Google Chrome"
Google Chrome:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

% otool -L "Google Chrome Framework"
Google Chrome Framework:
    @executable_path/../Frameworks/Google Chrome Framework.framework/Versions/100.0.4896.127/Google Chrome Framework (compatibility version 4896.0.127, current version 4896.0.127)
    /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate (compatibility version 1.0.0, current version 4.0.0)
    /System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/AVFoundation.framework/Versions/A/AVFoundation (compatibility version 1.0.0, current version 2.0.0)
    /System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
    /System/Library/Frameworks/ImageCaptureCore.framework/Versions/A/ImageCaptureCore (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore (compatibility version 1.2.0, current version 1.11.0)
    /System/Library/Frameworks/CoreVideo.framework/Versions/A/CoreVideo (compatibility version 1.2.0, current version 1.5.0)
    /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
    /System/Library/Frameworks/SecurityInterface.framework/Versions/A/SecurityInterface (compatibility version 1.0.0, current version 55159.0.0)
    /System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 60157.60.19)
    /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 56.0.0)
    /System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics (compatibility version 64.0.0, current version 1557.3.2)
    /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 1141.1.0)
    /System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork (compatibility version 1.0.0, current version 1327.0.4)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1856.105.0)
    /System/Library/Frameworks/CoreText.framework/Versions/A/CoreText (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 2113.20.111)
    /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1856.105.0)
    /System/Library/Frameworks/OpenDirectory.framework/Versions/A/OpenDirectory (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/ServiceManagement.framework/Versions/A/ServiceManagement (compatibility version 1.0.0, current version 2235.60.3)
    /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 1163.60.3)
    /System/Library/Frameworks/Metal.framework/Versions/A/Metal (compatibility version 1.0.0, current version 258.17.0, weak)
    /System/Library/Frameworks/IOSurface.framework/Versions/A/IOSurface (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/CoreMedia.framework/Versions/A/CoreMedia (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/Carbon.framework/Versions/A/Carbon (compatibility version 2.0.0, current version 165.0.0)
    /System/Library/Frameworks/Quartz.framework/Versions/A/Quartz (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 23.0.0)
    /System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox (compatibility version 1.0.0, current version 1000.0.0)
    /System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/VideoToolbox.framework/Versions/A/VideoToolbox (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/LocalAuthentication.framework/Versions/A/LocalAuthentication (compatibility version 1.0.0, current version 984.60.3)
    /System/Library/Frameworks/IOBluetooth.framework/Versions/A/IOBluetooth (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/CoreBluetooth.framework/Versions/A/CoreBluetooth (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/CoreMIDI.framework/Versions/A/CoreMIDI (compatibility version 1.0.0, current version 69.0.0)
    /System/Library/Frameworks/MediaAccessibility.framework/Versions/A/MediaAccessibility (compatibility version 1.0.0, current version 62.0.0)
    /System/Library/Frameworks/ForceFeedback.framework/Versions/A/ForceFeedback (compatibility version 1.0.0, current version 1.0.2)
    /System/Library/Frameworks/GameController.framework/Versions/A/GameController (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/CoreWLAN.framework/Versions/A/CoreWLAN (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/CoreLocation.framework/Versions/A/CoreLocation (compatibility version 1.0.0, current version 2665.0.9)
    /System/Library/Frameworks/AuthenticationServices.framework/Versions/A/AuthenticationServices (compatibility version 1.0.0, current version 612.3.6, weak)
    /System/Library/Frameworks/SafariServices.framework/Versions/A/SafariServices (compatibility version 535.0.0, current version 612.3.6, weak)
    /System/Library/Frameworks/UserNotifications.framework/Versions/A/UserNotifications (compatibility version 1.0.0, current version 1.0.0, weak)
    /System/Library/Frameworks/MetalKit.framework/Versions/A/MetalKit (compatibility version 1.0.0, current version 151.0.0, weak)
    /System/Library/Frameworks/MediaPlayer.framework/Versions/A/MediaPlayer (compatibility version 1.0.0, current version 1.0.0, weak)
    /System/Library/Frameworks/Vision.framework/Versions/A/Vision (compatibility version 1.0.0, current version 5.0.71, weak)
    /System/Library/Frameworks/ScreenTime.framework/Versions/A/ScreenTime (compatibility version 1.0.0, current version 500.6.1, weak)
    /usr/lib/libcups.2.dylib (compatibility version 2.0.0, current version 2.14.0)
    /usr/lib/libbsm.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libpmenergy.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libpmsample.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libsandbox.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libbz2.1.0.dylib (compatibility version 1.0.0, current version 1.0.8)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
github-actions[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

bradcray commented 2 years ago

If possible and appropriate, I'd like to see this remain open as it remains a concern for us, and it sounded as though @carlocab, @Bo98, and others had been having productive thoughts about it on #97618.

github-actions[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

bradcray commented 2 years ago

Thanks for keeping this alive, @Bo98!

carlocab commented 2 years ago

I am exploring solutions to this at #106925.

carlocab commented 2 years ago

I experimented with removing the libc++ and libc++abi builds from the llvm formula. Results are at https://github.com/Homebrew/homebrew-core/actions/runs/2897386665.

A few notes:

kencu commented 2 years ago

although those projects/runtimes (libcxx/libcxxabi) need to be enabled for the build, there are toggles to allow you to install the libraries or not:

https://github.com/llvm/llvm-project/blob/d9ff670330a80f2d597822bfd574842ad8fc55b3/libcxx/CMakeLists.txt#L157

https://github.com/llvm/llvm-project/blob/d9ff670330a80f2d597822bfd574842ad8fc55b3/libcxxabi/CMakeLists.txt#L80

you probably already knew this, of course.

carlocab commented 2 years ago

Yes, I vaguely recalled some LIBCXX_INSTALL_* flags, but I wasn't sure what the exact form was.

However: if we don't want to link with different versions of libc++, then we probably don't want to mix headers from one but use the library of the other either.

kencu commented 2 years ago

we at the unnamed always install and use the latest headers that match the compiler version, but (usually) don’t install libcxx/libcxxabi. It has worked generally well for us; you might find the same. MacOS is designed to deploy on older targets, so the newer headers have guards in them for the capabilities of the libcxx you are deploying to.

this install technique was originally set up by one of the Apple compiler engineers who previously was involved way back when — I just continued that along. At one point I installed a newer version of libcxx/libcxxabi as well that can be specifically optionally used during builds on older systems, but there are conflicts sometimes as you have found too…

best of luck! you’re doing great work here!

mppf commented 2 years ago

@carlocab -

  • I'm still suspicious that Chapel is the only one reporting issues with this.

I just wanted to point out some ways that the Chapel package might be unique:

bradcray commented 2 years ago

20+ other formulae link with LLVM's libc++ instead of the one in /usr/lib. I'm still suspicious that Chapel is the only one reporting issues with this.

I just wanted to point out some ways that the Chapel package might be unique...

@carlocab: Following up on Michael's hypothesis, could you share the list of 20+ formulae you're referring to above? I'm curious whether (through inspection or reasoning or inquiries) we could confirm or deny the guess.

carlocab commented 2 years ago

Following up on Michael's hypothesis, could you share the list of 20+ formulae you're referring to above? I'm curious whether (through inspection or reasoning or inquiries) we could confirm or deny the guess.

Here's a list:

A number of these also link with libclang-cpp, but I haven't checked them all to see if none link with libclang.

  • we use libclang and llvm instead of just llvm or just libclang
  • we use llvm-config to find llvm dependencies (vs cmake stuff)

I notice that brew linkage (or, equivalently, otool) does not report linkage with libclang. Does Chapel dlopen libclang? If so, that may explain the issues you're seeing.

Why would using llvm-config make a difference?

mppf commented 2 years ago

Another odd thing about Chapel is that we have some relatively strange workarounds for this issue...

I notice that brew linkage (or, equivalently, otool) does not report linkage with libclang. Does Chapel dlopen libclang? If so, that may explain the issues you're seeing.

Why would using llvm-config make a difference?

We are statically linking with clang and LLVM libraries on Mac OS X because of bugs in llvm-config on Mac OS X.

The static linking is a workaround. From a PR message talking about it ( https://github.com/chapel-lang/chapel/pull/19272 ) :

Note that we statically link both LLVM and clang on Mac OS X (since PR #18727). This PR leaves it this way. Historically, there have been problems with upstream LLVM and Homebrew in building a dynamic library that works with llvm-config (in particular the versioned file, like libLLVM-11.dylib, is missing). See this LLVM bug report. I expect that we will be able to allow dynamic linking on Mac OS X once we are able to establish that the issue is resolved.

It's my understanding that projects using cmake to find clang do not have this issue, but I could be wrong. So, perhaps statically linking with LLVM and clang libraries something strange that we are doing that other projects do not.

carlocab commented 2 years ago

Not sure if doing away with static linking will help, but you can definitely do that with llvm@1{2,3,4}:

❯ /usr/local/opt/llvm/bin/llvm-config --link-shared; echo $?
0
❯ /usr/local/opt/llvm/bin/llvm-config --libs
-lLLVM-14
❯ /usr/local/opt/llvm/bin/llvm-config --libfiles
/usr/local/Cellar/llvm/14.0.6_1/lib/libLLVM-14.dylib

In particular, you should no longer see the issue reported at llvm/llvm-project#39599 using a recent enough LLVM packaged with Homebrew.

mppf commented 2 years ago

I think a good next step would be to see if the reproducer in this issues original post still has the problem if it is changed to dynamically link.

mppf commented 2 years ago

I've reproduced the problem in the OP with static linking and the current LLVM 14 Homebrew package on my system:

$ clang++ -c pretend-in-llvm.cpp -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -c -fno-rtti -I/usr/local/Cellar/llvm/14.0.6_1/include -std=c++14 -stdlib=libc++   -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
$ clang++ pretend-in-llvm.o -o a.out -lclangFrontend -lclangSerialization -lclangDriver -lclangCodeGen -lclangParse -lclangSema -lclangAnalysis -lclangEdit -lclangASTMatchers -lclangAST -lclangLex -lclangBasic -Wl,-rpath,/usr/local/Cellar/llvm/14.0.6_1/lib -L/usr/local/Cellar/llvm/14.0.6_1/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lLLVM-14 -L/usr/local/lib
$ ./a.out
EC.message is No such file or directory
EC == no_such_file_or_directory is 0

(it is supposed to print out "is 1").

However, I'm still seeing the problem with dynamic linking:

$ clang++ -c pretend-in-llvm.cpp -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -c -fno-rtti -I/usr/local/Cellar/llvm/14.0.6_1/include -std=c++14 -stdlib=libc++   -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
$ clang++ pretend-in-llvm.o -o a.out -lclang-cpp -Wl,-rpath,/usr/local/Cellar/llvm/14.0.6_1/lib -L/usr/local/Cellar/llvm/14.0.6_1/lib -Wl,-search_paths_first -Wl,-headerpad_max_install_names -lLLVM-14 -L/usr/local/lib
$ ./a.out                               (git)-[remotes/upstream/main] 
EC.message is No such file or directory
EC == no_such_file_or_directory is 0

The workaround (adding -L/Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk/usr/lib early in the link command) still seems to work.

So, unfortunately, it does not seem that switching Chapel to dynamic linking will help with this problem. Nonetheless, I'd like to look at doing that anyway, to make it less unique in this regard, and to better match what we do on linux.

carlocab commented 1 year ago

Chapel should no longer be linking with LLVM libc++. From the latest version of Chapel distributed in Homebrew (1.28.0_1):

System libraries:
  /usr/lib/libSystem.B.dylib
  /usr/lib/libc++.1.dylib
  /usr/lib/libncurses.5.4.dylib
Homebrew libraries:
  /usr/local/opt/gmp/lib/libgmp.10.dylib (gmp)
  /usr/local/opt/llvm@14/lib/libLLVM.dylib (llvm@14)
  /usr/local/opt/llvm@14/lib/libclang-cpp.dylib (llvm@14)

Compare this with the bottle built at #110783:

System libraries:
  /usr/lib/libSystem.B.dylib
  /usr/lib/libc++.1.dylib
  /usr/lib/libncurses.5.4.dylib
Homebrew libraries:
  /usr/local/opt/gmp/lib/libgmp.10.dylib (gmp)
  /usr/local/opt/llvm/lib/libLLVM.dylib (llvm)
  /usr/local/opt/llvm/lib/libc++.1.dylib (llvm)
  /usr/local/opt/llvm/lib/libclang-cpp.dylib (llvm)

Incidentally, this may be what distinguishes Chapel from the other formulae that use LLVM libc++ referenced above: I don't think any of them have brew linkage reporting both of /usr/lib/libc++.1.dylib and /usr/local/opt/llvm/lib/libc++.1.dylib.

Incidentally, something weird I noticed, in the ARM bottles:

System libraries:
  /usr/lib/libSystem.B.dylib
  /usr/lib/libc++.1.dylib
Homebrew libraries:
  /opt/homebrew/opt/gmp/lib/libgmp.10.dylib (gmp)
  /opt/homebrew/opt/llvm/lib/libLLVM.dylib (llvm)
  /opt/homebrew/opt/llvm/lib/libc++.1.dylib (llvm)
  /opt/homebrew/opt/llvm/lib/libclang-cpp.dylib (llvm)

There is no linkage with ncurses, whereas the Intel bottles have it.

Not sure if this is an issue.