search

llvm / llvm-project

The LLVM Project is a collection of modular and reusable compiler and toolchain technologies.
http://llvm.org
Other
28.48k stars 11.77k forks source link

errors when using `clang-repl` interpreter in project on mac #64237

Open samuelpmish opened 1 year ago

samuelpmish commented 1 year ago

I'm using clang-repl to embed a C++ interpreter in a library to enable JIT compilation, and I've been really happy with the results so far. However, when building this project on a mac, the JIT compiler is failing on even trivial programs. For example, creating the interpreter with

  std::vector<const char *> flags = {
    "-march=native", "-Xclang", "-emit-llvm-only", 
  };

  auto CI = llvm::cantFail(IncrementalCompilerBuilder::create(flags));
  interpreter = llvm::cantFail(Interpreter::create(std::move(CI)));

and using it to compile

#include <functional>

runs into errors 

In file included from <<< inputs >>>:1:
In file included from input_line_0:2:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/include/c++/v1/functional:510:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/include/c++/v1/__functional/bind.h:17:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/include/c++/v1/tuple:224:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/include/c++/v1/iosfwd:99:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/include/c++/v1/__mbstate_t.h:29:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/include/c++/v1/wchar.h:123:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/include/wchar.h:89:10: fatal error: 'stdarg.h' file not found
#include <stdarg.h>
         ^~~~~~~~~~

that aren't encountered when compiling with the same file with clang++ from the same LLVM installation. I've tried comparing the include search paths in each case by adding "-v" to the compilation flags and comparing the results.

clang-repl output when including "-v"

#include "..." search starts here:
#include <...> search starts here:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/include/c++/v1
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/include
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/System/Library/Frameworks (framework directory)

clang++ output when including "-v"

#include "..." search starts here:
#include <...> search starts here:
/opt/homebrew/opt/llvm/bin/../include/c++/v1
/opt/homebrew/Cellar/llvm/16.0.6/lib/clang/16/include
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/include
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/System/Library/Frameworks (framework directory)

The obvious difference was that the first include search paths were different, so I tried manually specifying those paths to the interpreter,

  std::vector<const char *> flags = {
    "-march=native", "-Xclang", "-emit-llvm-only", 
    "-isystem", "/opt/homebrew/Cellar/llvm/16.0.6/lib/clang/16/include",
    "-I", "/opt/homebrew/opt/llvm/bin/../include/c++/v1"
  };

and that does make the first two paths agree with the ones shown by clang++ -v ...:

#include "..." search starts here:
#include <...> search starts here:
 /opt/homebrew/opt/llvm/bin/../include/c++/v1
 /opt/homebrew/Cellar/llvm/16.0.6/lib/clang/16/include
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/include/c++/v1
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/include
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/System/Library/Frameworks (framework directory)
End of search list.

but the clang-repl interpreter still emits a long list of errors when trying to compile the #include <functional> directive:

In file included from /opt/homebrew/opt/llvm/bin/../include/c++/v1/functional:503:
In file included from /opt/homebrew/opt/llvm/bin/../include/c++/v1/__algorithm/search.h:14:
In file included from /opt/homebrew/opt/llvm/bin/../include/c++/v1/__algorithm/iterator_operations.h:15:
In file included from /opt/homebrew/opt/llvm/bin/../include/c++/v1/__iterator/advance.h:26:
In file included from /opt/homebrew/opt/llvm/bin/../include/c++/v1/cstdlib:87:
/opt/homebrew/opt/llvm/bin/../include/c++/v1/stdlib.h:150:34: error: unknown type name 'ldiv_t'
inline _LIBCPP_INLINE_VISIBILITY ldiv_t div(long __x, long __y) _NOEXCEPT {
                                 ^
/opt/homebrew/opt/llvm/bin/../include/c++/v1/stdlib.h:151:12: error: no member named 'ldiv' in the global namespace
  return ::ldiv(__x, __y);
         ~~^
/opt/homebrew/opt/llvm/bin/../include/c++/v1/stdlib.h:154:34: error: unknown type name 'lldiv_t'
inline _LIBCPP_INLINE_VISIBILITY lldiv_t div(long long __x,
                                 ^
/opt/homebrew/opt/llvm/bin/../include/c++/v1/stdlib.h:156:12: error: no member named 'lldiv' in the global namespace

There are a handful of similar issues in xeus-cling, like https://github.com/jupyter-xeus/xeus-cling/issues/403 https://github.com/jupyter-xeus/xeus-cling/issues/412

Does anyone know how to configure a clang-repl interpreter for use in projects outside of the LLVM source tree, on mac?

Is it possible to make the default clang-repl configuration mimic the implicit assumptions of clang++ from the same LLVM build?

llvm version 16.0.6 installed with homebrew m1 mac running 13.4

makslevental commented 1 year ago

@samuelpmish

You need to do add the right defines so that MacOSX12.3.sdk/usr/include/c++/v1/__config does the right thing; for example, for me, to get clang-repl> #include "mlir/IR/OperationSupport.h" to work correctly I did

  std::vector<const char *> Argv = {
      "-xc++",
      "-DMLIR_CUDA_CONVERSIONS_ENABLED=1",
      "-DMLIR_ROCM_CONVERSIONS_ENABLED=1",
      "-DNDEBUG",
      "-D_DEBUG",
      "-D_GLIBCXX_ASSERTIONS",
      "-D_LIBCPP_ENABLE_HARDENED_MODE",
      "-D__STDC_CONSTANT_MACROS",
      "-D__STDC_FORMAT_MACROS",
      "-D__STDC_LIMIT_MACROS",
      "-I/Users/mlevental/dev_projects/llvm-project/cmake-build-release/include",
      "-I/Users/mlevental/dev_projects/llvm-project/cmake-build-release/tools/mlir/include",
      "-I/Users/mlevental/dev_projects/llvm-project/cmake-build-release/tools/mlir/lib/Dialect/IRDL",
      "-I/Users/mlevental/dev_projects/llvm-project/llvm/include",
      "-I/Users/mlevental/dev_projects/llvm-project/mlir/include",
      "-I/Users/mlevental/dev_projects/llvm-project/mlir/lib/Dialect/IRDL",
      "-O3",
      "-UNDEBUG",
      "-Wall",
      "-Wc++98-compat-extra-semi",
      "-Wcast-qual",
      "-Wcovered-switch-default",
      "-Wctad-maybe-unsupported",
      "-Wdelete-non-virtual-dtor",
      "-Werror=date-time",
      "-Werror=mismatched-tags",
      "-Werror=unguarded-availability-new",
      "-Wextra",
      "-Wimplicit-fallthrough",
      "-Wmisleading-indentation",
      "-Wmissing-field-initializers",
      "-Wno-long-long",
      "-Wno-noexcept-type",
      "-Wno-unused-parameter",
      "-Wnon-virtual-dtor",
      "-Wstring-conversion",
      "-Wsuggest-override",
      "-Wwrite-strings",
      "-arch", "arm64",
      "-fPIC",
      "-fcolor-diagnostics",
      "-fdiagnostics-color",
      "-fno-exceptions",
      "-funwind-tables",
      "-fvisibility-inlines-hidden",
      "-isysroot", "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk",
      "-pedantic",
      "-std=c++17",
      "-v"};

I got this set of args by looking at my build.ninja for a successfully compiling lib so it can most likely be ablated. Note this is for tip commit.

but I also had to comment out void* operator new(__SIZE_TYPE__, void* __p) noexcept; in clang/lib/Interpreter/Interpreter.cpp. I plan to patch/file a bug for that.

samuelpmish commented 12 months ago

I'm hoping to include clang-repl in a library, so I'd need an automatic way to determine the right flags to set for a user's setup. The curious part is that when installing clang from homebrew, none of these extra flags are necessary. So, it seems clang already knows the right default settings-- is it possible to have clang-repl mimic that behaviour?

If not, then I guess I need to figure out some way to get CMake to emit all the flags passed to clang when compiling C++, and then generate a new sourcefile with that information so that it might be passed to clang-repl. This approach is a much less pleasant, though.

makslevental commented 12 months ago

If not, then I guess I need to figure out some way to get CMake to emit all the flags passed to clang when compiling C++, and then generate a new sourcefile with that information so that it might be passed to clang-repl. This approach is a much less pleasant, though.

you can look in the generated makefile or compile_commands.json or build.ninja

samuelpmish commented 12 months ago

you can look in the generated makefile or compile_commands.json or build.ninja

Yes-- I know where compile flags are located, but that isn't automated (which means the burden is on the user).

Would you want to use a library that required you to do something like:

  1. configure cmake
  2. manually read through some file
  3. copy bits of it of that file a new C++ sourcefile, and reformat it to be syntactically correct
  4. and add that new file to the build
  5. reconfigure cmake
  6. pass the flags from that new file to clang-repl

I know I wouldn't.

Also, it likely doesn't work if the user is compiling this library with a non-clang compiler, but still wants to use the clang-repl features.

makslevental commented 12 months ago

@samuelpmish I don't disagree with you in the least about the UX but

  1. I'm not a clang-repl code owner so I'm just trying to be helpful;
  2. You're expecting a lot from "cutting edge" technology.

My personal recommendation: if you want this feature, investigate it, put up a prototype, and ask for help. This will lower the "potential barrier" for someone else to jump in and collaborate in order to get it done.

Or you can wait for someone else to get to it as a function of their own priorities 🤷.

samuelpmish commented 12 months ago

Thank you for the suggestions! Sorry for being impatient in my earlier response.

complyue commented 3 weeks ago

I'd report that 18.1.8 seems to work now.

I started out to build the in-tree clang-repl on macOS 12.7, encountered similar issues (unknown type name 'lldiv_t' etc.). Then per consulting https://github.com/Homebrew/homebrew-core/blob/da26dd20d93fea974312a0177989178f0a28d211/Formula/l/llvm.rb#L107-L160 I ended up with -DDEFAULT_SYSROOT="/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/" added to cmake cmdl to get a working clang-repl built in-tree.

cyue@cymp llvm % llvm-build/bin/clang-repl --Xcc="-v"
clang version 18.1.8 (https://github.com/llvm/llvm-project.git 3b5b5c1ec4a3095ab096dd780e84d7ab81f3d7ff)
Target: x86_64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /ws/Wander/llvm/llvm-build/bin
 "/ws/Wander/llvm/llvm-build/bin/clang-repl" -cc1 -triple x86_64-apple-macosx12.0.0 -Wundef-prefix=TARGET_OS_ -Werror=undef-prefix -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -emit-obj -mrelax-all -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name "<<< inputs >>>" -mrelocation-model pic -pic-level 2 -mframe-pointer=all -ffp-contract=on -fno-rounding-math -funwind-tables=2 -fcompatibility-qualified-id-block-type-checking -fvisibility-inlines-hidden-static-local-var -fbuiltin-headers-in-system-modules -fdefine-target-os-macros -target-cpu penryn -tune-cpu generic -debugger-tuning=lldb -fdebug-compilation-dir=/ws/Wander/llvm -target-linker-version 820.1 -v -fcoverage-compilation-dir=/ws/Wander/llvm -resource-dir /ws/Wander/llvm/llvm-build/lib/clang/18 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/ -internal-isystem /ws/Wander/llvm/llvm-build/bin/../include/c++/v1 -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include -internal-isystem /ws/Wander/llvm/llvm-build/lib/clang/18/include -internal-externc-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -fdeprecated-macro -ferror-limit 19 -stack-protector 1 -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fskip-odr-check-in-gmf -fcxx-exceptions -fexceptions -fmax-type-align=16 -fcolor-diagnostics -fincremental-extensions -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o "<<< inputs >>>.o" -x c++ "<<< inputs >>>"
clang -cc1 version 18.1.8 based upon LLVM 18.1.8 default target x86_64-apple-darwin21.6.0
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include"
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk//Library/Frameworks"
#include "..." search starts here:
#include <...> search starts here:
 /ws/Wander/llvm/llvm-build/bin/../include/c++/v1
 /ws/Wander/llvm/llvm-build/lib/clang/18/include
 /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include
 /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk//System/Library/Frameworks (framework directory)
End of search list.
clang-repl> #include<functional>
clang-repl>

Without -DDEFAULT_SYSROOT="/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/", it'll use /usr/include instead of /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include, thus lacking macOS things (expected via #include_next) I'd guess.

I guess the OP's culprit is xcode's libcxx get in the way of

https://github.com/llvm/llvm-project/blob/3b5b5c1ec4a3095ab096dd780e84d7ab81f3d7ff/libcxx/include/math.h#L300-L302

by observing

...
#include <...> search starts here:
 /opt/homebrew/opt/llvm/bin/../include/c++/v1
 /opt/homebrew/Cellar/llvm/16.0.6/lib/clang/16/include
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk/usr/include/c++/v1
...

I suggest that llvm's in-tree version of libcxx should have compatibility guarantee for clang built from the same project repo, while AppleClang's libcxx may render your clang unusable unless you use clang lib built from roughly the same version branch. So maybe shipping libcxx with your app is a reasonable approach.

And later versions of llvm code base (18.1.8 at least) won't matter with xcode's libcxx, if configured properly, maybe -DLLVM_CREATE_XCODE_TOOLCHAIN=OFF is crucial? I didn't bisect it as that's too time costing.

Anyway, with release/18.x branch checkout, this script built clang as an internal project of llvm, with a working clang-repl:

# cf.
#  https://github.com/Homebrew/homebrew-core/blob/da26dd20d93fea974312a0177989178f0a28d211/Formula/l/llvm.rb#L107-L160
#
cmake -DDEFAULT_SYSROOT="/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/" \
    -DLLVM_LINK_LLVM_DYLIB=ON \
    -DLLVM_BUILD_EXTERNAL_COMPILER_RT=ON \
    -DLLVM_ENABLE_PROJECTS="clang" \
    -DLLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi;libunwind" \
    -DLLVM_ENABLE_LIBCXX=ON \
    -DLIBCXX_INSTALL_MODULES=ON \
    -DLLVM_ENABLE_EH=ON \
    -DLLVM_ENABLE_FFI=ON \
    -DLLVM_ENABLE_RTTI=ON \
    -DLLVM_INCLUDE_DOCS=OFF \
    -DLLVM_INCLUDE_TESTS=OFF \
    -DLLVM_INSTALL_UTILS=OFF \
    -DLLVM_OPTIMIZED_TABLEGEN=OFF \
    -DLLVM_CREATE_XCODE_TOOLCHAIN=OFF \
    -DCLANG_FORCE_MATCHING_LIBCLANG_SOVERSION=OFF \
    -DCOMPILER_RT_ENABLE_IOS=OFF \
    -DCOMPILER_RT_ENABLE_WATCHOS=OFF \
    -DCOMPILER_RT_ENABLE_TVOS=OFF \
    -DLLVM_TARGETS_TO_BUILD="Native" \
    -DCMAKE_BUILD_TYPE=Release -G Ninja \
    -S llvm-project/llvm -B llvm-build

cd llvm-build

ninja -j10