search

llvm / llvm-project

The LLVM Project is a collection of modular and reusable compiler and toolchain technologies.
http://llvm.org
Other
27.14k stars 11.11k forks source link

[Bazel][Clang Tidy] Unable to find standard library #63890

Open jathu opened 1 year ago

jathu commented 1 year ago

System

OS: macOS 13.4.1 Arch: M2 Pro LLVM version: 53e33807860f3b48b6f409b186a3c76de5cf1bf8 Bazel version: 6.2.1

Problem

I'm currently building clang-tidy using bazel. Using the binary results in (standard) file not found diagnostic error.

$ bazel build @llvm-project//clang-tools-extra/clang-tidy:clang-tidy
$ bazel-bin/external/llvm-project/clang-tools-extra/clang-tidy/clang-tidy --config-file .clang-tidy example.cpp -- -std=c++17

Error while trying to load a compilation database:
Could not auto-detect compilation database for file "example.cpp"
No compilation database found in /Users/someone/project or any parent directory
fixed-compilation-database: Error while opening fixed database: No such file or directory
json-compilation-database: Error while opening JSON database: No such file or directory
Running without flags.
1 error generated.
Error while processing /Users/someone/project/example.cpp.
/Users/someone/project/example.cpp:1:10: error: 'iostream' file not found [clang-diagnostic-error]
    1 | #include <iostream>
      |          ^~~~~~~~~~
Found compiler error(s).

Context

example.cpp file:

#include <iostream>

auto main(int /*argc*/, char** /*argv[]*/) -> int {
  std::cout << "hello world" << std::endl;
  return 0;
}

.clang-tidy config file:

Checks: >
  -*,
  bugprone-*,
  google-*,
  misc-*,
  modernize-*,
  performance-*,
  portability-*,
  readability-*

WORKSPACE file:

new_git_repository(
    name = "llvm-raw",
    build_file_content = "# empty",
    commit = "53e33807860f3b48b6f409b186a3c76de5cf1bf8",
    remote = "https://github.com/llvm/llvm-project.git",
)

load("@llvm-raw//utils/bazel:configure.bzl", "llvm_configure")

llvm_configure(name = "llvm-project")

.bazelrc file:

build --cxxopt=-std=c++17
build --host_cxxopt=-std=c++17
build --repo_env=CC=clang++

Xcode info:

$ xcode-select --print-path
/Library/Developer/CommandLineTools

$ xcrun --show-sdk-path
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk

Attempts

I tried explicitly including a sysroot, but that results in a different error:

$ bazel-bin/external/llvm-project/clang-tools-extra/clang-tidy/clang-tidy \
  --config-file .clang-tidy \
  example.cpp \
  -- \
  -std=c++17 \
  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk

34253 warnings and 1 error generated.
Error while processing /Users/someone/project/example.cpp.
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/wchar.h:89:10: error: 'stdarg.h' file not found [clang-diagnostic-error]
   89 | #include <stdarg.h>
      |          ^~~~~~~~~~
Suppressed 34253 warnings (34253 in non-user code).
Use -header-filter=.* to display errors from all non-system headers. Use -system-headers to display errors from system headers as well.
Found compiler error(s).

Tagging for visibility: @aaronmondal @gchatelet

llvmbot commented 1 year ago

@llvm/issue-subscribers-clang-tidy

jathu commented 1 year ago

I just realized these are missing builtin headers, so I built the headers:

$ bazel build @llvm-project//clang:builtin_headers_gen

Then included them, which results in errors in the system headers:

$ bazel-bin/external/llvm-project/clang-tools-extra/clang-tidy/clang-tidy \
  --config-file .clang-tidy \
  example.cpp \
  -- \
  -std=c++17 \
  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk \
  -isystem bazel-out/darwin_arm64-fastbuild/bin/external/llvm-project/clang/staging/include

34846 warnings and 20 errors generated.
Error while processing /Users/someone/project/example.cpp.
error: too many errors emitted, stopping now [clang-diagnostic-error]
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__functional/hash.h:646:34: error: template argument for template type parameter must be a type [clang-diagnostic-error]
  646 | struct _LIBCPP_TEMPLATE_VIS hash<nullptr_t>
      |                                  ^~~~~~~~~
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__functional/hash.h:638:17: note: template parameter is declared here
  638 | template <class _Tp>
      |                 ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__functional/hash.h:646:34: error: use of undeclared identifier 'nullptr_t'; did you mean 'nullptr'? [clang-diagnostic-error]
  646 | struct _LIBCPP_TEMPLATE_VIS hash<nullptr_t>
      |                                  ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__functional/hash.h:647:29: error: template argument for template type parameter must be a type [clang-diagnostic-error]
  647 |   : public __unary_function<nullptr_t, size_t>
      |                             ^~~~~~~~~
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__functional/unary_function.h:41:17: note: template parameter is declared here
   41 | template <class _Arg, class _Result>
      |                 ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__functional/hash.h:647:29: error: use of undeclared identifier 'nullptr_t'; did you mean 'nullptr'? [clang-diagnostic-error]
  647 |   : public __unary_function<nullptr_t, size_t>
      |                             ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__memory/unique_ptr.h:173:3: error: non-static data member cannot be constexpr; did you intend to make it const? [clang-diagnostic-error]
  173 |   _LIBCPP_CONSTEXPR unique_ptr(nullptr_t) _NOEXCEPT : __ptr_(__value_init_tag(), __value_init_tag()) {}
      |   ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__config:728:31: note: expanded from macro '_LIBCPP_CONSTEXPR'
  728 | #    define _LIBCPP_CONSTEXPR constexpr
      |                               ^

... pruned for brevity
aaronmondal commented 1 year ago

I'm not too familiar with MacOS, but a few things come to my mind:

rules_ll builds clang-tidy as part of the default toolchain, currently from 9feed59a9143cce23f26b4dd939078a9cbbd9002. So at least we know that this shouldn't affect Linux.

https://github.com/eomii/rules_ll/blob/468cd1af16c70dacc54acdd60bfee986b8a4c126/MODULE.bazel#L35

Seems like a good opportunity to bump LLVM there to a more recent commit, but for now I'd guess is that this is caused by some incompatibility with your host toolchain (i.e. the compiler that you get when running clang --version).

jathu commented 1 year ago

Try C++20 instead of 17. Do the error messages change?

Bumping it to C++20 results in the same error.

What compiler are you using for your host C++ toolchain to build clang-tidy? If this is older than Clang 16 that could be an issue.

$ clang++ --version
Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: arm64-apple-darwin22.5.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

It looks like you build against a local libcxx

Perhaps I should use the libcxx from llvm-project instead. I'll try it!

Seems like the issue might be the fact that I'm building using an older host clang, where as the source for clang-tidy is the latest version. I'll try building using the source libcxx and the commit you suggested.

Edit: I just realized the libcxx targets aren't exposed through the Bazel overlay.

aaronmondal commented 1 year ago

Building libcxx with a "standard" overlay would be a bad idea. You'd almost always want to build libcxx with clang from the same commit. I.e. you need an entire bootstrap process and custom toolchain to use libcxx in a sensible way. https://github.com/eomii/rules_ll/blob/37b5721a8083d36332eaca1f62fcc43b8aba7bd1/llvm-project-overlay/libcxx/BUILD.bazel#L277

Clang 14.0.3 is more than a year old. That's likely the reason this breaks. You probably don't need a custom libcxx, but simply a more recent clang. AFAIK on MacOS Homebrew usually has recent stable versions.

matt-sm commented 1 year ago

I stumbled across this thread because I had been facing a similar issue with a Bazel-built clang-tidy, but I'm on Ubuntu 20.04 using gcc 9.4.0.

I kept getting:

/usr/include/stdlib.h:31:10: error: 'stddef.h' file not found [clang-diagnostic-error]

And fixed by adding -- -I/path/to/bazel-bin/external/llvm-project/clang/staging/include

But should I have to do this?

When building clang-tidy in a traditional cmake way, all the clang headers are installed adjacent to the clang-tidy binary and it just works.

jathu commented 1 year ago

@aaronmondal I tried building clang-tidy with v16 and still get the same error.

$ bazel-bin/external/llvm-project/clang-tools-extra/clang-tidy/clang-tidy \
  --config-file .clang-tidy \
  example.cpp \
  -- \
  -std=c++17 \
  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk \
  -isystem bazel-out/darwin_arm64-fastbuild/bin/external/llvm-project/clang/staging/include

34846 warnings and 20 errors generated.
Error while processing /Users/someone/project/example.cpp.
error: too many errors emitted, stopping now [clang-diagnostic-error]
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__functional/hash.h:646:34: error: template argument for template type parameter must be a type [clang-diagnostic-error]
  646 | struct _LIBCPP_TEMPLATE_VIS hash<nullptr_t>
      |                                  ^~~~~~~~~
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__functional/hash.h:638:17: note: template parameter is declared here
  638 | template <class _Tp>
      |                 ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__functional/hash.h:646:34: error: use of undeclared identifier 'nullptr_t'; did you mean 'nullptr'? [clang-diagnostic-error]
  646 | struct _LIBCPP_TEMPLATE_VIS hash<nullptr_t>
      |                                  ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__functional/hash.h:647:29: error: template argument for template type parameter must be a type [clang-diagnostic-error]
  647 |   : public __unary_function<nullptr_t, size_t>
      |                             ^~~~~~~~~
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__functional/unary_function.h:41:17: note: template parameter is declared here
   41 | template <class _Arg, class _Result>
      |                 ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__functional/hash.h:647:29: error: use of undeclared identifier 'nullptr_t'; did you mean 'nullptr'? [clang-diagnostic-error]
  647 |   : public __unary_function<nullptr_t, size_t>
      |                             ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__memory/unique_ptr.h:173:3: error: non-static data member cannot be constexpr; did you intend to make it const? [clang-diagnostic-error]
  173 |   _LIBCPP_CONSTEXPR unique_ptr(nullptr_t) _NOEXCEPT : __ptr_(__value_init_tag(), __value_init_tag()) {}
      |   ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__config:728:31: note: expanded from macro '_LIBCPP_CONSTEXPR'
  728 | #    define _LIBCPP_CONSTEXPR constexpr
      |                               ^

... pruned for brevity

I'm actually starting to wonder if this is due to how we configured the clang-tidy binary Bazel target? If the version of the clang used to build clang-tidy is the issue, wouldn't that show up in the compilation step? Or is the issue simply the fact that I'm trying to use clang-tidy-17, built using clang-16, on a macOS library that perhaps only supports clang-14?

@matt-sm I think we might be able to fix this by perhaps providing the builtin headers as data files for the binary.

aaronmondal commented 1 year ago

Ahh sorry I'm just now seeing that you didn't post a build command but a run comand :sweat_smile:

The build step shouldn't invoke any config files and should look something like 

bazel build @llvm-project//clang-tools-extra/clang-tidy

The invocation you posted looks like an invocation of the actual executable. Then this isn't a build problem with the bazel-overlay at all and you'd probably get the exact same issue with a binary-installation of clang-tidy.

If the build invocation you're trying to run in clang-tidy here usually passes, then you can add something like -MJmyfile to the build invocation and it will write a JSON file called myfile that stores a clang-tidy-compatible compile command stub. If you want to use that file as compile_commands.json you can add square brackets around the JSON object and then pass it to clang tidy (the compile_commands.json needs to store an array of objects, not an object).

You could also manually copy-paste the invocation stored in myfile to the command line.

This way your clang-tidy will use the same build invocation that you'd usually use to build the file, including the correct sysroot and include paths.

matt-sm commented 1 year ago

In my case, both a cmake-built clang-tidy and a bazel-built clang-tidy have the same myfile (apart from different directory values) eg.

{ "directory": "/path/to/bazel-bin/external/llvm-project/clang-tools-extra/clang-tidy", "file": "placeholder.cpp", "output": "placeholder.o", "arguments": ["", "-xc++", "placeholder.cpp", "-o", "placeholder.o", "-c", "--target=x86_64-unknown-linux-gnu"]}

and the errors I'm seeing are purely to do with the fact the bazel-built clang tidy does not have the builtin includes in the default location.

jathu commented 1 year ago

@aaronmondal I think there might be some misunderstanding. The tool builds fine — the issue is with actually using the generated binary. Note the following:

The binary is generated and we can successfully run it:

$ bazel build @llvm-project//clang-tools-extra/clang-tidy:clang-tidy

$ bazel-bin/external/llvm-project/clang-tools-extra/clang-tidy/clang-tidy --version
LLVM (http://llvm.org/):
  LLVM version 17.0.0git
  DEBUG build with assertions.

When we invoke this binary on our example file, we get the following error:

$ bazel-bin/external/llvm-project/clang-tools-extra/clang-tidy/clang-tidy \
  --config-file .clang-tidy \
  example.cpp \
  -- \
  -std=c++20

/Users/someone/project/example.cpp:1:10: error: 'iostream' file not found [clang-diagnostic-error]
    1 | #include <iostream>
      |          ^~~~~~~~~~

This invocation seems to hint that it can't find libc++ headers, so we point to where it exists. On macOS there is a command to determine this

$ bazel-bin/external/llvm-project/clang-tools-extra/clang-tidy/clang-tidy \
  --config-file .clang-tidy \
  example.cpp \
  -- \
  -std=c++20 \
  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk

34253 warnings and 1 error generated.
Error while processing /Users/someone/project/example.cpp.
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/wchar.h:89:10: error: 'stdarg.h' file not found [clang-diagnostic-error]
   89 | #include <stdarg.h>
      |          ^~~~~~~~~~

This now throws an error that it can't find the built-in headers included with clang-tidy. The tool usually looks for this in <clang-tidy-path>/../include/c++/v1. This is the same error that @matt-sm was running into. We can work around this by building the headers and manually including them in the path:

$ bazel build @llvm-project//clang:builtin_headers_gen

$ bazel-bin/external/llvm-project/clang-tools-extra/clang-tidy/clang-tidy \
  --config-file .clang-tidy \
  example.cpp \
  -- \
  -std=c++20 \
  -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk \
  -isystem bazel-out/darwin_arm64-fastbuild/bin/external/llvm-project/clang/staging/include

34846 warnings and 20 errors generated.
Error while processing /Users/someone/project/example.cpp.
error: too many errors emitted, stopping now [clang-diagnostic-error]
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__functional/hash.h:646:34: error: template argument for template type parameter must be a type [clang-diagnostic-error]
  646 | struct _LIBCPP_TEMPLATE_VIS hash<nullptr_t>
      |                                  ^~~~~~~~~
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__functional/hash.h:638:17: note: template parameter is declared here
  638 | template <class _Tp>
      |                 ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__functional/hash.h:646:34: error: use of undeclared identifier 'nullptr_t'; did you mean 'nullptr'? [clang-diagnostic-error]
  646 | struct _LIBCPP_TEMPLATE_VIS hash<nullptr_t>
      |                                  ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__functional/hash.h:647:29: error: template argument for template type parameter must be a type [clang-diagnostic-error]
  647 |   : public __unary_function<nullptr_t, size_t>
      |                             ^~~~~~~~~
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__functional/unary_function.h:41:17: note: template parameter is declared here
   41 | template <class _Arg, class _Result>
      |                 ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__functional/hash.h:647:29: error: use of undeclared identifier 'nullptr_t'; did you mean 'nullptr'? [clang-diagnostic-error]
  647 |   : public __unary_function<nullptr_t, size_t>
      |                             ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__memory/unique_ptr.h:173:3: error: non-static data member cannot be constexpr; did you intend to make it const? [clang-diagnostic-error]
  173 |   _LIBCPP_CONSTEXPR unique_ptr(nullptr_t) _NOEXCEPT : __ptr_(__value_init_tag(), __value_init_tag()) {}
      |   ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__config:728:31: note: expanded from macro '_LIBCPP_CONSTEXPR'
  728 | #    define _LIBCPP_CONSTEXPR constexpr
      |                               ^

... pruned for brevity

As you can see we now see errors in the system headers. I wasn't able to make progress past this.

Regarding your point about compile commands, we shouldn't need to generate and use them. The binary should work exactly how a pre-built binary should work. As it is right now, the binary target in this repo is not usable.

keith commented 1 year ago

On the original issue it seems like the issue is that the -resource-dir being passed by the bazel built clang-tidy doesn't exist. I discovered this by passing -v and diffing the invocation against a cmake built binary that ran successfully. On my machine the path was -resource-dir /private/var/tmp/_bazel_ksmiley/1f5b368478bebbb21f947ab4e9583c39/execroot/__main__/bazel-out/darwin_arm64-fastbuild/bin/external/llvm-project/clang-tools-extra/lib/clang/17 and it doesn't exist at all (seems like clang should error in that case?). Manually passing a valid one works, seems like we should debug why that directory doesn't exist if clang-tidy depends on it

jathu commented 12 months ago

@keith your suggestion was correct — it kept failing to find the builtin headers, even when we included the ad-hoc system path. I made a diff to include the headers in the binary https://reviews.llvm.org/D158942

rupprecht commented 11 months ago

I'm not able to reproduce that last failure above, but I don't have that -isysroot path to reproduce w/ so that's likely the important bit here.

$ bazel build --config=ci -k \
  @llvm-project//clang-tools-extra/clang-tidy:clang-tidy \
  @llvm-project//clang:builtin_headers_gen
$ bazel-bin/external/llvm-project/clang-tools-extra/clang-tidy/clang-tidy \
  --config-file /tmp/.clang-tidy /tmp/sample.cpp -- \
  -std=c++20 -isystem bazel-bin/external/llvm-project/clang/staging/include

22244 warnings generated.
/tmp/sample.cpp:4:33: warning: do not use 'std::endl' with streams; use '\n' instead [performance-avoid-endl]
    4 |   std::cout << "hello world" << std::endl;
      |                                 ^~~~~~~~~
      |                                 '\n'
Suppressed 22243 warnings (22243 in non-user code).
Use -header-filter=.* to display errors from all non-system headers. Use -system-headers to display errors from system headers as well.
rupprecht commented 11 months ago

Since you're setting sysroot, I think that changes how isystem is interpreted. Maybe -resource-dir will work better for you instead:

$ bazel-bin/external/llvm-project/clang-tools-extra/clang-tidy/clang-tidy \
  --config-file /tmp/.clang-tidy /tmp/sample.cpp -- \
  -std=c++20 -resource-dir=bazel-bin/external/llvm-project/clang/staging

(note to drop "include" when specifying as -resource-dir)

jathu commented 10 months ago

@rupprecht I've tried the following attempts with similar errors:

$ bazel-bin/external/llvm-project/clang-tools-extra/clang-tidy/clang-tidy --config-file .clang-tidy example.cpp -- -v -std=c++20 -isystem bazel-bin/external/llvm-project/clang/staging/include

/tmp/example.cpp:1:10: error: 'iostream' file not found [clang-diagnostic-error]
    1 | #include <iostream>
      |          ^~~~~~~~~~
Found compiler error(s).
$ bazel-bin/external/llvm-project/clang-tools-extra/clang-tidy/clang-tidy --config-file .clang-tidy example.cpp -- -v -std=c++20 -resource-dir=bazel-bin/external/llvm-project/clang/staging

/tmp/example.cpp:1:10: error: 'iostream' file not found [clang-diagnostic-error]
    1 | #include <iostream>
      |          ^~~~~~~~~~
Found compiler error(s).
$ bazel-bin/external/llvm-project/clang-tools-extra/clang-tidy/clang-tidy --config-file .clang-tidy example.cpp -- -std=c++20 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -resource-dir bazel-out/darwin_arm64-fastbuild/bin/external/llvm-project/clang/staging

Assertion failed: (LHS.isValid() && RHS.isValid() && "Passed invalid source location!"), function isBeforeInTranslationUnit, file SourceManager.cpp, line 2026.
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.  Program arguments: bazel-bin/external/llvm-project/clang-tools-extra/clang-tidy/clang-tidy --config-file .clang-tidy example.cpp -- -v -std=c++20 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -resource-dir bazel-out/darwin_arm64-fastbuild/bin/external/llvm-project/clang/staging
1.  /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/pthread/pthread_impl.h:63:35: current parser token ')'
[1]    27534 abort      bazel-bin/external/llvm-project/clang-tools-extra/clang-tidy/clang-tidy    --