search

hedronvision / bazel-compile-commands-extractor

Goal: Enable awesome tooling for Bazel users of the C language family.
Other
689 stars 114 forks source link

Error occurs when used with envoy. #120

Closed j2gg0s closed 1 year ago

j2gg0s commented 1 year ago

As a newbie of cpp and bazel, i'm not sure if it's a bug.

  1. I use hedron_compile_commands with commit sha 80ac7efb33a65f0cc627635e5b958ea3eb03231d.

  2. clangd

    clangd --version
Apple clangd version 13.1.6 (clang-1316.0.21.2.5)
Features: mac+xpc

  3. When i access 'source/common/network/tcp_listener_impl.h', clangd return response

    ["<---",1,"clangd",{"response":{"method":"textDocument/publishDiagnostics","jsonrpc":"2.0","params":{"diagnostics":[{"source":"clang","relatedInformation":[],"range":{"end":{"character":31,"line":8},"start":{"character":9,"line":8}},"code":"pp_file_not_found","message":"'base_listener_impl.h' file not found","severity":1}

  4. related bazel

    envoy_cc_library(
name = "listener_lib",
srcs = [
    "base_listener_impl.cc",
    "tcp_listener_impl.cc",
    "udp_listener_impl.cc",
],
hdrs = [
    "base_listener_impl.h",
    "tcp_listener_impl.h",
    "udp_listener_impl.h",
],

  5. generated compile_commands.json

    {
"file": "source/common/network/tcp_listener_impl.h",
"arguments": [
"clang",
"-xc++",
"-D_FORTIFY_SOURCE=1",
"-fstack-protector",
"-fcolor-diagnostics",
"-Wall",
"-Wthread-safety",
"-Wself-assign",
"-fno-omit-frame-pointer",
"-O0",
"-DDEBUG",
"-std=c++11",
"-iquote",
".",
"-iquote",
"bazel-out/darwin_arm64-fastbuild/bin",
"-iquote",
"external/com_google_absl",
"-iquote",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_google_absl",
"-iquote",
"external/com_github_fmtlib_fmt",
"-iquote",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_github_fmtlib_fmt",
"-iquote",
"external/envoy_api",
"-iquote",
"bazel-out/darwin_arm64-fastbuild/bin/external/envoy_api",
"-iquote",
"external/com_google_googleapis",
"-iquote",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_google_googleapis",
"-iquote",
"external/com_google_protobuf",
"-iquote",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_google_protobuf",
"-iquote",
"external/com_envoyproxy_protoc_gen_validate",
"-iquote",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_envoyproxy_protoc_gen_validate",
"-iquote",
"external/com_googlesource_code_re2",
"-iquote",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_googlesource_code_re2",
"-iquote",
"external/com_github_cncf_udpa",
"-iquote",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_github_cncf_udpa",
"-iquote",
"external/opencensus_proto",
"-iquote",
"bazel-out/darwin_arm64-fastbuild/bin/external/opencensus_proto",
"-iquote",
"external/com_github_gabime_spdlog",
"-iquote",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_github_gabime_spdlog",
"-iquote",
"external/com_github_cyan4973_xxhash",
"-iquote",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_github_cyan4973_xxhash",
"-iquote",
"external/com_github_jbeder_yaml_cpp",
"-iquote",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_github_jbeder_yaml_cpp",
"-iquote",
"external/boringssl",
"-iquote",
"bazel-out/darwin_arm64-fastbuild/bin/external/boringssl",
"-iquote",
"external/envoy",
"-iquote",
"bazel-out/darwin_arm64-fastbuild/bin/external/envoy",
"-iquote",
"external/com_googlesource_googleurl",
"-iquote",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_googlesource_googleurl",
"-iquote",
"external/com_github_circonus_labs_libcircllhist",
"-iquote",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_github_circonus_labs_libcircllhist",
"-iquote",
"external/com_google_googletest",
"-iquote",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_google_googletest",
"-iquote",
"external/com_github_nlohmann_json",
"-iquote",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_github_nlohmann_json",
"-iquote",
"external/com_github_google_quiche",
"-iquote",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_github_google_quiche",
"-iquote",
"external/com_github_grpc_grpc",
"-iquote",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_github_grpc_grpc",
"-iquote",
"external/com_github_mirror_tclap",
"-iquote",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_github_mirror_tclap",
"-iquote",
"external/bazel_tools",
"-iquote",
"bazel-out/darwin_arm64-fastbuild/bin/external/bazel_tools",
"-Ibazel-out/darwin_arm64-fastbuild/bin/source/common/event/_virtual_includes/dispatcher_lib",
"-Ibazel-out/darwin_arm64-fastbuild/bin/external/com_google_protobuf/_virtual_includes/any_proto",
"-Ibazel-out/darwin_arm64-fastbuild/bin/external/com_google_protobuf/_virtual_includes/descriptor_proto",
"-Ibazel-out/darwin_arm64-fastbuild/bin/external/com_google_protobuf/_virtual_includes/duration_proto",
"-Ibazel-out/darwin_arm64-fastbuild/bin/external/com_google_protobuf/_virtual_includes/empty_proto",
"-Ibazel-out/darwin_arm64-fastbuild/bin/external/com_google_protobuf/_virtual_includes/struct_proto",
"-Ibazel-out/darwin_arm64-fastbuild/bin/external/com_google_protobuf/_virtual_includes/timestamp_proto",
"-Ibazel-out/darwin_arm64-fastbuild/bin/external/com_google_protobuf/_virtual_includes/wrappers_proto",
"-Ibazel-out/darwin_arm64-fastbuild/bin/source/common/common/_virtual_includes/logger_impl_lib_standard",
"-Ibazel-out/darwin_arm64-fastbuild/bin/source/common/common/_virtual_includes/thread_impl_lib_posix",
"-Ibazel-out/darwin_arm64-fastbuild/bin/source/common/event/_virtual_includes/signal_impl_lib_posix",
"-Ibazel-out/darwin_arm64-fastbuild/bin/source/common/api/_virtual_includes/os_sys_calls_lib",
"-Ibazel-out/darwin_arm64-fastbuild/bin/source/common/filesystem/_virtual_includes/watcher_lib",
"-Ibazel-out/darwin_arm64-fastbuild/bin/source/common/filesystem/_virtual_includes/filesystem_impl_lib_posix",
"-Ibazel-out/darwin_arm64-fastbuild/bin/source/common/filesystem/_virtual_includes/directory_iterator_impl_lib_posix",
"-Ibazel-out/darwin_arm64-fastbuild/bin/source/common/quic/platform/_virtual_includes/quiche_export_impl_lib",
"-Ibazel-out/darwin_arm64-fastbuild/bin/source/common/quic/platform/_virtual_includes/quiche_lower_case_string_impl_lib",
"-Ibazel-out/darwin_arm64-fastbuild/bin/source/common/quic/platform/_virtual_includes/quiche_logging_impl_lib",
"-Ibazel-out/darwin_arm64-fastbuild/bin/external/com_github_google_quiche/_virtual_includes/quiche_common_platform_default_quiche_platform_impl_header_policy_impl_lib",
"-Ibazel-out/darwin_arm64-fastbuild/bin/external/com_github_google_quiche/_virtual_includes/quiche_common_platform_default_quiche_platform_impl_command_line_flags_impl_lib",
"-Ibazel-out/darwin_arm64-fastbuild/bin/external/com_github_google_quiche/_virtual_includes/quiche_common_platform_default_quiche_platform_impl_flag_utils_impl_lib",
"-Ibazel-out/darwin_arm64-fastbuild/bin/external/com_github_google_quiche/_virtual_includes/quiche_common_platform_default_quiche_platform_impl_reference_counted_impl_lib",
"-Ibazel-out/darwin_arm64-fastbuild/bin/external/com_github_google_quiche/_virtual_includes/quiche_common_platform_default_quiche_platform_impl_testvalue_impl_lib",
"-Ibazel-out/darwin_arm64-fastbuild/bin/external/com_github_google_quiche/_virtual_includes/quiche_common_platform_default_quiche_platform_impl_thread_local_impl_lib",
"-Ibazel-out/darwin_arm64-fastbuild/bin/external/com_github_google_quiche/_virtual_includes/quiche_common_platform_default_quiche_platform_impl_time_utils_impl_lib",
"-Ibazel-out/darwin_arm64-fastbuild/bin/external/com_github_google_quiche/_virtual_includes/quiche_common_platform_default_quiche_platform_impl_prefetch_impl_lib",
"-Ibazel-out/darwin_arm64-fastbuild/bin/source/common/quic/platform/_virtual_includes/quic_base_impl_lib",
"-Ibazel-out/darwin_arm64-fastbuild/bin/source/common/quic/platform/_virtual_includes/quiche_flags_impl_lib",
"-Ibazel-out/darwin_arm64-fastbuild/bin/source/common/quic/platform/_virtual_includes/quiche_mem_slice_impl_lib",
"-Ibazel-out/darwin_arm64-fastbuild/bin/source/common/quic/platform/_virtual_includes/quiche_platform_iovec_impl_lib",
"-Ibazel-out/darwin_arm64-fastbuild/bin/external/com_github_google_quiche/_virtual_includes/quiche_common_platform_default_quiche_platform_impl_server_stats_impl_lib",
"-Ibazel-out/darwin_arm64-fastbuild/bin/source/common/quic/platform/_virtual_includes/quiche_stack_trace_impl_lib",
"-Ibazel-out/darwin_arm64-fastbuild/bin/external/com_github_google_quiche/_virtual_includes/quiche_common_platform_default_quiche_platform_impl_client_stats_impl_lib",
"-Ibazel-out/darwin_arm64-fastbuild/bin/external/com_github_google_quiche/_virtual_includes/quiche_common_platform_default_quiche_platform_impl_mutex_impl_lib",
"-Ibazel-out/darwin_arm64-fastbuild/bin/external/com_github_google_quiche/_virtual_includes/quiche_common_platform_default_quiche_platform_impl_containers_impl_lib",
"-isystem",
"external/com_github_fmtlib_fmt/include",
"-isystem",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_github_fmtlib_fmt/include",
"-isystem",
"external/com_google_protobuf/src",
"-isystem",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_google_protobuf/src",
"-isystem",
"bazel-out/darwin_arm64-fastbuild/bin/external/envoy/bazel/foreign_cc/zlib/include",
"-isystem",
"external/com_github_gabime_spdlog/include",
"-isystem",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_github_gabime_spdlog/include",
"-isystem",
"bazel-out/darwin_arm64-fastbuild/bin/external/envoy/bazel/foreign_cc/event/include",
"-isystem",
"external/com_github_jbeder_yaml_cpp/include",
"-isystem",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_github_jbeder_yaml_cpp/include",
"-isystem",
"external/boringssl/src/include",
"-isystem",
"bazel-out/darwin_arm64-fastbuild/bin/external/boringssl/src/include",
"-isystem",
"external/envoy/bazel/external/http_parser",
"-isystem",
"bazel-out/darwin_arm64-fastbuild/bin/external/envoy/bazel/external/http_parser",
"-isystem",
"bazel-out/darwin_arm64-fastbuild/bin/external/envoy/bazel/foreign_cc/nghttp2/include",
"-isystem",
"external/com_github_circonus_labs_libcircllhist/src",
"-isystem",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_github_circonus_labs_libcircllhist/src",
"-isystem",
"external/com_google_googletest/googlemock",
"-isystem",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_google_googletest/googlemock",
"-isystem",
"external/com_google_googletest/googlemock/include",
"-isystem",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_google_googletest/googlemock/include",
"-isystem",
"external/com_google_googletest/googletest",
"-isystem",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_google_googletest/googletest",
"-isystem",
"external/com_google_googletest/googletest/include",
"-isystem",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_google_googletest/googletest/include",
"-isystem",
"external/com_github_nlohmann_json/include",
"-isystem",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_github_nlohmann_json/include",
"-isystem",
"external/com_github_nlohmann_json/external/nlohmann_json_lib",
"-isystem",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_github_nlohmann_json/external/nlohmann_json_lib",
"-isystem",
"external/com_github_mirror_tclap/include",
"-isystem",
"bazel-out/darwin_arm64-fastbuild/bin/external/com_github_mirror_tclap/include",
"-MD",
"-MF",
"bazel-out/darwin_arm64-fastbuild/bin/test/common/network/_objs/listener_impl_test/listener_impl_test.d",
"-DFMT_HEADER_ONLY",
"-DSPDLOG_FMT_EXTERNAL",
"-DNGHTTP2_STATICLIB",
"-DGTEST_HAS_ABSL=1",
"-DBAZEL_CURRENT_REPOSITORY=\"\"",
"-frandom-seed=bazel-out/darwin_arm64-fastbuild/bin/test/common/network/_objs/listener_impl_test/listener_impl_test.o",
"-isysroot",
"/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk",
"-F/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/Library/Frameworks",
"-F/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks",
"-no-canonical-prefixes",
"-pthread",
"-DABSL_MIN_LOG_LEVEL=4",
"-std=c++17",
"-Wall",
"-Wextra",
"-Werror",
"-Wnon-virtual-dtor",
"-Woverloaded-virtual",
"-Wold-style-cast",
"-Wformat",
"-Wformat-security",
"-Wvla",
"-Wno-deprecated-declarations",
"-Wreturn-type",
"-fno-limit-debug-info",
"-Wgnu-conditional-omitted-operand",
"-Wc++2a-extensions",
"-Wrange-loop-analysis",
"-Wno-range-loop-analysis",
"-DABSL_MALLOC_HOOK_MMAP_DISABLE",
"-DENVOY_HANDLE_SIGNALS",
"-DENVOY_OBJECT_TRACE_ON_DUMP",
"-D__APPLE_USE_RFC_3542",
"-DENVOY_ADMIN_HTML",
"-DENVOY_ENABLE_QUIC",
"-DENVOY_GOOGLE_GRPC",
"-no-canonical-prefixes",
"-Wno-builtin-macro-redefined",
"-D__DATE__=\"redacted\"",
"-D__TIMESTAMP__=\"redacted\"",
"-D__TIME__=\"redacted\"",
"-target",
"arm64-apple-macosx12.3",
"-c",
"test/common/network/listener_impl_test.cc",
"-o",
"bazel-out/darwin_arm64-fastbuild/bin/test/common/network/_objs/listener_impl_test/listener_impl_test.o"
],
"directory": "/Users/j2gg0s/cpp/src/envoy"
}

  6. after i add extra include to compile_commands.json, everything is ok

       "-iquote",
   ".",
   "-iquote",
+      "source/common/network",
+      "-iquote",
   "bazel-out/darwin_arm64-fastbuild/bin",
   "-iquote",
   "external/com_google_absl",
cpsauer commented 1 year ago

Hey, @j2gg0s! Thanks for your interest in the tool and writing in. Great to meet you--and sorry to hear it's not just working out of the box.

It's not immediately obvious to me what's going on here, unfortunately, but here's what I'm thinking:

I think it's fairly unlikely that Bazel uses that iquote you added when compiling test/common/network/listener_impl_test.cc but that this tool drops it. Instead, assuming the test compiles cleanly, it seems more likely that this is a clangd issue, since clangd shouldn't show errors for commands that run cleanly. Maybe we'll need to work around that root issue.

I'm also a bit surprised that clang(d) needs that additional iquote to find other header, since that looks like a relative import; those two headers are in the same folder.

I then tried to reproduce this on my end, downloading envoy and making myself a compile_commands.json from your command (without the addition), with entries for both "source/common/network/tcp_listener_impl.h" and "test/common/network/listener_impl_test.cc". But I didn't see an include error like yours on #include "base_listener_impl.h". (I did see some other, unrelated errors since I hadn't run a build yet or used this tool, but I was able to CMD+click into "base_listener_impl.h", and clangd was happy with inheriting : public BaseListenerImpl, so I think it was finding it okay. This is all also on macOS, but with clangd 16.0.2 (the latest).)

Next steps:

Make sure you're seeing this issue with the latest clangd: If you were indeed using apple clangd (per your logs above), instead grab the latest clangd (per the instructions in this tool's README under VSCode--or manually if you're using another editor). Testing with the latest is the first thing the folks at clangd would ask you to do, just to make sure the issue hasn't already been fixed. (If you already set up the lastest clangd vscode, per the README, heads up that the clangd extension will use the clangd it downloaded but will not put the downloaded on your PATH for clangd --version. Instead, it puts it in ~/Library/Application Support/Code/User/globalStorage/llvm-vs-code-extensions.vscode-clangd/install/<clangd_version>/clangd_<clangd_version>/bin/clangd)

If you can see the spurious error with the latest version of clangd, it's also worth checking to see if the problem appears in test/common/network/listener_impl_test.cc, since that'd narrow down the issue a bit. And then, I think I should be asking you to also file this this over at clangd/clangd, so they take a look at it, too. https://github.com/clangd/clangd/issues

Let me know how it goes! Wish I could just snap my fingers and fix it, but I think this is the best I can do for now.

Cheers, Chris

P.S. If you end up wanting me to mess around with it more, anything special I should know about how you're invoking this tool? (Like among the numbered paths in the README?)

j2gg0s commented 1 year ago

@cpsauer bazel-compile-commands-extractor is simple but powerful, thanks for your work.

I don't thinks this is bug of clangd or extractor right now, its should be a problem with my build method.

I use vim8 on mac, with vim-lsp. clangd && clang's version is 13.1.6, installed through xcode. Build envoy by bazel build envoy, generate compile_command.json by bazel run @hedron_compile_commands//:refresh_all

With your answer, i upgrade my clangd and try some other build method. After some tries, the error disappeared

current:

I'm going to learn this question after i basically reading the code of envoy.

j2gg0s commented 1 year ago

Sadly, i don't know how to properly use targets.

I add these to WORKSPACE

load("@hedron_compile_commands//:refresh_compile_commands.bzl", "refresh_compile_commands")
refresh_compile_commands(
    name = "refresh_compile_commands",

    # Specify the targets of interest.
    # For example, specify a dict of targets and any flags required to build.
    targets = {
      "//:envoy": "",
    },
    # No need to add flags already in .bazelrc. They're automatically picked up.
    # If you don't need flags, a list of targets is also okay, as is a single target string.
    # Wildcard patterns, like //... for everything, *are* allowed here, just like a build.
      # As are additional targets (+) and subtractions (-), like in bazel query https://docs.bazel.build/versions/main/query.html#expressions
    # And if you're working on a header-only library, specify a test or binary target that compiles it.
)

and run with error

bazel run :refresh_compile_commands
ERROR: Traceback (most recent call last):
        File "/Users/j2gg0s/cpp/src/envoy/WORKSPACE", line 45, column 25, in <toplevel>
                refresh_compile_commands(
        File "/private/var/tmp/_bazel_j2gg0s/9da33bef089f9fb714acb1e2343ff6d2/external/hedron_compile_commands/refresh_compile_commands.bzl", line 82, column 26, in refresh_compile_commands
                _check_python_version(name = version_checker_script_name, to_run = name)
Error in _check_python_version: '_check_python_version' can only be called during the loading phase
ERROR: Error computing the main repository mapping: error loading package 'external': Package 'external' contains errors
cpsauer commented 1 year ago

Hey @j2gg0s! Just getting back from a trip and catching up.

^ That error is that you need to call refresh_compile_commands from a BUILD file, not WORKSPACE! As in the README, we'd recommend using (or creating) //BUILD.

Cheers, Chris