Same compile command repeated many times.

[SoftwareApe]

Version used: abdd06e05c7949721dba4bf1ae465bde16b9d3e1

First of all, thank you for providing such an easy to use tool, which also provides nice debugging output that tells you which generated files are missing.

However I noticed the compile_commands.json produced is huge for my project.

Looking at the compile_commands.json there seems to be an issue.

1. I can look at the arguments and folder and find the same compile_command repeated 400 times. The only thing that's different on every entry is the "file" part of the compile command.
2. The "file" is some header, that's maybe part of the translation unit, but not the source file in question.

[cpsauer]

Hey, Carl! Thanks for trying the tool, for writing in, and for being kind.

We're outputting one command per header because clangd needs them to robustly avoid a variety of error cases, especially some around directly opening header-only libraries. (More details in https://github.com/clangd/clangd/issues/123 if you're really curious. Some discussion also in this section of the README.)

If you don't need or want the entries for the headers, just pass in exclude_headers = "all", as per this section.

Let me know if that resolves things for you--or if there are more things I should know!

Cheers, Chris

[SoftwareApe]

@cpsauer thank you, I indeed didn't see these in the readme. I activated both exclude_external_sources == True and exclude_headers = "all", excluding external sources and excluding headers and it works for me.

load("@hedron_compile_commands//:refresh_compile_commands.bzl", "refresh_compile_commands")
refresh_compile_commands(
    name = "refresh_compile_commands",
    exclude_external_sources = True,
    exclude_headers = "all",
)

[cpsauer]

Sweet! Carl, for my continued learning: what are you using the compile commands for, if not clangd?

[SoftwareApe]

For SonarQube static analysis, so we only need to analyze internal targets, and only one compile command per translation unit.

[cpsauer]

Got it! Thanks. Lots more people are using this for SonarQube than I'd have guessed when I first wrote it!

Heads that you might sometimes still get multiple commands for the same file if it's compiled multiple ways, e.g., cross compiled for multiple platforms. If that's a meaningful problem, please do lmk.

Are there any other needs for use with SonarQube that I should know about? Or any tricky setup things we should document? I'm curious also how you found this tool!

[SoftwareApe]

@cpsauer I found it by Googling for compile_commands.json and Bazel. I found a few tools and this one seemed to get the most references and be the most maintained.

If there are multiple versions of the same translation unit I think that's fine, since they might cause different static analysis results. I only wanted to avoid having the same analysis executed 400 times.

In terms of setup it was quite easy to get to running with the refresh_all command.

For static analysis one thing that could be interesting is producing only the generated files, which worked with this command for us:

$bazel build $(bazel query 'filter(".*\.(?:cpp|hpp|inl|c|h|cc|hh|cxx|hxx)$", kind("generated file", //...:*))')

This generates all the C or C++ file targets without running any actual build on them, which is quite neat, so we don't need to wait for a full build to finish. You currently mention running a build for them to exist, but this is more minimal, so the static analysis can start before the build has finished, speeding up our CI a lot.

[SoftwareApe]

@cpsauer small follow-up. I got far enough to get SonarQube to start it's analysis. However I got an issue with symlink loops due to the external folder.

I noticed this folder is created by the compile commands tooling.

>>> Automatically added //external workspace link:

Looking through the code it would be nice to be able to disable this symlinking.

[SoftwareApe]

follow-up follow-up, without the external folder, the sonar-scanner works, but doesn't find some files 🤕

[SoftwareApe]

Replacing the link to external in the compile commands with the fully qualified target seems to make SonarQube happy enough.

sed -i "s|\"external/|\"$(readlink -f external)/|g" compile_commands.json

[cpsauer]

Thanks a bunch for getting back to me, Carl!

Replying to each in turn.

• Good to hear we were able to convince you from a quick google. IMO this is definitely the best game in town for this need. (Wouldn't have spent time building it otherwise...)
• I'd love to just build the generated files--but there's a problem with that command in the general case. If targets configure each other, like to cross-compile (what Bazel's docs would call transitions) that'll build to the wrong location--and potentially the wrong files, since query will try to issue build instructions to the un-configured versions of the targets. Sounds like you aren't hitting that particular edge case, but I thought I should give you a heads up. I filed an issue for us to track about this https://github.com/bazelbuild/bazel/issues/17660
  • This did remind me that the generated file warning should always encourage people to run the build with --keep_going, so I just pushed up a commit to fix that. Thanks!
• External link: This one's thorny. We need that external link to make sure the workspace has the same structure and file set at the build sandbox (as you found out with the missing files). It turns out auto-patching it is trickier than it might seem--and there are quite a few valid flag settings, e.g. "-Iexternal/curl/lib", patterns that will slip through that sed. Hence requiring the symlink as a massively simplifying assumption.
  • I'm really surprise by the symlink loop... Does SonarQube give any more specifics? Might it be worth filing a bug with them to better tolerate?

Thanks again, Chris

[SoftwareApe]

@cpsauer Thank you! I'm still in the process of getting things to work and not 100% there yet, so any pointers are helpful 👍 .

I've noted a link to https://github.com/bazelbuild/bazel/issues/17660 in my code, in case the file generation breaks things in the future. Especially since cross-compilation is likely going to be integrated in the next few months. Right now it seems to work for us. But if in the future there's an issue here at least we know where to look.

Regarding the external link I don't like this hack either, and I can see how it could create issues. I will ask SonarSource to clarify why their scanner crashes here. I looked at the error message more in detail, and it seems to be the actual issue is a permissions issue. SonarQube tries to index everything within the workspace, but some files linked through external don't have read permissions. I can't find a setting to exclude these properly in SonarQube though.

[SoftwareApe]

I had a simpler idea for resolving this issue while waiting for a fix from SonarSource. This also keeps the paths the same, and therefore doesn't thrash the cache.

# Create an external symlink outside the indexed directory
ln -s $(readlink -f external) ../external
# Replace all references to it in compile_commands.json
sed -i "s|\"external/|\"../external/|g" compile_commands.json
## Remove symlinked folder where the sonar-scanner tries indexing it
rm -rf external

[cpsauer]

:) Thanks for being great, Carl. You're very welcome; thanks for your persistence and ingenuity and for communicating with SonarSource to make this better for everyone.

[I'm surprised to hear about the lack of read permissions...is there more I should know there?]

[SoftwareApe]

Hi Chris,

interesting update here. Moving external to ../external works, and SonarQube doesn't seem to care so much about read permissions outside the indexed folder. This generated some more log messages regarding the symlink loops.

I'm getting this log here:

skipping symbolic link <project-dir>/../external/glew/lib/x86_64-linux-gnu/hdf5/openmpi/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/glew/lib/x86_64-linux-gnu/hdf5/serial/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/tinyxml2/bin/X11/X11/X11/X11/X11/X11 -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/tinyxml2/lib/x86_64-linux-gnu/hdf5/openmpi/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/tinyxml2/lib/x86_64-linux-gnu/hdf5/serial/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/pcl/bin/X11/X11/X11/X11/X11/X11 -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/pcl/lib/x86_64-linux-gnu/hdf5/openmpi/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/pcl/lib/x86_64-linux-gnu/hdf5/serial/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/curl/bin/X11/X11/X11/X11/X11/X11 -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/curl/lib/x86_64-linux-gnu/hdf5/openmpi/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/curl/lib/x86_64-linux-gnu/hdf5/serial/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/gl/bin/X11/X11/X11/X11/X11/X11 -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/gl/lib/x86_64-linux-gnu/hdf5/openmpi/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/gl/lib/x86_64-linux-gnu/hdf5/serial/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/sdl2/bin/X11/X11/X11/X11/X11/X11 -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/sdl2/lib/x86_64-linux-gnu/hdf5/openmpi/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/sdl2/lib/x86_64-linux-gnu/hdf5/serial/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/glvnd/bin/X11/X11/X11/X11/X11/X11 -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/glvnd/lib/x86_64-linux-gnu/hdf5/openmpi/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/glvnd/lib/x86_64-linux-gnu/hdf5/serial/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/pngpp/bin/X11/X11/X11/X11/X11/X11 -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/pngpp/lib/x86_64-linux-gnu/hdf5/openmpi/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/pngpp/lib/x86_64-linux-gnu/hdf5/serial/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/console_bridge/bin/X11/X11/X11/X11/X11/X11 -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/console_bridge/lib/x86_64-linux-gnu/hdf5/openmpi/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/console_bridge/lib/x86_64-linux-gnu/hdf5/serial/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/opencv/bin/X11/X11/X11/X11/X11/X11 -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/opencv/lib/x86_64-linux-gnu/hdf5/openmpi/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/opencv/lib/x86_64-linux-gnu/hdf5/serial/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/egl/bin/X11/X11/X11/X11/X11/X11 -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/egl/lib/x86_64-linux-gnu/hdf5/openmpi/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/egl/lib/x86_64-linux-gnu/hdf5/serial/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/openssl/bin/X11/X11/X11/X11/X11/X11 -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/openssl/lib/x86_64-linux-gnu/hdf5/openmpi/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/openssl/lib/x86_64-linux-gnu/hdf5/serial/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/openexr/bin/X11/X11/X11/X11/X11/X11 -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/openexr/lib/x86_64-linux-gnu/hdf5/openmpi/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/openexr/lib/x86_64-linux-gnu/hdf5/serial/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/zlib/bin/X11/X11/X11/X11/X11/X11 -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/zlib/lib/x86_64-linux-gnu/hdf5/openmpi/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/zlib/lib/x86_64-linux-gnu/hdf5/serial/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/atomic/bin/X11/X11/X11/X11/X11/X11 -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/atomic/lib/x86_64-linux-gnu/hdf5/openmpi/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/atomic/lib/x86_64-linux-gnu/hdf5/serial/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/ncurses/bin/X11/X11/X11/X11/X11/X11 -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/ncurses/lib/x86_64-linux-gnu/hdf5/openmpi/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/ncurses/lib/x86_64-linux-gnu/hdf5/serial/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/glut/bin/X11/X11/X11/X11/X11/X11 -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/glut/lib/x86_64-linux-gnu/hdf5/openmpi/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
skipping symbolic link <project-dir>/../external/glut/lib/x86_64-linux-gnu/hdf5/serial/lib/lib/lib/lib/lib/lib -- too many levels of symbolic links.
13:08:02.662 WARN: Property 'sonar.cxx.cobertura.reportPaths': cannot find any files matching the Ant pattern(s) '<project-dir>/../**/*-coverage-report.xml'

Indeed all these folders have symbolic links to . which seems kind of loopy to me.

Regarding the file permissions there seem to be some files (I'm not sure what these are) that don't have read permissions, e.g.

external/pcl/NX symlinks to /usr/NX, and /usr/NX/var/db/ports seems have no read permissions for users.

It's just that SonarQube too eagerly tries to read everything without a fallback in case it's not possible. I'm guessing the average non-Bazel project doesn't trigger such corner cases, because usually you can read everything in your source folder.

[cpsauer]

Huh! Bizarre. Indeed feels very unnecessarily loopy. Surprised they're set up that way; I haven't seen structures like that in my repos. I wonder what's setting up those triple-based paths. Perhaps rules_foreign_cc calling some external build system. If you do discover why, I'd be curious, but a SonarQube ignore function--or a softer failure--does seems like the fix for structures like that.