clangd extract query-driver from compilation_commands.json

[petr-tik]

If/when https://github.com/emacs-lsp/lsp-mode/pull/2303 lands, we will be able to find compile-commands.json in the project subtree. This might be useful for another UX quality of life improvement. Even after lsp-clangd launches succesfully, it might start complaining about includes that it cannot resolve.

By default, clangd uses the default compiler invokation to look for headers, which causes issues for people on MacOS or people using non-standard C++ compilers.

https://github.com/clangd/clangd/issues?q=is%3Aissue+is%3Aopen+%22query-driver%22

However, once clangd has access to the compilation_commands.json it should be able to extract the exact compiler command from the command field for each file. The clangd team have an enhancement issue and maybe will auto-detect it in the future. https://github.com/clangd/clangd/issues/539

I wanted to suggest a heuristic to auto-set it after compile_commands is succesfully found. I am thinking about a monster bash command that will grep the compile_commands.json extract all commands and return the most frequent. It's hacky and dirty, but might be useful.

The compilation_commands.json is of the following format

head -30 compile_commands.json
[
{
  "directory": "/home/petr_tik/Coding/rr/build",
  "command": "/usr/bin/c++ -DRR_VERSION=\\\"5.3.0\\\" -I/home/petr_tik/Coding/rr/include -I/home/petr_tik/Coding/rr/third-party/proc-service -I/home/petr_tik/Coding/rr/third-party/brotli/include -I/home/petr_tik/Coding/rr/build  -D__USE_LARGEFILE64 -pthread -msse2 -D__MMX__ -D__SSE__ -D__SSE2__ -D__STDC_LIMIT_MACROS -D__STDC_FORMAT_MACROS -std=c++14  -pthread -g -o CMakeFiles/test-monitor.dir/src/test-monitor/test-monitor.cc.o -c /home/petr_tik/Coding/rr/src/test-monitor/test-monitor.cc",
  "file": "/home/petr_tik/Coding/rr/src/test-monitor/test-monitor.cc"
},
{
  "directory": "/home/petr_tik/Coding/rr/build",
  "command": "/usr/bin/cc -DRR_VERSION=\\\"5.3.0\\\" -I/home/petr_tik/Coding/rr/include -I/home/petr_tik/Coding/rr/third-party/proc-service -I/home/petr_tik/Coding/rr/third-party/brotli/include -I/home/petr_tik/Coding/rr/build  -D__USE_LARGEFILE64 -pthread -msse2 -D__MMX__ -D__SSE__ -D__SSE2__ -Wstrict-prototypes -std=gnu11  -pthread -g -o CMakeFiles/ftrace_helper.dir/src/ftrace/ftrace_helper.c.o -c /home/petr_tik/Coding/rr/src/ftrace/ftrace_helper.c",
  "file": "/home/petr_tik/Coding/rr/src/ftrace/ftrace_helper.c"
},
{
  "directory": "/home/petr_tik/Coding/rr/build",
  "command": "/usr/bin/cc -DRR_VERSION=\\\"5.3.0\\\" -I/home/petr_tik/Coding/rr/include -I/home/petr_tik/Coding/rr/third-party/proc-service -I/home/petr_tik/Coding/rr/third-party/brotli/include -I/home/petr_tik/Coding/rr/build  -D__USE_LARGEFILE64 -pthread -msse2 -D__MMX__ -D__SSE__ -D__SSE2__ -Wstrict-prototypes -std=gnu11  -pthread -g -o CMakeFiles/core_count.dir/src/chaos-test/core_count.c.o -c /home/petr_tik/Coding/rr/src/chaos-test/core_count.c",
  "file": "/home/petr_tik/Coding/rr/src/chaos-test/core_count.c"
},
{
  "directory": "/home/petr_tik/Coding/rr/build",
  "command": "/usr/bin/cc -DRR_VERSION=\\\"5.3.0\\\" -I/home/petr_tik/Coding/rr/include -I/home/petr_tik/Coding/rr/third-party/proc-service -I/home/petr_tik/Coding/rr/third-party/brotli/include -I/home/petr_tik/Coding/rr/build  -D__USE_LARGEFILE64 -pthread -msse2 -D__MMX__ -D__SSE__ -D__SSE2__ -Wstrict-prototypes -std=gnu11  -pthread -g -m32 -Wall -Wextra -DDEBUG -UNDEBUG -g3 -O0 -Werror -o CMakeFiles/watchpoint_unaligned_32.dir/32/watchpoint_unaligned.c.o -c /home/petr_tik/Coding/rr/build/32/watchpoint_unaligned.c",
  "file": "/home/petr_tik/Coding/rr/build/32/watchpoint_unaligned.c"
},
{
  "directory": "/home/petr_tik/Coding/rr/build",
  "command": "/usr/bin/cc -DRR_VERSION=\\\"5.3.0\\\" -I/home/petr_tik/Coding/rr/include -I/home/petr_tik/Coding/rr/third-party/proc-service -I/home/petr_tik/Coding/rr/third-party/brotli/include -I/home/petr_tik/Coding/rr/build  -D__USE_LARGEFILE64 -pthread -msse2 -D__MMX__ -D__SSE__ -D__SSE2__ -Wstrict-prototypes -std=gnu11  -pthread -g -m32 -Wall -Wextra -DDEBUG -UNDEBUG -g3 -O0 -Werror -o CMakeFiles/watchpoint_size_change_32.dir/32/watchpoint_size_change.c.o -c /home/petr_tik/Coding/rr/build/32/watchpoint_size_change.c",
  "file": "/home/petr_tik/Coding/rr/build/32/watchpoint_size_change.c"
},
{
  "directory": "/home/petr_tik/Coding/rr/build",
  "command": "/usr/bin/cc -DRR_VERSION=\\\"5.3.0\\\" -I/home/petr_tik/Coding/rr/include -I/home/petr_tik/Coding/rr/third-party/proc-service -I/home/petr_tik/Coding/rr/third-party/brotli/include -I/home/petr_tik/Coding/rr/build  -D__USE_LARGEFILE64 -pthread -msse2 -D__MMX__ -D__SSE__ -D__SSE2__ -Wstrict-prototypes -std=gnu11  -pthread -g -m32 -Wall -Wextra -DDEBUG -UNDEBUG -g3 -O0 -Werror -o CMakeFiles/watchpoint_no_progress_32.dir/32/watchpoint_no_progress.c.o -c /home/petr_tik/Coding/rr/build/32/watchpoint_no_progress.c",
  "file": "/home/petr_tik/Coding/rr/build/32/watchpoint_no_progress.c"

After the potential-query-driver is extracted we can check if it's executable and if so, set up as an arg for clangd.

One potential problem might be the absolute path, which can be a mess with clangd over tramp. Another problem are potentially large json files (i have seen 15000 LoC files) that we can either take a head -100 of or do something else.

[yyoncho]

CC @sebastiansturm @danielmartin .

[danielmartin]

Your heuristic may work, but I'm not sure if doing this from the LSP client will be fast enough. To avoid penalizing the common case, maybe we could apply this heuristic only in very specific cases (like the clangd repo issue mentions, for example when the include errors are for includes with angle brackets, ie. they are probably missing system headers).

After the potential-query-driver is extracted we can check if it's executable and if so, set up as an arg for clangd.

We shouldn't whitelist arbitrary binaries, that opens for a vulnerability if someone crafts a special compile_commands.json. We'd need to ask the user something like "Should I run /path/driver to query for additional system headers and improve code intelligence?" (not sure if it's the best wording).

Could you give it a try so we can see how this approach works in code? I'm not sure if a Bash script is the best approach, I think we could do the file grepping using Emacs file/buffer APIs? It'll probably be faster.

[danielmartin]

I've also bookmarked your PR https://github.com/emacs-lsp/lsp-mode/pull/2303 to review it when I have some time this weekend. Thank you for your contributions to lsp-mode!