[Compile Commands] Quotes are stripped from preprocessor definitions in cache json files

[SE2Dev]

It would seem that for a compile_commands.json that includes a command which defines preprocessor values as strings, the quotes are stripped when the cquery caches the files. For example: in a compile_commands.json that includes the following:

    "directory": "...",
    "file": "...",
    "command": "... -D__BUILD_CONFIG__=\"Debug\" -D__BUILD_PLATFORM__=\"x64\" -D__BUILD_PROJECT__=\"demo\" ..."
  },

The resulting cache file would look similar to this:

{
    "version": 5,
    "last_modification_time": 1511303313,
    "import_file": "...",
    "args": [
        "cc",
        "-D__BUILD_CONFIG__=Debug",
        "-D__BUILD_PLATFORM__=x64",
        "-D__BUILD_PROJECT__=demo",
        "-DDEBUG",
        "...",
        "-xc++",
        "-std=c++11",
        "-resource-dir="/misc/cquery/clang_resource_dir"
    ]
}

Notice how the values for __BUILD_PROJECT__, etc. no longer have their quotes; This results in errors such as the following:

[jacobdufault]

This may be a bug upstream in libclang. I don't believe cquery does any special processing w.r.t. quotes .

[SE2Dev]

I suppose that is possible - however it's worth noting that the code itself does compile with MSVC, GCC, and most importantly, Clang regardless of the presence of these errors provided by cquery.

[SE2Dev]

This appears to be a combination of a number of issues - none of which are the fault of cquery. First, the official documentation for the JSon Compilation Database format (found here), provides the following example:

[
  { "directory": "/home/user/llvm/build",
    "command": "/usr/bin/clang++ -Irelative -DSOMEDEF=\"With spaces, quotes and \\-es.\" -c -o file.o file.cc",
    "file": "file.cc" },
  ...
]

This example does not work as expected. When libclang loads a compile_commands.json it appears to go through multiple stages: loading the JSon data, and applying transformations for escape characters. When loading the example above, the \" pieces of the string are transformed into " and used by the first transformation stage to guarantee that all content between the two quotes are part of the same define (for SOMEDEF). Similary the \\ is transformed into a single \ and then ignored. To get the expected behavior one would need the following:

[
    {
        "directory": "/home/user/llvm/build",
        "command": "/usr/bin/clang++ -Irelative -DSOMEDEF=\"\\\"With spaces, quotes and \\\\-es.\\\"\" -c -o file.o file.cc",
        "file": "file.cc"
    }
]

Additionally the script I was using to generate compile_commands.json was not correctly generating these escape patterns, which in turn was creating the issue I was experiencing.

[MaskRay]

In this case "arguments" is a better fit than "command".

[Boddlnagg]

I just want to mention that I'm running into the same problem with compile_commands.json generated by Ninja (which seems to be "supported" according to your documentation). Ninja writes "command", not "arguments", and if the "command" includes these kinds of escaped quotes (in -D defines), the command doesn't seem to get parsed succesfully at all, which means that cquery (in VS Code) doesn't work at all ... but I wasn't able to find any error message indicating what went wrong. Manually editing my compile_commands.json to switch to "arguments" made cquery work (it took a while for me to figure this out!), but the quotes still seem to be removed from the -D argument, because I'm then getting compile errors where the define is used (similar to @SE2Dev's example).

This might not be cquery's fault, but it seems to make cquery unusable in some cases. I'm probably going to need to write a script that transforms "command" to "arguments" and "fixes" the escape sequences in a way that works (if possible) ...

[MaskRay]

@Boddlnagg What's the tool you use to generate build.ninja? If it is cmake, cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=YES. Don't use ninja -t compdb

[Boddlnagg]

I do use ninja -t compdb, because the project doesn't use cmake, nor any other off-the-shelf meta-build tool. I'm generating the ninja buildfile myself, and hoped that this would (as a byproduct) allow me to get compile_commands.json for free. But it seems like I need to do that manually as well.

[MaskRay]

A proper "arguments" output from Ninja requires adding vector<string> functions https://github.com/ninja-build/ninja/blob/master/src/eval_env.cc#L98 This does not seem easy to fix in the upstream.

You can also chime in in https://github.com/ninja-build/ninja/pull/1381

[Boddlnagg]

This is not related to command vs arguments after all. Ninja generates a command that contains -DSOMEDEF=\\\"foobar\\\", this works with rtags, but not with cquery. I can postprocess the compile_commands.json with in the way that @SE2Dev suggested and rewrite the problematic parameters to -DSOMEDEF=\"\\\"foobar\\\"\" (using a sed script sed --expression='s/\(\\\\\\\"[^"]\+\\\\\\\"\)/\\\"\1\\\"/g')

Therefore I think that this is a cquery bug.

[jacobdufault]

Given this works with rtags this seems like a valid issue. @Boddlnagg the code to handle this is in project.cc, would you like to take a look at fixing?

[MaskRay]

this is a bug of ninja. its command line expansion code produces a string, instead of a list of arguments. when it emits compdb, there is no proper way to determine the argument boundary.

rtags does more scrubbing to "address" the issue of bad compdb producers, but this should be fixed in ninja

On Wed, Mar 28, 2018, 08:52 Jacob Dufault notifications@github.com wrote:

Given this works with rtags this seems like a valid issue. @Boddlnagg https://github.com/Boddlnagg the code to handle this is in project.cc, would you like to take a look at fixing?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cquery-project/cquery/issues/73#issuecomment-376937540, or mute the thread https://github.com/notifications/unsubscribe-auth/AAZaQlz68dZwhLeKZ3sYUTzomjC6d8ECks5ti7GwgaJpZM4QmynJ .

[MaskRay]

this reopened issue does not make much sense. put the warning on the wiki page

On Wed, Mar 28, 2018, 09:10 Ray emacsray@gmail.com wrote:

this is a bug of ninja. its command line expansion code produces a string, instead of a list of arguments. when it emits compdb, there is no proper way to determine the argument boundary.

rtags does more scrubbing to "address" the issue of bad compdb producers, but this should be fixed in ninja

On Wed, Mar 28, 2018, 08:52 Jacob Dufault notifications@github.com wrote:

Given this works with rtags this seems like a valid issue. @Boddlnagg https://github.com/Boddlnagg the code to handle this is in project.cc, would you like to take a look at fixing?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/cquery-project/cquery/issues/73#issuecomment-376937540, or mute the thread https://github.com/notifications/unsubscribe-auth/AAZaQlz68dZwhLeKZ3sYUTzomjC6d8ECks5ti7GwgaJpZM4QmynJ .

[jacobdufault]

https://github.com/Andersbakken/rtags/blob/2abf5213815a7389df28742464d4a9ef0335f91f/src/Server.cpp#L525-L538

It seems like this is how rtags handles it. It doesn't seem like such a burden to get it working automatically.

[Boddlnagg]

I think it is fundamentally impossible to fix this in Ninja, since for Ninja a command is also just simply a string, not an array of arguments. Ninja is cross-platform, and on Windows Ninja just passes that string to CreateProcess, and then the process itself is responsible for parsing that string.

Furthermore, as was already mentioned in this thread, the command and arguments JSON fields should be exchangable, and I would expect cquery to work with any of them just fine. It is not only Ninja that produces a single command string (Bear does the same thing, if I remember correctly).

@jacobdufault Maybe I'm able to find some time and come up with a PR, but it might take a while

[jacobdufault]

Sure, take your time.

[MaskRay]

% cat b.cc
#include <stdio.h>
int main() { puts(D); }

% clang++ b.cc -o b -DD=\\\"ab\\\" && ./b
In file included from <built-in>:341:
<command line>:1:12: warning: missing terminating '"' character [-Winvalid-pp-token]
#define D \"ab\"

% clang++ b.cc -o b '-DD="ab"' && ./b  
ab

If you put extra double quotes around arguments, it will conflict with other sane compile_commands.json producers.

ninja is not suitable to produce compile_commands.json and such use is a minority. You could improve the current https://github.com/cquery-project/cquery/wiki/compile_commands.json#stdout-of-an-external-command facility instead of spending more lines of code to work around the ninja quirk.

[jacobdufault]

@MaskRay let's see what @Boddlnagg can come up with.

If working around the quirk is relatively simple then that is a significantly better solution than adding to the wiki. If we solve it here, then no user has to deal with it.

I'm also hesitant to recommend cquery's ability to run stdout of an external command, since it doesn't support windows. Also, it seems to duplicate something that can be done pretty easily by the build system, so I'm more inclined to remove it entirely.

[jacobdufault]

Reading over the issue again, I think we should either fix libclang so it properly extracts the quotes or write a custom compile_commands.json loader.

[Boddlnagg]

Reading over the issue again, I think we should either fix libclang so it properly extracts the quotes or write a custom compile_commands.json loader.

That sounds like the right solution to me as well. But I don't think that I can help much there ...

[svorobiev]

this breaks clang-check in exact same way as it breaks cquery.

Currently I work around it by checking the json for escapes, and moving offending -Ds into into cquery.index.extraClangArguments

[jacobdufault]

@svorobiev It'd be awesome if you filed a bug against upstream, esp since clang-check is broken.