Generate compile_commands.json

[grafikrobot]

Many IDEs and downstream tools want to parse a compile_commands.json. The ideal solution for B2 would be to have single command actions to record. But since B2 currently doesn't guarantee such single command actions we need something else until we get those. The following is a "kludge" way of doing it. But something is better than nothing. What we need to implement:

1. Add a feature to register action pre-execution hooks in the engine side. Such hooks would be given the action script string and can process the string. Possibly returning a different string to execute.
2. Add a compile commands pre-launch step that loads an existing "compile_commands.json" at project root and writes it out at B2 exit.
3. Add a action pre-exec hook to watch for per-toolset compile (and link?) commands that extracts the single command line invocation (by matching a regex in the action script). Control would be by adding a target variable COMPILE_COMMAND that sets a regex that extracts the command.
4. Change, initially, the gcc/clang toolset to set the COMPILE_COMMAND for each appropriate action.

[vinniefalco]

What is this "load an existing compile_commands.json?" Why?

[grafikrobot]

What is this "load an existing compile_commands.json?" Why?

Because B2 can do very fine grained incremental builds. And if you want to get a full picture of the project you can't throw away the history of what you previously compiled on each b2 invocation. Especially if it's an invocation that's a no-op, which would create an empty json file.

[vinniefalco]

That is a nice to have feature but maybe not strictly required for an initial proof of concept.

[grisumbras]

I think, the described approach is actually not needed. Running with different build requests, and appending commands to compile_commands.json is both unnecessary complexity and potentially problematic, when e.g. some targets are removed from the build scripts.

Rather, I think something like b2 variant=debug,release --compile-db should create a new file for all jam targets it considers for update (not necessarily updates). Maybe we also don't want it to actually try updating them, although there is already -n flag for that.

There's also a question what to do with ancillary targets e.g. created by ac module. Maybe potentially add an engile rule that marks a target as hidden?

Now, that I think of it, what if we can mark Jam targets as watched (maybe by setting a variable on them), and then their actions get recorded into some list in the form of command output sources?

[grafikrobot]

After my many questions in slack, adjusted plan:

1. Add a feature to register action pre-execution hooks in the engine side. Such hooks would be given the action script string and can process the string. Possibly returning a different string to execute.
2. Add a action pre-exec hook to watch for per-toolset compile (and link?) commands that extracts the single command line invocation (by matching a regex in the action script). Control would be by adding a target variable COMMAND_DATABASE that sets a regex that extracts the command.
3. Change, initially, the gcc/clang toolset to set the variable for each appropriate action.
4. Add a command line arg --command-database=json that simulates the build request (i.e. equivalent to -n -a). That collects all the registered action commands and writes out the resulting json to <build-dir>/compile_commands.json.
5. Add a command line arg --command-database-out=<filename> to specify an override to the file name and location.

[grisumbras]

LGTM Also, AFAIK, link commands aren't needed.

[grafikrobot]

Step 0? Structured data output to JSON complete: https://github.com/bfgroup/b2/commit/c30bce8e2df15aebaf6ab646238a4fec30327346

[grafikrobot]

WIP here.. https://github.com/bfgroup/b2/tree/feature/compiledb