Generate JSON Compilation Database

[bschommer]

Desired Behavior

JSON Compilation Databases are currently widely used for C/C++ tooling and in general the format is general enough that one could image using it also for some OCaml tooling. This would also help when one develops C/C++ bindings, since then one could use the builtin support of many editors and other tools that rely on the information stored in the database.

I would suggest adding (generate_compilation_database <value>) to the dune-project in order to enable/disable generation of compilation databases.

Example

The generated compilation database for example could look like this:

[
  { "directory": "/home/user/build",
    "command": "/usr/bin/ocamlopt.opt -Irelative -c file.ml",
    "file": "file.ml" },
    { "directory": "/home/user/build",
    "command": "/usr/bin/clang -Irelative -DSOMEDEF=\"With spaces, quotes and \\-es.\" -c -o foo.o foo.c",
    "file": "foo.c" },
  ...
]

[nojb]

Did not know about this; looks interesting!

[bobot]

Partly linked to ocaml/merlin#737 for the use with OCaml tooling. But for C/C++ it has an interest in itself.

[ghost]

I remember that we looked into it for Merlin a few years ago (the issue pointed by @bobot) and it didn't seem suitable for the job. So that I'm not convinced this would be that useful for OCaml tooling.

For improving developing C/C++ support with Dune, why not. This would be one more feature would we have to maintain, so it all depends how much code this would add to Dune and how maintainable it would be. Ideally, it would be best if such a feature could be provided outside of Dune.

[bschommer]

I remember that we looked into it for Merlin a few years ago (the issue pointed by @bobot) and it didn't seem suitable for the job. So that I'm not convinced this would be that useful for OCaml tooling.

For improving developing C/C++ support with Dune, why not. This would be one more feature would we have to maintain, so it all depends how much code this would add to Dune and how maintainable it would be. Ideally, it would be best if such a feature could be provided outside of Dune.

I'm not sure how one could implement writing the compilation database outside of dune.

The format is already close enough to the information printed for logging, so I would suspect that one could reuse some parts of this.

[ghost]

You could probably generate it from the output of dune rules.

BTW, are JSON compilation databases used outside of the C/C++ world?

[bschommer]

You could probably generate it from the output of dune rules.

I will look into that, thanks for the hint.

BTW, are JSON compilation databases used outside of the C/C++ world?

Not that I'm aware of, the main uses I have seem are either for using it in combination with clangd or clang-tidy and some other static analysis tools used for C/C++.

[ghost]

I will look into that, thanks for the hint.

No problem. If you do want to use dune rules in the end, we should add the following arguments as we did for dune describe: --lang 2.6 --format csexp. So that the output is stable and easy to parse.

Not that I'm aware of, the main uses I have seem are either for using it in combination with clangd or clang-tidy and some other static analysis tools used for C/C++.

Ok. So in this case it does seem better to implement it outside of Dune.

[bschommer]

No problem. If you do want to use dune rules in the end, we should add the following arguments as we did for dune describe: --lang 2.6 --format csexp. So that the output is stable and easy to parse.

That would have been my next question, how to parse the output efficiently.

[ghost]

Do you want to have a look a adding support for csexp output in dune rules? Essentially, you need to extract the Format module and a couple more things from bin/describe.ml so that it can be reused in bin/print_rules.ml.

You can then use the csexp library to parse the output. You can use it combination with sexplib and ppx_sexp_conv if you want, or do the parsing manually.

If later you wanted to release a tool, it would also be good to add support for versioning via a --lang argument so that new releases of Dune don't break your tool. But that's not immediately necessary for a prototype.

[ghost]

BTW, you might also want an option to serialise the action field as a shell command rather than the DSL. That'll make it simpler to produce the JSON compilation database.

[bschommer]

Thanks, I will have a look a this.

[edwintorok]

I implemented this as a separate tool here that parses dune rules: https://github.com/edwintorok/dune-compiledb/ It becomes a little bit awkward trying to invoke it from dune itself (dune really doesn't like nested calls, plus it'll be inside the build dir not the source dir), so in order to test it I wrote some cram style tests, usage should be pretty simple:

dune rules | dune-compiledb

Let me know whether this is useful and would like to keep it as a separate tool or add its functionality to dune itself.

I wrote this because I have tool that generates C headers and manually setting up clangd to find them is tedious.

BTW if you do not have generated files, and do not use any custom includes then the following also works for clangd and is a lot simpler (but I need the compilation database anyway for other reasons, e.g. goblint also uses it):

(rule
 ; editor integration: generate include paths for LSPs such as clangd
 (target compile_flags.txt)
 (mode promote)
 (enabled_if
  (= %{system} linux))
 (action
  (with-stdout-to
   compile_flags.txt
   (pipe-stdout
    (progn
     (echo "-Wall -Wextra -Wstrict-prototypes -D_FORTIFY_SOURCE=2 ")
     (echo
      %{ocaml-config:ocamlc_cppflags}
      %{ocaml-config:ocamlc_cflags}
      -I%{ocaml_where}
      -I)
     (system pwd))
    (system "xargs -n1 echo") ; the format is a single flag per line
    ))))

BTW, are JSON compilation databases used outside of the C/C++ world? Not that I'm aware of, the main uses I have seem are either for using it in combination with clangd or clang-tidy and some other static analysis tools used for C/C++. Ok. So in this case it does seem better to implement it outside of Dune.

goblint uses them, it is an OCaml tool for statically analyzing C code, but can be used on OCaml C stubs ;)