Add support for compile commands json

[agauniyal]

Feature Request - Most of the build systems nowadays have support for generating compilation database file, vscode cpp plugin should also support it. For more information - http://clang.llvm.org/docs/JSONCompilationDatabase.html

[d-xo]

Would love to see this, setting up include paths / compiler options (especially on large projects) is one of the (very few) pain points for c++ development in vs code.

[imikejackson]

I gave up setting up our CMake based project because I would have to manually enter a large number of include paths into the settings file. If there were support for the compile_commands file that CMake can generate automatically the setup would be super simple. Here is hoping this could be integrated into VSCode in the near future.

[MathiasMagnus]

Indeed, just came across the exact same issue. Playing along with cmake-server mode, this type of info isn't hard to obtain. Its only a matter of coming up with the responsibility, of which extension should take care of populating the list of include files.

[imikejackson]

So I wrote a short c++ program to parse the compile_commands.json file and rewrite the c_cpp_properties.json file. This still did not allow any completions beyond the basic "Ctags" style completions. My guess (and reading around a bit) is that the clang based code completion is not a full blown code completion like in other IDE's such as Xcode, QtCreator and Visual Studio. If anyone wants the c++ code they are welcome to the hack.

[MathiasMagnus]

The C++ Docs page still suggests that

Q: IntelliSense isn't working. A: In this release, IntelliSense isn't supported. We plan to enable this and other features in future releases.

Which to me implies that full-blown IntelliSense is known not to work. I might be wrong about this, but AFAIK IntelliSense in big brother VS is done through Roslyn which would have to be ported over to Linux/OSX as well as hooking it into the editor of Code.

Correct me if I'm wrong.

[sean-mcmanus]

Yes, full-blown IntelliSense isn't finished yet. Roslyn is a .NET C#/VB compiler, and not a C++ compiler.

[d-xo]

Just as a note, the ycmd plugin from vscode offers this functionality (as well as proper autocomplete via clang):

https://github.com/richard1122/vscode-youcompleteme

[MathiasMagnus]

I know, hope never dies, but could you give us a heads up where this feature is inside the pipeline? There have been no words from it from the CMake side (as far as @vector-of-bool is concerned).

[sean-mcmanus]

Yes, it's still in the pipeline, near the front.

[agauniyal]

@xwvvvvwx but it has some irritating bugs, check out the issues.

[vector-of-bool]

I'd really like there to be some tighter integration with the build system. With the capabilities brought by cmake-server, I think that support for C and C++ in VSCode can be brought to parity with more heavy-duty IDEs. I also work in an environment with many custom include paths and compile flags, and when those are not used when processing the source files the resulting editing experience is next to useless. The build parameters are available for the tools to use, it's just a matter of making use of them in the tools that already exist.

I've been working on exposing an extension API from my own CMake extension specifically to help support native tooling, and I'd be really happy to see tools use the information provided by the build system rather than some other source. In the end, the project's build system (like CMake) is the ultimate source of knowledge on how to process the project and using any other source of information will result in a sub-par tooling experience.

[sean-mcmanus]

Yes, I agree. Visual Studio 2017 supports CMake in the open folder scenario with IntelliSense, but that only runs on Windows. Support for CMake (in the C++ extension) is further down the pipeline for VS Code.

[defunctzombie]

Is there a way to hook into setting the cpp include paths via an extension? This would allow temporary work-arounds to create simple extensions that would parse compile_commands files for include paths to extend the built-in c++ intellisense system.

[sean-mcmanus]

@defunctzombie You just read/parse/write the c_cpp_properties.json. You can look in the out/src/LanguageServer/C_Cpp_ConfigurtionProperties.js file to see how we do it.

[defunctzombie]

@sean-mcmanus out/src/LanguageServer/C_Cpp_ConfigurtionProperties.js would this be in the installed extension on my computer or is this referencing a file in some repo?

[sean-mcmanus]

@defunctzombie In the installed extensions folder, i.e. in the /Users/\<user>/.vscode/extensions/ms-vscode.cpptools-0.11.0 folder (might vary somewhat by OS).

[defunctzombie]

@sean-mcmanus Found the file and looked through it. I see how and where it sets up some features on the language server but would appreciate a pointer on which API call it makes to add includes. I found the vscode extension API docs but did not find a mention of adding include paths.

[sean-mcmanus]

@defunctzombie Do you see the parsePropertiesFile() method? That file calls fs.readFileSync and JSON.parse to read and parse our c_cpp_properties.json file. So you just use configurations[i].includePath or configurations[i].browse.path to access the list of includePaths (the browse.path is for our symbol engine, which is expected to match the locations of all cpp/header files for which symbols are desired, and the other one is for our IntelliSense engine, which is expected to match your compiler settings for the particular C/C++ files you want accurate error squiggles, hover, etc. for (when the IntelliSenseEngine setting is Default).

[defunctzombie]

@sean-mcmanus Thank you! Just to clarify so I don't travel down a bad path, vscode will scan any paths I add to configurations[i].includePath and configurations[i].browse.path. There is no other API call I need to make to inform vscode of the configuration.

[sean-mcmanus]

@defunctzombie No other API needs to be called. We have a file watcher on the c_cpp_properties.json that will run whenever the file is changed, so the changes should be picked up.

[MathiasMagnus]

@sean-mcmanus

Yes, it's still in the pipeline, near the front.

This statement was quite a while back (Feb 17). Also, the one month radio silence doesn't do much good for hopes on finally being able to use VS Code as a serious alternative to anything else. (That is for C++ development)

[sean-mcmanus]

@MathiasMagnus What feature are you talking about? Better autocomplete? I think that's what I was referring to. We're close to releasing it.

[MathiasMagnus]

@sean-mcmanus I was talking about the feature this issue refers to: supporting compile_commands.json Manually adding include paths is tedious, as is adding targets for debugging. @vector-of-bool has been pushing as well for the MS extension to support parsing compile_commands.json (which is ought to be the canonical way of detecting external build systems). CMake Tools would take on the role of populating compile_commands.json, but there's no use if the C++ extension does not care.

[jhasse]

Btw: CMake isn't the only build system with support for generating compile_commands.json, Waf can also do this: https://github.com/waf-project/waf/blob/master/waflib/extras/clang_compilation_database.py

[bobbrow]

We have begun designing how support for compile_commands.json should work in our extension. The current plan is to add a new property to the configuration objects in c_cpp_properties.json that tells the extension where to find the compilation database. If it is missing or corrupt, we will use the includePath and defines instead.

What would you expect the behavior to be if you open a source file not described by the compilation database (or a header file not included by a source file in the database)?

• Use the same includePath/defines as another source/header file in the same folder?
• Use the includePath/defines in the c_cpp_properties.json file?
• Use just the default system include path (similar to what we do for opening files without opening a workspace)

[vector-of-bool]

When I do look up for a file in the compdb, I do a fairly simple path normalization so that headers and sources resolve the same. Like this

[bobbrow]

Sorry, to be a bit more clear, the scenario I'm trying to describe is more about when new files have been added to the workspace, but compile_commands.json has not been regenerated, or perhaps you are looking at some other source or header file that is not actually used by the active configuration. For example, suppose folder "MyProject" has A.cpp, B.cpp, and C.cpp and compile_commands.json only has entries for A.cpp and B.cpp because it is out of date or because C.cpp is not actually compiled for the active configuration. How would you expect the language server to behave when C.cpp is opened?

We have code to help us map headers to source files, but there are cases where those source files may not exist in the database and we would like to have a well defined behavior for those cases. Disabling all language server features is undesirable though we could consider a policy to turn off linting for files not present in the database.

[vector-of-bool]

The best bet I'd take would be to use options for a file with the most similar absolute path (ie, files in the same directory probably use the same compile options). Failing that, I think the best would be to stick with the latter option of disabling linting or falling back on some user configurable "default" compile flags for unknown files.

[viveksjain]

Using the same options as another file in the directory would work for most projects. But what if the other files in the directory each have different flags/options? How do you decide which one is most "similar"? I'm not sure. Falling back to c_cpp_properties.json seems reasonable.

[MathiasMagnus]

I am the maintainer of the CMake build automation of triSYCL repository, which for instance has single source file targets packed into a single folder. Although the organizing incentive was grouping source files of similar properties together, it is by chance they share compilation flags. I cannot even really tell, there are 100+ targets.

I agree with @viveksjain, that a fallback would be cleaner than assuming something. My preferred UX would be notifying through pop-up yellow bar that a file is not present in compile-commands.json and thus a fallback is applied. For instance, I rename a source file but would forget to update the CMake scripts, or I rename a folder and multiple files lose their connection to the build process. Or I copy a new folder into the source tree containing C/C++ source files which are not yet part of the CMake scripts. Perhaps this notification could be turned-off from the settings json for rare corner cases one might not have thought of.

This notification will be misleading for CMake if somebody is using the GLOB feature, but AFAIK its use has been discouraged for multiple years now due to hazards similar to this. (Modifying files in the build tree which GLOB will follow but due to the incremental nature of the CMakeCache.txt the old/removed file name will remain in the cache resulting in broken builds.)

[bobbrow]

@MathiasMagnus it would be nice if VS Code gave us a way to insert a yellow bar above the code file (similar to what Visual Studio does). The closest thing to that in VS Code are the pop down notifications that usually render on top of your code file, can stack up quickly, and need to be dismissed with a button click (or timer expiration). We generally consider those to be obnoxious and avoid them when possible.

Another option is the status bar, but we think that notifications in the status bar tend to be missed since the icons are so small and way off in the periphery.

Our weapon of choice so far has been the problems window since reporting a problem to VS Code results in a squiggle drawn in your code that you can hover over to see the problem (in addition to being able to see it in the problems window). This makes the issue prominent enough that you can see it, but also non-invasive. The downside is that, unlike a pop down notification, it cannot be dismissed until the problem is addressed. We currently use this strategy to inform developers that we've taken a different action than they might expect. For example, if you are using the new IntelliSense engine and we are unable to resolve a #include, we revert back to using the old IntelliSense engine and turn off linting. We let you know about this change in behavior by adding a problem on the line of the #include that we could not resolve that explains what we are doing and why.

Our current plan is to use the pop down notification only for the case where we cannot find or open your compile_commands.json file. If you open a file that we can't trace back to the compilation database we would report a "problem" to VS Code in the file which would result in a green "information" squiggle drawn on the first line of code in the file. Does that seem reasonable?

[MathiasMagnus]

@bobbrow Yes, that seems totally legit.

[MathiasMagnus]

Could we get a status update on how far along the implementation is? It's closer to three months than two, that the feature was tackled. I was very much hoping to meet it in the September update to cpptools, but was somewhat disappointed it didn't make it. Should we get our hopes up for October?

[sean-mcmanus]

Sorry for the wait. We're still testing it, fixing broken automated tests, possibly some bug fixes, etc. I think we'll be able to release it "soon".

[ruipires]

Is there any prediction on when the next release will be? I can't wait to take this feature for a test drive!

[bobbrow]

I plan to release in the next hour. I am just preparing release notes and updated documentation right now.

[bobbrow]

0.14.0 has been released. If you encounter any issues with the implementation, please open a new issue.

[MathiasMagnus]

Awesome! Thanks guys!

I tried giving the feature a spin, but did not succeed. I did not find much about this feature in the docs (nothing to be precise), but I went ahead and tried getting it to work non the less.

I discovered, that the c_cpp_properties.json schema has a node "compileCommands", which I inserted into the win32 configuration and gave it the value "${workspaceRoot}/build/compile_commands.json" because this is where CMake Tools by default generates its database file. I looked inside for a simple C++ file and it was fairly slim.

CMakeLists.txt

cmake_minimum_required(VERSION 3.5)
project(vscodecpp LANGUAGES CXX)
add_executable(${PROJECT_NAME} Main.cpp)
target_include_directories(${PROJECT_NAME} PRIVATE "C:/")

compile_commands.json

[
{
  "directory": "c:/Users/Matty/Source/vscodecpp/build",
  "command": "C:\\Kellekek\\MICROS~1\\VISUAL~1\\Preview\\VC\\Tools\\MSVC\\1412~1.258\\bin\\Hostx64\\x64\\cl.exe  /nologo /TP  -IC:\\  /DWIN32 /D_WINDOWS /W3 /GR /EHsc /MDd /Zi /Ob0 /Od /RTC1   /FoCMakeFiles\\vscodecpp.dir\\Main.cpp.obj /FdTARGET_COMPILE_PDB /FS -c C:\\Users\\Matty\\Source\\vscodecpp\\Main.cpp",
  "file": "C:/Users/Matty/Source/vscodecpp/Main.cpp"
}
]

After trying to build, I looked into c_cpp_properties, but did not see, that the include paths would've been updated, even though the compile_comamnds.json had /IC:\\. Is the database file of the wrong format, or did I do something wrong? Also, does the feature populate launch.json when properly configured?

[jophippe]

@bbabiksc check out https://github.com/Microsoft/vscode-cpptools/blob/master/Documentation/LanguageServer/c_cpp_properties.json.md for documentation @MathiasMagnus That's because we don't update the includePath and defines properties when a compileCommands file is specified. Instead, we just ignore those properties entirely and use the ones found in the compileCommands file. Opening that Main.cpp file should result in no include errors. We currently don't use the compileCommands file to populate launch.json. I recommend filing an issue to ask for that as a feature request if you want it. Also, to everyone here, if you're having problems with using the compileCommands property, please open a separate issue instead of commenting on this one, so we can better track them. Thanks!

[teki]

The plugin works nicely, but it did find a 108048 files to parse, so it consumes a core constantly. I have around 13000 source files in my tree (cpp, c, h), not sure where the other 95000 are coming from. Is it possible that some recursive header searching goes wrong and tries to parse all the SDKs on my computer? (MacOS)

[sean-mcmanus]

@teki The number of files will include all files recursively in your browse.path setting, even files that are not actually parsed -- the filenames are simply added to the database in case it's #included in a file later (in which case it would be parsed). We're considering ways to report a better number in the future.

[teki]

@sean-mcmanus That makes sense, so I removed all of them. My theory was that I don't really need them anymore, because I have a compile_commands.json file. I reset the database, restarted VSCode, and intellisense is working (completion + type info on hover), but go to definition does not. Is this expected?

[jophippe]

@teki Our current implementation of go to definition and a few other features uses the browse paths and not the include paths from compile_commands.json, so you need to have both set up to get all the features to work.

[sean-mcmanus]

@teki If you only add your ${workspaceRoot} or whatever non-system headers to your browse.path, then you can see how approximately how many files will really be parsed and how long it takes to parse the 13k source files. Then you can add the system files and compare the behavior (it shouldn't be parsing very many of the system headers if your limitSymbolsToIncludedHeaders is set to true, and the time to add the files to the database is relatively low compared to parsing). You can also increase the C_Cpp.loggingLevel to Information to see if there are any files/directories being parsed that shouldn't be.

[teki]

@sean-mcmanus Thanks for the info! I checked it again and we do have a ton of headers in our tree :(. It is ~15k per platform and we have 5 platforms. So it does not matter that the compile_commands.json only references one platform a time, having the ${workspaceRoot} in browse.path will trigger of the parsing all of them. I will try to move the per platform dependecies out of the tree, that should help.

[teki]

A simple solution is to create separate configurations for the different platforms and be more specific in browse.path, ex for the ios platform:

                "path": [
                    "${workspaceRoot}/src",
                    "${workspaceRoot}/deps/ios"
                ],

[teki]

I have a new problem. Have a generated header file out of tree and can not make the IntelliSense find it. When I add the path of the include file to browse.path then it will find it, but it will still report it as missing and suggests to add the right path to includePath. But adding it there does not change the situation. It will still report it as missing and offers the right path to be added to includePath again.

[d-xo]

Our current implementation of go to definition and a few other features uses the browse paths and not the include paths from compile_commands.json, so you need to have both set up to get all the features to work.

@jophippe What was the reason for this decision?

[bobbrow]

@xwvvvvwx, it's because we haven't implemented goto definition/declaration using the new IntelliSense engine yet. It is still using the tag parser (which looks at the browse.path).

[sean-mcmanus]

@teki If you use multiple configurations with a different browse.path, you'll probably want to specify different databaseFilenames too. The compileCommands and includePath are not cumulative currently. Are you able to get the external file added to the compile commands or do you think we should combine the includePath with the compileCommands instead of ignoring the includePath?