jacobdufault / cquery

C/C++ language server supporting multi-million line code base, powered by libclang. Emacs, Vim, VSCode, and others with language server protocol support. Cross references, completion, diagnostics, semantic highlighting and more
MIT License
2.35k stars 163 forks source link

Audit code-base for working-file assumptions; client is not required to call didOpen #685

Open jpeach opened 6 years ago

jpeach commented 6 years ago

Reproduced with a debug build of aa74ae57.

I'm playing with some toy client code that calls the "textDocument/definition". Under most circumstances, query crashes when it attempts to handle my query. Here's the sequence of input requests:

Content-Length: 256
{"method":"initialize","params":{"processId":26035,"rootUri":"/Users/jpeach/src/mesos.git","initializationOptions":{"cacheDirectory":"/Users/jpeach/src/mesos.git/.cache"},"capabilities":{},"trace":"messages","workspaceFolders":null},"id":0,"jsonrpc":"2.0"}

Content-Length: 183
{"method":"textDocument/definition","params":{"textDocument":{"uri":"/Users/jpeach/src/mesos.git/src/master/master.cpp"},"position":{"line":94,"character":40}},"id":1,"jsonrpc":"2.0"}

Content-Length: 183
{"method":"textDocument/definition","params":{"textDocument":{"uri":"/Users/jpeach/src/mesos.git/src/master/master.cpp"},"position":{"line":94,"character":40}},"id":2,"jsonrpc":"2.0"}

Content-Length: 183
{"method":"textDocument/definition","params":{"textDocument":{"uri":"/Users/jpeach/src/mesos.git/src/master/master.cpp"},"position":{"line":94,"character":40}},"id":3,"jsonrpc":"2.0"}

Content-Length: 191
{"method":"textDocument/definition","params":{"textDocument":{"uri":"file:///Users/jpeach/src/mesos.git/src/master/master.cpp"},"position":{"line":439,"character":10}},"id":4,"jsonrpc":"2.0"}

Request ID 3 succeeds (it's a hard-coded test call), but request ID 4 crashes cquery.

(lldb) bt
* thread #1, name = 'querydb', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x20)
    frame #0: 0x00000001075aaf11 cquery`(anonymous namespace)::Handler_TextDocumentDefinition::Run((anonymous namespace)::In_TextDocumentDefinition*) [inlined] std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__is_long(this="") const at string:1221
    frame #1: 0x00000001075aaf11 cquery`(anonymous namespace)::Handler_TextDocumentDefinition::Run((anonymous namespace)::In_TextDocumentDefinition*) [inlined] std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__get_pointer(this="") const at string:1315
    frame #2: 0x00000001075aaf11 cquery`(anonymous namespace)::Handler_TextDocumentDefinition::Run((anonymous namespace)::In_TextDocumentDefinition*) [inlined] std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::data(this="") const at string:1129
    frame #3: 0x00000001075aaf11 cquery`(anonymous namespace)::Handler_TextDocumentDefinition::Run((anonymous namespace)::In_TextDocumentDefinition*) [inlined] std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::operator std::__1::basic_string_view<char, std::__1::char_traits<char> >(this="") const at string:816
  * frame #4: 0x00000001075aaf11 cquery`(anonymous namespace)::Handler_TextDocumentDefinition::Run(this=0x000000010794e508, request=0x00007f9756f09b30)::In_TextDocumentDefinition*) at text_document_definition.cc:121
    frame #5: 0x00000001075a9f84 cquery`BaseMessageHandler<(anonymous namespace)::In_TextDocumentDefinition>::Run(this=0x000000010794e508, message=<unavailable>) at message_handler.h:102
    frame #6: 0x0000000107361185 cquery`QueryDbMainLoop(db=0x00007ffee8976c30, project=0x00007ffee8977498, file_consumer_shared=0x00007ffee8977398, import_manager=0x00007ffee8976e80, status=0x00007ffee8976e70, timestamp_manager=0x00007ffee8976e08, semantic_cache=0x00007ffee8977468, working_files=0x00007ffee8977400, clang_complete=0x00007ffee89770a0, include_complete=0x00007ffee8976f28, global_code_complete_cache=0x00007f974ee00860, non_global_code_complete_cache=0x00007f974ee008f0, signature_cache=0x00007f974ee00980) at command_line.cc:189
    frame #7: 0x00000001073626b8 cquery`RunQueryDbThread(bin_name="/Users/jpeach/upstream/cquery/build/debug/bin/cquery") at command_line.cc:272
    frame #8: 0x00000001073636ca cquery`LanguageServerMain(bin_name="/Users/jpeach/upstream/cquery/build/debug/bin/cquery") at command_line.cc:388
    frame #9: 0x0000000107366869 cquery`main(argc=3, argv=0x00007ffee8979920, env=0x00007ffee8979940) at command_line.cc:515
    frame #10: 0x00007fff740f2015 libdyld.dylib`start + 1

The crash happens because working_file is NULL, and we dereference it here.

(lldb) p working_file
(WorkingFile *) $9 = 0x0000000000000000

I guess that working_file is NULL because working_files is empty:

(lldb) p *working_files
(WorkingFiles) $10 = {
  files = size=0 {}
  files_mutex = {
    __m_ = (__sig = 1297437786, __opaque = char [56] @ 0x00007f8bfff58a30)
  }
}

I'm not really sure why working_files would be empty depending on the requested position, but it seems like cquery should emit a diagnostic in this case.

Here's the startup log in case that provides any clues:

( uptime  ) [ thread name/id]              file:line  |
(   0.000s) [main thread  ]        loguru.hpp:1887  | arguments: /Users/jpeach/upstream/cquery/build/debug/bin/cquery --log-all-to-stderr --record=/tmp/cquery
(   0.000s) [main thread  ]        loguru.hpp:1890  | Current dir: /Users/jpeach/src/mesos.git
(   0.000s) [main thread  ]        loguru.hpp:1892  | stderr verbosity: 0
(   0.000s) [main thread  ]        loguru.hpp:1893  | -----------------------------------
(   0.002s) [stdin        ]            lsp.cc:25    | RecordPath: client=/Users/jpeach/src/mesos.git, normalized=/Users/jpeach/src/mesos.git
(   0.002s) [querydb      ]     initialize.cc:507   | Init parameters: {"compilationDatabaseCommand":"","compilationDatabaseDirectory":"","cacheDirectory":"/Users/jpeach/src/mesos.git/.cache","cacheFormat":"json","resourceDirectory":"","discoverSystemIncludes":true,"extraClangArguments":[],"progressReportFrequencyMs":500,"emitQueryDbBlocked":false,"showDocumentLinksOnIncludes":true,"codeLens":{"localVariables":true},"completion":{"enableSnippets":true,"detailedLabel":false,"dropOldRequests":true,"filterAndSort":true,"includeMaxPathSize":30,"includeSuffixWhitelist":[".h",".hpp",".hh"],"includeBlacklist":[],"includeWhitelist":[]},"diagnostics":{"blacklist":[],"whitelist":[],"frequencyMs":0,"onParse":true,"onType":true},"highlight":{"blacklist":[],"whitelist":[]},"index":{"attributeMakeCallsToCtor":true,"blacklist":[],"whitelist":[],"comments":2,"enabled":true,"logSkippedPaths":false,"threads":0},"workspaceSymbol":{"maxNum":1000,"sort":true},"xref":{"maxNum":2000},"enableIndexOnDidChange":false}
(   0.002s) [querydb      ]     initialize.cc:512   | [querydb] Initialize in directory /Users/jpeach/src/mesos.git with uri /Users/jpeach/src/mesos.git
(   0.002s) [querydb      ]     initialize.cc:561   | Using -resource-dir=/Users/jpeach/upstream/cquery/build/debug/lib/clang+llvm-6.0.0-x86_64-apple-darwin/lib/clang/6.0.0
(   0.002s) [stdout       ]          timer.cc:35    | [e2e] Running initialize took 0.482ms
>> (   0.003s) [querydb      ]        project.cc:470   | Using .cquery arguments from /Users/jpeach/src/mesos.git/.cquery
(   0.043s) [querydb      ]        project.cc:482   | Using .cquery arguments %clang, %cpp -std=c++11, -working-directory=., -I3rdparty/stout/include, -I3rdparty/libprocess/include, -Iinclude, -Isrc
(   0.057s) [querydb      ]clang_system_include_extractor.cc:76    | Using compiler drivers /Users/jpeach/upstream/cquery/build/debug/bin/cquery-clang, clang++, g++
(   0.059s) [querydb      ]clang_system_include_extractor.cc:88    | Running /Users/jpeach/upstream/cquery/build/debug/bin/cquery-clang -E -x c++ - -v
(   0.060s) [querydb      ]clang_system_include_extractor.cc:93    | Output:
(   0.059s) [querydb      ]        loguru.hpp:1767  | atexit
(   0.060s) [querydb      ]clang_system_include_extractor.cc:88    | Running clang++ -E -x c++ - -v -working-directory=/Users/jpeach/src/mesos.git/
(   0.081s) [querydb      ]clang_system_include_extractor.cc:93    | Output:
Apple LLVM version 9.1.0 (clang-902.0.39.1)
Target: x86_64-apple-darwin17.5.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1 -triple x86_64-apple-macosx10.13.0 -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -E -disable-free -disable-llvm-verifier -discard-value-names -main-file-name - -mrelocation-model pic -pic-level 2 -mthread-model posix -mdisable-fp-elim -fno-strict-return -masm-verbose -munwind-tables -target-cpu penryn -target-linker-version 351.8 -v -dwarf-column-info -debugger-tuning=lldb -resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/9.1.0 -working-directory /Users/jpeach/src/mesos.git/ -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk -I/usr/local/include -stdlib=libc++ -fdeprecated-macro -fdebug-compilation-dir /Users/jpeach/src/mesos.git -ferror-limit 19 -fmessage-length 0 -stack-protector 1 -fblocks -fobjc-runtime=macosx-10.13.0 -fencode-extended-block-signature -fcxx-exceptions -fexceptions -fmax-type-align=16 -fdiagnostics-show-option -o - -x c++ -
clang -cc1 version 9.1.0 (clang-902.0.39.1) default target x86_64-apple-darwin17.5.0
ignoring nonexistent directory "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include/c++/v1"
ignoring nonexistent directory "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/local/include"
ignoring nonexistent directory "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/Library/Frameworks"
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/9.1.0/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include
 /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/System/Library/Frameworks (framework directory)
End of search list.
# 1 "<stdin>"
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 353 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "<stdin>" 2
(   0.081s) [querydb      ]        project.cc:161   | Using system include directory flags
  -isystem/usr/local/include
  -isystem/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1
  -isystem/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/9.1.0/include
  -isystem/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
  -isystem/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include
  -iframework/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/System/Library/Frameworks
(   0.081s) [querydb      ]        project.cc:165   | To disable this set the discoverSystemIncludes config option to false.
(   0.149s) [querydb      ]        project.cc:710   | angle_include_dir: /Users/jpeach/src/mesos.git/src/
(   0.149s) [querydb      ]        project.cc:710   | angle_include_dir: /Users/jpeach/src/mesos.git/include/
(   0.149s) [querydb      ]        project.cc:710   | angle_include_dir: /Users/jpeach/src/mesos.git/3rdparty/libprocess/include/
(   0.149s) [querydb      ]        project.cc:710   | angle_include_dir: /Users/jpeach/src/mesos.git/3rdparty/stout/include/
(   0.150s) [querydb      ]          timer.cc:35    | [perf] Loaded compilation entries (592 files) took 147.147097ms
(   0.150s) [querydb      ]     initialize.cc:613   | Starting 6 indexers
...
jacobdufault commented 6 years ago

Thanks, should be fixed in 94a1a5c9d1911e919da3f5ab085fa3b938854505.

jpeach commented 6 years ago

@jacobdufault I still reproduce the same crash after this commit:

(lldb) down
frame #4: 0x000000010f738151 cquery`(anonymous namespace)::Handler_TextDocumentDefinition::Run(this=0x000000010fae2758, request=0x00007fdc4441d3a0)::In_TextDocumentDefinition*) at text_document_definition.cc:121
   118        if (!has_symbol) {
   119          lsPosition position = request->params.position;
   120          const std::string& buffer = working_file->buffer_content;
-> 121          std::string_view query = LexIdentifierAroundPos(position, buffer);
   122          std::string_view short_query = query;
   123          {
   124            auto pos = query.rfind(':');
(lldb) p working_file
(WorkingFile *) $2 = 0x0000000000000000
(lldb) p *request
((anonymous namespace)::In_TextDocumentDefinition) $4 = {
  RequestInMessage = {
    id = (type = kInt, value = 8)
  }
  params = {
    textDocument = {
      uri = (raw_uri_ = "file:///Users/jpeach/src/mesos.git/src/master/master.cpp")
    }
    position = (line = 40, character = 40)
  }
}
(lldb) p *working_files
(WorkingFiles) $5 = {
  files = size=0 {}
  files_mutex = {
    __m_ = (__sig = 1297437786, __opaque = char [56] @ 0x00007fc950fbb050)
  }
}

I don't think it is a race condition, since it seems related to the Position I send in the "textDocument/definition" request.

This request is OK:

Content-Length: 192

{"method":"textDocument/definition","params":{"textDocument":{"uri":"file:///Users/jpeach/src/mesos.git/src/master/master.cpp"},"position":{"line":400,"character":40}},"id":12,"jsonrpc":"2.0"}

This request crashes cquery:

Content-Length: 191

{"method":"textDocument/definition","params":{"textDocument":{"uri":"file:///Users/jpeach/src/mesos.git/src/master/master.cpp"},"position":{"line":439,"character":9}},"id":16,"jsonrpc":"2.0"}
jpeach commented 6 years ago

Debugged some more. The reason some Positions don't crash is because, has_symbol in Handler_TextDocumentDefinition::Run ends up being true (the Position resolves as a header file). The reason that there are no files in WorkingFiles is that my client code never sends the "textDocument/didOpen" message.

jacobdufault commented 6 years ago

Yea, I think cquery should emit an error if the working file fails to resolve, since it is a client error. Leaving open as a TODO for that

jpeach commented 6 years ago

On May 24, 2018, at 8:55 PM, Jacob Dufault notifications@github.com wrote:

Yea, I think cquery should emit an error if the working file fails to resolve, since it is a client error. Leaving open as a TODO for that

Yes, that behavior makes sense to me. Is this really a client error though? I didn't see anything in the LSP spec that requires a client to sent an open notification for a file that it wants to query.

jacobdufault commented 6 years ago

I suppose you are right, didOpen/didClose is more about where the document content source of truth is.