search

microsoft / vscode-cpptools

Official repository for the Microsoft C/C++ extension for VS Code.
Other
5.53k stars 1.56k forks source link

Extension locks up when closing files on FUSE #10479

Open bobbrow opened 1 year ago

bobbrow commented 1 year ago 
          @sean-mcmanus (picking up from @jsimonlane's report with the same behavior)

C/C++ Extension v1.11.5 C/C++ Extension Pack v1.2.0 Linux (remote VS Code over SSH from a Windows VS Code instance)

The Debug logs don't seem to show anything particularly interesting, just regular logs when jumping to token definitions:

idle loop: reparsing the active document
Checking for syntax errors: /home/justinw/path/to/sources/sor_main.cc
Queueing IntelliSense update for files in translation unit of: /home/justinw/path/to/sources/sor_main.cc
textDocument/hover: /home/justinw/path/to/sources/sor_main.cc (id: 789)
Request canceled: 789
cpptools/finishUpdateSquiggles
Error squiggle count: 0
textDocument/definition: /home/justinw/path/to/sources/sor_main.cc (id: 790)
cpptools/getSemanticTokens: /home/justinw/path/to/sources/sor_main.cc (id: 791)
cpptools/getFoldingRanges: /home/justinw/path/to/sources/sor_main.cc (id: 792)
cpptools/getCodeActions: path/to/dependencies/dist/include/tslog.h (id: 793)
textDocument/didOpen: path/to/dependencies/dist/include/tslog.h

Here's the ps aux entry for a spinning cpptools-srv process:

justinw  1030006 75.6  0.7 5229968 432940 ?      Sl   18:35  11:40 /home/justinw/.vscode-server/extensions/ms-vscode.cpptools-1.11.5-linux-x64/bin/cpptools-srv 872753 {D26E0D8F-EB4F-44AD-9FDB-39007E7D5A24}

And here's a call stack for that cpptools-srv proccess, assuming I pulled it correctly:


Thread 35 (LWP 1040762):
#0  0x00000000007ec865 in void std::__introsort_loop<__gnu_cxx::__normal_iterator<char const**, std::vector<char const*, std::allocator<char const*> > >, long, __gnu_cxx::__ops::_Iter_less_iter>(__gnu_cxx::__normal_iterator<char const**, std::vector<char const*, std::allocator<char const*> > >, __gnu_cxx::__normal_iterator<char const**, std::vector<char const*, std::allocator<char const*> > >, long, __gnu_cxx::__ops::_Iter_less_iter) ()
#1  0x0000000000775287 in sqlite3Insert ()
#2  0x00007fe878243318 in ?? ()
#3  0x0000000000000000 in ?? ()

Thread 13 (LWP 1030018):
#0  0x00007fe879c4929c in ?? ()
#1  0x0000000000000000 in ?? ()

Thread 12 (LWP 1030017):
#0  0x00007fe879c4600c in ?? ()
#1  0x00007fe84401a350 in ?? ()
#2  0x0000000000000000 in ?? ()

Thread 11 (LWP 1030016):
#0  0x00007fe879c4600c in ?? ()
#1  0x0000000000000014 in ?? ()
#2  0x000000000000005a in ?? ()
#3  0x00007fe870ff8e20 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 10 (LWP 1030015):
#0  0x00007fe879c4600c in ?? ()
#1  0x0000000000000014 in ?? ()
#2  0x0000000000000059 in ?? ()
#3  0x00007fe8717f9e20 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 9 (LWP 1030014):
#0  0x00007fe879b7121f in ?? ()
#1  0x00000000ffffffff in ?? ()
#2  0x0000000000000000 in ?? ()

Thread 8 (LWP 1030013):
#0  0x00007fe879c4635b in ?? ()
#1  0x000000000000077c in ?? ()
#2  0x0000000000000000 in ?? ()

Thread 7 (LWP 1030012):
#0  0x00007fe879c4600c in ?? ()
#1  0x0000000000000003 in ?? ()
#2  0x0000000000000002 in ?? ()
#3  0x00007fe872ffccf0 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 6 (LWP 1030011):
#0  0x00007fe879c4600c in ?? ()
#1  0x0000000000000003 in ?? ()
#2  0x0000000000000005 in ?? ()
#3  0x00007fe8737fdcf0 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 5 (LWP 1030010):
#0  0x00007fe879c4600c in ?? ()
#1  0x0000000000000003 in ?? ()
#2  0x0000000000000006 in ?? ()
#3  0x00007fe873ffecf0 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 4 (LWP 1030009):
#0  0x00007fe879c4600c in ?? ()
#1  0x0000000000000003 in ?? ()
#2  0x0000000000000003 in ?? ()
#3  0x00007fe878a46cf0 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 3 (LWP 1030008):
#0  0x00007fe879c4600c in ?? ()
#1  0x0000000000000003 in ?? ()
#2  0x0000000000000001 in ?? ()
#3  0x00007fe879247cf0 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 2 (LWP 1030007):
#0  0x00007fe879c4600c in ?? ()
#1  0x0000000000000003 in ?? ()
#2  0x0000000000000000 in ?? ()

Thread 1 (LWP 1030006):
#0  0x00007fe879c4635b in ?? ()
#1  0x00000000000003bc in ?? ()
#2  0x0000000000000000 in ?? ()

Originally posted by @justin39 in https://github.com/microsoft/vscode-cpptools/issues/9774#issuecomment-1230721637

bobbrow commented 1 year ago

@justin39 The earlier post said cpptools is spinning, but your post says cpptools-srv has the issue -- which process is it? Your logging indicates a file was opened and then it seems like it's stuck with that in cpptools and there is no "sending compilation args for" which would indicate a cpptools-srv was launched for it. Your call stack is invalid. You have set your "program" correctly in the launch.json -- it appears to not be pulling in the correct symbols...and it seems like you should be attaching to cpptools instead. Also, the version of VS Code like 1.70.1 shouldn't matter. You may want to try 1.12.2 (pre-release) which is planning to ship as 1.12.3 (release) tomorrow.

Originally posted by @sean-mcmanus

bobbrow commented 1 year ago

@sean-mcmanus cpptools-srv is the culprit as far as high CPU usage goes - not 100% clear on the relationship between the two processes, though.

there is no "sending compilation args for"

That appeared higher up in the logs, my bad! Should have captured that.

I can try attaching to cpptools instead - will try that shortly.

Originally posted by @justin39

bobbrow commented 1 year ago 

Thread 25 (Thread 0x7fa2de7fc700 (LWP 932576)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7ffc5c7b3e44) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7ffc5c7b3df0, cond=0x7ffc5c7b3e18) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7ffc5c7b3e18, mutex=0x7ffc5c7b3df0) at pthread_cond_wait.c:655
#3  0x0000000000e9eb9c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#4  0x000000000057a755 in vscode::message_deque<int, false>::pop_impl(bool) ()
#5  0x0000000000540a44 in std::_Function_handler<void (), vscode::message_handler::main_loop()::$_7>::_M_invoke(std::_Any_data const&) ()
#6  0x0000000000e6dc9a in msvc::thread_helper_t::thread_entry(void*) ()
#7  0x00007fa30b580fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8  0x00007fa30b4b1eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 24 (Thread 0x7fa2deffd700 (LWP 932575)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7ffc5c7b440c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x3259310, cond=0x7ffc5c7b43e0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7ffc5c7b43e0, mutex=0x3259310) at pthread_cond_wait.c:655
#3  0x0000000000e9eb9c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#4  0x00000000004bd888 in void std::_V2::condition_variable_any::wait<std::unique_lock<std::mutex> >(std::unique_lock<std::mutex>&) ()
#5  0x000000000057c07b in msvc::event_t::wait_and_reset() ()
#6  0x000000000054163b in std::_Function_handler<void (), vscode::message_handler::main_loop()::$_14>::_M_invoke(std::_Any_data const&) ()
#7  0x0000000000e6dc9a in msvc::thread_helper_t::thread_entry(void*) ()
#8  0x00007fa30b580fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9  0x00007fa30b4b1eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 23 (Thread 0x7fa2df7fe700 (LWP 932574)):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7fa2df7fdde8, expected=0, futex_word=0x7ffc5c7b414c) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  __pthread_cond_wait_common (abstime=0x7fa2df7fdde8, mutex=0x7ffc5c7b40f8, cond=0x7ffc5c7b4120) at pthread_cond_wait.c:539
#2  __pthread_cond_timedwait (cond=0x7ffc5c7b4120, mutex=0x7ffc5c7b40f8, abstime=0x7fa2df7fdde8) at pthread_cond_wait.c:667
#3  0x000000000057bfe2 in bool std::condition_variable::wait_for<long, std::ratio<1l, 1000l>, vscode::message_deque<int, false>::pop_impl(bool, int&, std::chrono::duration<long, std::ratio<1l, 1000l> >)::{lambda()#1}>(std::unique_lock<std::mutex>&, std::chrono::duration<long, std::ratio<1l, 1000l> > const&, vscode::message_deque<int, false>::pop_impl(bool, int&, std::chrono::duration<long, std::ratio<1l, 1000l> >)::{lambda()#1}) ()
#4  0x000000000057bcd8 in vscode::message_deque<int, false>::pop_impl(bool, int&, std::chrono::duration<long, std::ratio<1l, 1000l> >) ()
#5  0x0000000000541542 in std::_Function_handler<void (), vscode::message_handler::main_loop()::$_13>::_M_invoke(std::_Any_data const&) ()
#6  0x0000000000e6dc9a in msvc::thread_helper_t::thread_entry(void*) ()
#7  0x00007fa30b580fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8  0x00007fa30b4b1eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 22 (Thread 0x7fa2ddffb700 (LWP 932573)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7ffc5c7b408c) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7ffc5c7b4038, cond=0x7ffc5c7b4060) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7ffc5c7b4060, mutex=0x7ffc5c7b4038) at pthread_cond_wait.c:655
#3  0x0000000000e9eb9c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#4  0x000000000057b225 in vscode::message_deque<vscode::vscode_client_message, false>::pop_impl(bool) ()
#5  0x00000000005413fc in std::_Function_handler<void (), vscode::message_handler::main_loop()::$_12>::_M_invoke(std::_Any_data const&) ()
#6  0x0000000000e6dc9a in msvc::thread_helper_t::thread_entry(void*) ()
#7  0x00007fa30b580fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8  0x00007fa30b4b1eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 21 (Thread 0x7fa2dd7fa700 (LWP 932572)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7ffc5c7b3fc8) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7ffc5c7b3f78, cond=0x7ffc5c7b3fa0) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7ffc5c7b3fa0, mutex=0x7ffc5c7b3f78) at pthread_cond_wait.c:655
#3  0x0000000000e9eb9c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#4  0x000000000057b485 in vscode::message_deque<vscode::message_handler::parse_file_entry, true>::pop_impl(bool) ()
#5  0x0000000000540e79 in std::_Function_handler<void (), vscode::message_handler::main_loop()::$_11>::_M_invoke(std::_Any_data const&) ()
#6  0x0000000000e6dc9a in msvc::thread_helper_t::thread_entry(void*) ()
#7  0x00007fa30b580fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8  0x00007fa30b4b1eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 20 (Thread 0x7fa2dcff9700 (LWP 932571)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7ffc5c7b3cc4) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7ffc5c7b3c70, cond=0x7ffc5c7b3c98) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7ffc5c7b3c98, mutex=0x7ffc5c7b3c70) at pthread_cond_wait.c:655
#3  0x0000000000e9eb9c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#4  0x000000000057b225 in vscode::message_deque<vscode::vscode_client_message, false>::pop_impl(bool) ()
#5  0x0000000000540d87 in std::_Function_handler<void (), vscode::message_handler::main_loop()::$_10>::_M_invoke(std::_Any_data const&) ()
#6  0x0000000000e6dc9a in msvc::thread_helper_t::thread_entry(void*) ()
#7  0x00007fa30b580fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8  0x00007fa30b4b1eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 19 (Thread 0x7fa2dffff700 (LWP 932570)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7ffc5c7b3c04) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7ffc5c7b3bb0, cond=0x7ffc5c7b3bd8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7ffc5c7b3bd8, mutex=0x7ffc5c7b3bb0) at pthread_cond_wait.c:655
#3  0x0000000000e9eb9c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#4  0x000000000057b225 in vscode::message_deque<vscode::vscode_client_message, false>::pop_impl(bool) ()
#5  0x0000000000540cb7 in std::_Function_handler<void (), vscode::message_handler::main_loop()::$_9>::_M_invoke(std::_Any_data const&) ()
#6  0x0000000000e6dc9a in msvc::thread_helper_t::thread_entry(void*) ()
#7  0x00007fa30b580fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8  0x00007fa30b4b1eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 18 (Thread 0x7fa2d67fc700 (LWP 872832)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x16f76b0 <cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x16f7660 <mutex>, cond=0x16f7688 <cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x16f7688 <cond>, mutex=0x16f7660 <mutex>) at pthread_cond_wait.c:655
#3  0x0000000000e88276 in uv_cond_wait ()
#4  0x0000000000e7ace6 in worker ()
#5  0x00007fa30b580fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007fa30b4b1eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 17 (Thread 0x7fa2d6ffd700 (LWP 872831)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x16f76b0 <cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x16f7660 <mutex>, cond=0x16f7688 <cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x16f7688 <cond>, mutex=0x16f7660 <mutex>) at pthread_cond_wait.c:655
#3  0x0000000000e88276 in uv_cond_wait ()
#4  0x0000000000e7ace6 in worker ()
#5  0x00007fa30b580fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007fa30b4b1eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 16 (Thread 0x7fa2d77fe700 (LWP 872830)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x16f76b0 <cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x16f7660 <mutex>, cond=0x16f7688 <cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x16f7688 <cond>, mutex=0x16f7660 <mutex>) at pthread_cond_wait.c:655
#3  0x0000000000e88276 in uv_cond_wait ()
#4  0x0000000000e7ace6 in worker ()
#5  0x00007fa30b580fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007fa30b4b1eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 15 (Thread 0x7fa2d7fff700 (LWP 872829)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x16f76b0 <cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x16f7660 <mutex>, cond=0x16f7688 <cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x16f7688 <cond>, mutex=0x16f7660 <mutex>) at pthread_cond_wait.c:655
#3  0x0000000000e88276 in uv_cond_wait ()
#4  0x0000000000e7ace6 in worker ()
#5  0x00007fa30b580fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007fa30b4b1eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 14 (Thread 0x7fa300ff9700 (LWP 872766)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7ffc5c7b3d80) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7ffc5c7b3d30, cond=0x7ffc5c7b3d58) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7ffc5c7b3d58, mutex=0x7ffc5c7b3d30) at pthread_cond_wait.c:655
#3  0x0000000000e9eb9c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#4  0x000000000057a1d5 in vscode::message_deque<vscode::browse_engine_update_action, false>::pop_impl(bool) ()
#5  0x00000000005404a9 in std::_Function_handler<void (), vscode::message_handler::main_loop()::$_6>::_M_invoke(std::_Any_data const&) ()
#6  0x0000000000e6dc9a in msvc::thread_helper_t::thread_entry(void*) ()
#7  0x00007fa30b580fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8  0x00007fa30b4b1eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 13 (Thread 0x7fa3017fa700 (LWP 872765)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7ffc5c7b3f04) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7ffc5c7b3eb0, cond=0x7ffc5c7b3ed8) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7ffc5c7b3ed8, mutex=0x7ffc5c7b3eb0) at pthread_cond_wait.c:655
#3  0x0000000000e9eb9c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#4  0x000000000057aa27 in vscode::message_deque<vscode::folding_ranges_params, false>::pop_impl(bool) ()
#5  0x0000000000540aef in std::_Function_handler<void (), vscode::message_handler::main_loop()::$_8>::_M_invoke(std::_Any_data const&) ()
#6  0x0000000000e6dc9a in msvc::thread_helper_t::thread_entry(void*) ()
#7  0x00007fa30b580fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8  0x00007fa30b4b1eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 12 (Thread 0x7fa301ffb700 (LWP 872764)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7ffc5c7b3b40) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7ffc5c7b3af0, cond=0x7ffc5c7b3b18) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7ffc5c7b3b18, mutex=0x7ffc5c7b3af0) at pthread_cond_wait.c:655
#3  0x0000000000e9eb9c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#4  0x0000000000579bf5 in vscode::message_deque<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, false>::pop_impl(bool) ()
#5  0x00000000005403cd in msvc::thread_t::invoker_t<vscode::message_handler::main_loop()::$_1>::invoke() ()
#6  0x0000000000e6dc9a in msvc::thread_helper_t::thread_entry(void*) ()
#7  0x00007fa30b580fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8  0x00007fa30b4b1eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 11 (Thread 0x7fa3027fc700 (LWP 872763)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7fa3027fa930) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa2e57fa200, cond=0x7fa3027fa908) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fa3027fa908, mutex=0x7fa2e57fa200) at pthread_cond_wait.c:655
#3  0x0000000000e9eb9c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#4  0x00000000004bd888 in void std::_V2::condition_variable_any::wait<std::unique_lock<std::mutex> >(std::unique_lock<std::mutex>&) ()
#5  0x000000000055329b in msvc::event_t::wait() ()
#6  0x0000000000e3dc49 in msvc::common_channel_t::wait_for_response(msvc::event_t&) ()
#7  0x0000000000e3d539 in msvc::common_channel_t::send_message(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) ()
#8  0x00000000005bb6d6 in cpp_intellisense::server_proxy::handle_contains(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool&) ()
#9  0x00000000005c8182 in intellisense_client::contains_source(char const*) ()
#10 0x00000000005e9541 in intellisense_client_factory::try_to_fetch(char const*, bool, bool) ()
#11 0x00000000004d9d02 in (anonymous namespace)::includes_for_file(vscode::file_uri const&) ()
#12 0x00000000004db38e in vscode::include_auto_complete::cache::populate(vscode::file_uri const&) ()
#13 0x00000000004f98f1 in vscode::message_handler::textDocument_didOpen(vscode::DidOpenTextDocumentParams) ()
#14 0x00000000004e7c41 in vscode::message_handler::dispatch(vscode::vscode_client_message&&, vscode::vscode_server_message&, vscode::message_handler::msg_proc_thread_token) ()
#15 0x00000000004e6b81 in vscode::message_handler::handle_message(vscode::vscode_client_message&&, vscode::message_handler::msg_proc_thread_token) ()
#16 0x000000000053fa7e in msvc::thread_t::invoker_t<vscode::message_handler::main_loop()::$_0>::invoke() ()
#17 0x0000000000e6dc9a in msvc::thread_helper_t::thread_entry(void*) ()
#18 0x00007fa30b580fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#19 0x00007fa30b4b1eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7fa302ffd700 (LWP 872762)):
#0  0x00007fa30b4b221f in epoll_wait (epfd=9, events=0x7fa302ff9dd0, maxevents=1024, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1  0x0000000000e7f7ca in uv.io_poll ()
#2  0x0000000000e7da68 in uv_run ()
#3  0x0000000000e46a0b in msvc::loop_t::run_loop() ()
#4  0x0000000000e4c295 in void std::__invoke_impl<void, void (*)(std::shared_ptr<msvc::loop_t>), std::shared_ptr<msvc::loop_t> >(std::__invoke_other, void (*&&)(std::shared_ptr<msvc::loop_t>), std::shared_ptr<msvc::loop_t>&&) ()
#5  0x0000000000e6dc9a in msvc::thread_helper_t::thread_entry(void*) ()
#6  0x00007fa30b580fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007fa30b4b1eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7fa3037fe700 (LWP 872761)):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7fa3037fdde0, expected=0, futex_word=0x14baf68 <vscode::g_thread_pool+304>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  __pthread_cond_wait_common (abstime=0x7fa3037fdde0, mutex=0x323c110, cond=0x14baf40 <vscode::g_thread_pool+264>) at pthread_cond_wait.c:539
#2  __pthread_cond_timedwait (cond=0x14baf40 <vscode::g_thread_pool+264>, mutex=0x323c110, abstime=0x7fa3037fdde0) at pthread_cond_wait.c:667
#3  0x00000000004bdcdd in std::cv_status std::_V2::condition_variable_any::wait_until<std::unique_lock<std::mutex>, std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::unique_lock<std::mutex>&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#4  0x0000000000625106 in unsigned int msvc::bitset_event_t::wait_for_any_set<long, std::ratio<1l, 1000l> >(std::chrono::duration<long, std::ratio<1l, 1000l> > const&, unsigned int) ()
#5  0x0000000000624190 in msvc::thread_t::invoker_t<vscode::thread_pool::thread_pool()::$_0>::invoke() ()
#6  0x0000000000e6dc9a in msvc::thread_helper_t::thread_entry(void*) ()
#7  0x00007fa30b580fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8  0x00007fa30b4b1eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7fa303fff700 (LWP 872760)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x14baf68 <vscode::g_thread_pool+304>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x323c110, cond=0x14baf40 <vscode::g_thread_pool+264>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x14baf40 <vscode::g_thread_pool+264>, mutex=0x323c110) at pthread_cond_wait.c:655
#3  0x0000000000e9eb9c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#4  0x00000000004bd888 in void std::_V2::condition_variable_any::wait<std::unique_lock<std::mutex> >(std::unique_lock<std::mutex>&) ()
#5  0x0000000000624753 in msvc::bitset_event_t::wait_any_set(unsigned int) ()
#6  0x0000000000623015 in vscode::thread_pool::do_work(unsigned long) ()
#7  0x0000000000e6dc9a in msvc::thread_helper_t::thread_entry(void*) ()
#8  0x00007fa30b580fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9  0x00007fa30b4b1eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7fa308b85700 (LWP 872759)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x14baf68 <vscode::g_thread_pool+304>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x323c110, cond=0x14baf40 <vscode::g_thread_pool+264>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x14baf40 <vscode::g_thread_pool+264>, mutex=0x323c110) at pthread_cond_wait.c:655
#3  0x0000000000e9eb9c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#4  0x00000000004bd888 in void std::_V2::condition_variable_any::wait<std::unique_lock<std::mutex> >(std::unique_lock<std::mutex>&) ()
#5  0x0000000000624753 in msvc::bitset_event_t::wait_any_set(unsigned int) ()
#6  0x0000000000623015 in vscode::thread_pool::do_work(unsigned long) ()
#7  0x0000000000e6dc9a in msvc::thread_helper_t::thread_entry(void*) ()
#8  0x00007fa30b580fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9  0x00007fa30b4b1eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7fa309386700 (LWP 872758)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x14baf68 <vscode::g_thread_pool+304>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x323c110, cond=0x14baf40 <vscode::g_thread_pool+264>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x14baf40 <vscode::g_thread_pool+264>, mutex=0x323c110) at pthread_cond_wait.c:655
#3  0x0000000000e9eb9c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#4  0x00000000004bd888 in void std::_V2::condition_variable_any::wait<std::unique_lock<std::mutex> >(std::unique_lock<std::mutex>&) ()
#5  0x0000000000624753 in msvc::bitset_event_t::wait_any_set(unsigned int) ()
#6  0x0000000000623015 in vscode::thread_pool::do_work(unsigned long) ()
#7  0x0000000000e6dc9a in msvc::thread_helper_t::thread_entry(void*) ()
#8  0x00007fa30b580fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9  0x00007fa30b4b1eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7fa309b87700 (LWP 872757)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7fa309b86b50) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7fa2e57e7a70, cond=0x7fa309b86b28) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7fa309b86b28, mutex=0x7fa2e57e7a70) at pthread_cond_wait.c:655
#3  0x0000000000e9eb9c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#4  0x00000000004bd888 in void std::_V2::condition_variable_any::wait<std::unique_lock<std::mutex> >(std::unique_lock<std::mutex>&) ()
#5  0x000000000055329b in msvc::event_t::wait() ()
#6  0x0000000000e3dc49 in msvc::common_channel_t::wait_for_response(msvc::event_t&) ()
#7  0x0000000000e3d539 in msvc::common_channel_t::send_message(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) ()
#8  0x00000000005bc395 in cpp_intellisense::server_proxy::handle_update_intellisense(cpp_intellisense::UpdateIntelliSenseRequest const&, cpp_intellisense::UpdateIntelliSenseResponse&) ()
#9  0x00000000005cdd00 in intellisense_client::update_intellisense(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, bool, reparse, update_intellisense_options) ()
#10 0x00000000005cfb2e in std::_Function_handler<cpp_intellisense::UpdateIntelliSenseResponse (), intellisense_client::queue_update_intellisense(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, bool, reparse, update_intellisense_options, std::function<void (cpp_intellisense::UpdateIntelliSenseResponse&&)>&&, std::function<void ()>&&)::$_1>::_M_invoke(std::_Any_data const&) ()
#11 0x00000000005e2ae0 in version_work_queue<cpp_intellisense::UpdateIntelliSenseResponse, 250>::worker_proc() ()
#12 0x00000000006242b2 in std::_Function_handler<void (), vscode::thread_pool::enqueue(std::function<void ()>&&, std::future<void>*)::$_2>::_M_invoke(std::_Any_data const&) ()
#13 0x00000000006230fd in vscode::thread_pool::do_work(unsigned long) ()
#14 0x0000000000e6dc9a in msvc::thread_helper_t::thread_entry(void*) ()
#15 0x00007fa30b580fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#16 0x00007fa30b4b1eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7fa30a388700 (LWP 872756)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x14baf68 <vscode::g_thread_pool+304>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x323c110, cond=0x14baf40 <vscode::g_thread_pool+264>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x14baf40 <vscode::g_thread_pool+264>, mutex=0x323c110) at pthread_cond_wait.c:655
#3  0x0000000000e9eb9c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#4  0x00000000004bd888 in void std::_V2::condition_variable_any::wait<std::unique_lock<std::mutex> >(std::unique_lock<std::mutex>&) ()
#5  0x0000000000624753 in msvc::bitset_event_t::wait_any_set(unsigned int) ()
#6  0x0000000000623015 in vscode::thread_pool::do_work(unsigned long) ()
#7  0x0000000000e6dc9a in msvc::thread_helper_t::thread_entry(void*) ()
#8  0x00007fa30b580fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9  0x00007fa30b4b1eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7fa30ab89700 (LWP 872755)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x14baf68 <vscode::g_thread_pool+304>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x323c110, cond=0x14baf40 <vscode::g_thread_pool+264>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x14baf40 <vscode::g_thread_pool+264>, mutex=0x323c110) at pthread_cond_wait.c:655
#3  0x0000000000e9eb9c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#4  0x00000000004bd888 in void std::_V2::condition_variable_any::wait<std::unique_lock<std::mutex> >(std::unique_lock<std::mutex>&) ()
#5  0x0000000000624753 in msvc::bitset_event_t::wait_any_set(unsigned int) ()
#6  0x0000000000623015 in vscode::thread_pool::do_work(unsigned long) ()
#7  0x0000000000e6dc9a in msvc::thread_helper_t::thread_entry(void*) ()
#8  0x00007fa30b580fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9  0x00007fa30b4b1eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7fa30b38a700 (LWP 872754)):
#0  0x00007fa30b4b221f in epoll_wait (epfd=3, events=0x7fa30b386dd0, maxevents=1024, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1  0x0000000000e7f7ca in uv.io_poll ()
#2  0x0000000000e7da68 in uv_run ()
#3  0x0000000000e46a0b in msvc::loop_t::run_loop() ()
#4  0x0000000000e4c295 in void std::__invoke_impl<void, void (*)(std::shared_ptr<msvc::loop_t>), std::shared_ptr<msvc::loop_t> >(std::__invoke_other, void (*&&)(std::shared_ptr<msvc::loop_t>), std::shared_ptr<msvc::loop_t>&&) ()
#5  0x0000000000e6dc9a in msvc::thread_helper_t::thread_entry(void*) ()
#6  0x00007fa30b580fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007fa30b4b1eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7fa30b38b780 (LWP 872753)):
#0  __GI___libc_read (nbytes=4096, buf=0x325aa80, fd=0) at ../sysdeps/unix/sysv/linux/read.c:26
#1  __GI___libc_read (fd=0, buf=0x325aa80, nbytes=4096) at ../sysdeps/unix/sysv/linux/read.c:24
#2  0x00007fa30b435680 in _IO_new_file_underflow (fp=0x7fa30b573a00 <_IO_2_1_stdin_>) at libioP.h:839
#3  0x00007fa30b4367c2 in __GI__IO_default_uflow (fp=0x7fa30b573a00 <_IO_2_1_stdin_>) at libioP.h:839
#4  0x00007fa30b430e78 in _IO_getc (fp=0x7fa30b573a00 <_IO_2_1_stdin_>) at getc.c:38
#5  0x0000000000ed096d in __gnu_cxx::stdio_sync_filebuf<char, std::char_traits<char> >::underflow() ()
#6  0x0000000000e93aea in std::basic_istream<char, std::char_traits<char> >& std::getline<char, std::char_traits<char>, std::allocator<char> >(std::basic_istream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char) ()
#7  0x00000000004e472c in vscode::message_handler::main_loop() ()
#8  0x00000000004df984 in main ()

Originally posted by @justin39

bobbrow commented 1 year ago

Our main thread is blocked in the didOpen waiting for a response from cpptools-srv for handle_contains. Another thread is handling an IntelliSense update (it may also be blocked on something). Are you able to repro the bug with only 1 cpp file and get a call stack for the cpptools-srv that is blocked so we can see what it is blocked on? In the past we've had deadlock bugs in which cpptools-srv would be stuck waiting for cpptools to process something (e.g. #1777 (comment) ), but it's also possible our parser is stuck in some parsing loop.

Originally posted by @sean-mcmanus

bobbrow commented 1 year ago

Are you able to repro the bug with only 1 cpp file

Actually, I don't think we've been able to repro this with only 1 cpp file open - it only seems to manifest with 2+ cpp files open at a time. Not very scientific testing, but at least in our small sample size having 2+ files open seemed to be the most reliable way to trigger the hang.

Originally posted by @justin39

bobbrow commented 1 year ago

@justin39 Yeah, a repro with more files and cpptools-srv is okay, but then you'd need to somehow identify which process to attach to to get the call stack for the right process...you could probably tell if you've attached to the correct process because you'd expect it to be doing some work related to responding to a request from cpptools. I'm not sure yet if we'll be able to get a repro ourselves, i.e. I don't believe we've noticed this internally.

Originally posted by @sean-mcmanus

bobbrow commented 1 year ago

This should be a thread dump for a hung cpptools-srv process:

Thread 15 (Thread 0x7f16cc4e4700 (LWP 907452)):
#0  0x000000000085d5aa in find_symbol(char const*, unsigned long, a_symbol_locator*) ()
#1  0x000000000086a989 in find_symbol_header(char const*, unsigned long, a_symbol_locator*) ()
#2  0x00000000007b7274 in get_token() ()
#3  0x00000000007ba7fd in cache_token_stream_until_matching_token(a_token_cache*, unsigned int, unsigned char*) ()
#4  0x00000000007c0818 in cache_token_stream_full(a_token_cache*, unsigned char*, unsigned int) ()
#5  0x00000000007d6a46 in cache_function_body(a_token_cache*, int, int*, unsigned int*, unsigned int*, a_source_position*, a_source_position*) ()
#6  0x00000000008a1e4b in complete_function_template_decl(a_tmpl_decl_state*, a_symbol*, a_func_info_block*, a_template_symbol_supplement**, a_source_position*) ()
#7  0x000000000089aee3 in template_declaration(a_tmpl_decl_state*) ()
#8  0x00000000008951e0 in template_or_specialization_declaration_full(a_tmpl_decl_state*, int, a_decl_parse_state*) ()
#9  0x000000000089b44a in template_or_specialization_declaration(a_token_kind_tag*, int, a_source_position*, int, a_decl_parse_state*) ()
#10 0x00000000008a0706 in template_directive_or_declaration(a_token_kind_tag*, int, a_source_position*) ()
#11 0x00000000005d1cc8 in scan_nonmember_declaration(a_decl_parse_state*, a_source_range*) ()
#12 0x00000000005da7fa in namespace_declaration(a_token_kind_tag*, int, int, a_source_position*, a_symbol**) ()
#13 0x00000000005d1f5c in scan_nonmember_declaration(a_decl_parse_state*, a_source_range*) ()
#14 0x00000000005d8548 in translation_unit() ()
#15 0x00000000008bf3a4 in process_translation_unit(char const*, int, an_exported_template_file*) ()
#16 0x0000000000530047 in cfe_main(int, char**) ()
#17 0x00000000008e3c86 in cfe_main_exception_handler(int, char**) ()
#18 0x0000000000530156 in edg_main(int, char**) ()
#19 0x00000000008dcf9c in edge_compiler_main(int, char const**) ()
#20 0x00000000004b9e96 in preparse(int, char const**, a_scout_store*, edge::translation_unit*) ()
#21 0x000000000047e95f in a_compiler_thread::compiler_thread_routine(a_compiler_thread*) ()
#22 0x00000000009b685a in msvc::thread_helper_t::thread_entry(void*) ()
#23 0x00007f16cd6dcfa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#24 0x00007f16cd60deff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 13 (Thread 0x7f16aeffd700 (LWP 899492)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x11559a4 <cond+44>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x1155950 <mutex>, cond=0x1155978 <cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x1155978 <cond>, mutex=0x1155950 <mutex>) at pthread_cond_wait.c:655
#3  0x00000000009c68b6 in uv_cond_wait ()
#4  0x00000000009b9326 in worker ()
#5  0x00007f16cd6dcfa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f16cd60deff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 12 (Thread 0x7f16af7fe700 (LWP 899491)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x11559a0 <cond+40>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x1155950 <mutex>, cond=0x1155978 <cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x1155978 <cond>, mutex=0x1155950 <mutex>) at pthread_cond_wait.c:655
#3  0x00000000009c68b6 in uv_cond_wait ()
#4  0x00000000009b9326 in worker ()
#5  0x00007f16cd6dcfa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f16cd60deff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 11 (Thread 0x7f16affff700 (LWP 899490)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x7f16942b4d70) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x7f16942b4d20, cond=0x7f16942b4d48) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x7f16942b4d48, mutex=0x7f16942b4d20) at pthread_cond_wait.c:655
#3  0x00000000009dd15c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#4  0x0000000000479ac5 in a_compiler_thread::start(edge::query_action_data&) ()
#5  0x00000000004791d6 in edge::an_edge_translation_unit_impl::reinitialize_without_fallback(bool, snapshot::snapshot_session_t, edge::file_position const&, edge::query_action_data&) ()
#6  0x0000000000472d66 in edge::an_edge_translation_unit_impl::reinitialize(bool, snapshot::snapshot_session_t, edge::file_position const&, edge::query_action_data&) ()
#7  0x00000000004657ba in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#8  0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#9  0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#10 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#11 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#12 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#13 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#14 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#15 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#16 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#17 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#18 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#19 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#20 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#21 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#22 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#23 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#24 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#25 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#26 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#27 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#28 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#29 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#30 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#31 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#32 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#33 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#34 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#35 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#36 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#37 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#38 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#39 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#40 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#41 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#42 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#43 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#44 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#45 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#46 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#47 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#48 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#49 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#50 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#51 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#52 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#53 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#54 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#55 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#56 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#57 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#58 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#59 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#60 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#61 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#62 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#63 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#64 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#65 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#66 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#67 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#68 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#69 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#70 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#71 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#72 0x00000000004658e3 in edge::scoped_query_manager::check_and_compile(edge::file_position const&, edge::file_position const&, edge::file_position const&, bool, bool, bool) ()
#73 0x0000000000473aa4 in edge::scoped_query_manager::get_trees_overlapping_span(a_canonicalized_file_name, msvc::strong_typedef_t<msvc::offset_traits>, msvc::strong_typedef_t<msvc::offset_traits>, int, edge::tree_query_result_builder*) ()
#74 0x00000000004b1502 in edge::intellisense_operation::symbol_at_cursor(bool, msvc::buffer<char>*) ()
#75 0x000000000042a20b in edge_intellisense_server_impl::handle_goto_def(cpp_intellisense::DocumentPosition const&, cpp_intellisense::goto_mode, cpp_intellisense::GoToDefResponse&)::{lambda(edge::intellisense_operation&, cpp_intellisense::GoToDefResponse&)#1}::operator()(edge::intellisense_operation&, cpp_intellisense::GoToDefResponse&) const ()
#76 0x0000000000429ce2 in void edge_intellisense_operation::perform_isense_operation_helper<cpp_intellisense::GoToDefResponse>(char const*, edge::intellisense_operation&, cpp_intellisense::DocumentPosition const&, std::function<void (edge::intellisense_operation&, cpp_intellisense::GoToDefResponse&)>&&, cpp_intellisense::GoToDefResponse&, edge::allocator*, snapshot::snapshot_session_t const&) ()
#77 0x000000000042945d in void edge_intellisense_operation::perform_isense_operation<cpp_intellisense::GoToDefResponse>(char const*, cpp_intellisense::DocumentPosition const&, std::function<void (edge::intellisense_operation&, cpp_intellisense::GoToDefResponse&)>&&, cpp_intellisense::GoToDefResponse&, snapshot::snapshot_session_t, bool, edge::allocator*) ()
#78 0x0000000000420532 in edge_intellisense_server_impl::handle_goto_def(cpp_intellisense::DocumentPosition const&, cpp_intellisense::goto_mode, cpp_intellisense::GoToDefResponse&) ()
#79 0x000000000044aa6d in cpp_intellisense::server_stub::handle_goto_def(msvc::deserializer_t&, msvc::serializer_t&) ()
#80 0x00000000004595e0 in msvc::base_channel_t::dispatch_message(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) ()
#81 0x000000000098bc03 in msvc::common_channel_t::perform_call(unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
#82 0x00000000009b9465 in worker ()
#83 0x00007f16cd6dcfa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#84 0x00007f16cd60deff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 10 (Thread 0x7f16c4ff9700 (LWP 899489)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x11559a4 <cond+44>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x1155950 <mutex>, cond=0x1155978 <cond>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x1155978 <cond>, mutex=0x1155950 <mutex>) at pthread_cond_wait.c:655
#3  0x00000000009c68b6 in uv_cond_wait ()
#4  0x00000000009b9326 in worker ()
#5  0x00007f16cd6dcfa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007f16cd60deff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7f16c57fa700 (LWP 899488)):
#0  0x00007f16cd60e21f in epoll_wait (epfd=3, events=0x7f16c57f6e10, maxevents=1024, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1  0x00000000009bde0a in uv.io_poll ()
#2  0x00000000009bc0a8 in uv_run ()
#3  0x0000000000991afb in msvc::loop_t::run_loop() ()
#4  0x00000000009975b5 in void std::__invoke_impl<void, void (*)(std::shared_ptr<msvc::loop_t>), std::shared_ptr<msvc::loop_t> >(std::__invoke_other, void (*&&)(std::shared_ptr<msvc::loop_t>), std::shared_ptr<msvc::loop_t>&&) ()
#5  0x00000000009b685a in msvc::thread_helper_t::thread_entry(void*) ()
#6  0x00007f16cd6dcfa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#7  0x00007f16cd60deff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7f16c5ffb700 (LWP 899487)):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7f16c5ffae20, expected=0, futex_word=0x114bbb8 <vscode::g_thread_pool+304>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  __pthread_cond_wait_common (abstime=0x7f16c5ffae20, mutex=0x19df7b0, cond=0x114bb90 <vscode::g_thread_pool+264>) at pthread_cond_wait.c:539
#2  __pthread_cond_timedwait (cond=0x114bb90 <vscode::g_thread_pool+264>, mutex=0x19df7b0, abstime=0x7f16c5ffae20) at pthread_cond_wait.c:667
#3  0x000000000045c30d in std::cv_status std::_V2::condition_variable_any::wait_until<std::unique_lock<std::mutex>, std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::unique_lock<std::mutex>&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#4  0x000000000093c866 in unsigned int msvc::bitset_event_t::wait_for_any_set<long, std::ratio<1l, 1000l> >(std::chrono::duration<long, std::ratio<1l, 1000l> > const&, unsigned int) ()
#5  0x000000000093b6d0 in msvc::thread_t::invoker_t<vscode::thread_pool::thread_pool()::$_0>::invoke() ()
#6  0x00000000009b685a in msvc::thread_helper_t::thread_entry(void*) ()
#7  0x00007f16cd6dcfa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#8  0x00007f16cd60deff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7f16c67fc700 (LWP 899486)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x114bbb8 <vscode::g_thread_pool+304>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x19df7b0, cond=0x114bb90 <vscode::g_thread_pool+264>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x114bb90 <vscode::g_thread_pool+264>, mutex=0x19df7b0) at pthread_cond_wait.c:655
#3  0x00000000009dd15c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#4  0x000000000093c248 in void std::_V2::condition_variable_any::wait<std::unique_lock<std::mutex> >(std::unique_lock<std::mutex>&) ()
#5  0x000000000093bcb3 in msvc::bitset_event_t::wait_any_set(unsigned int) ()
#6  0x000000000093a555 in vscode::thread_pool::do_work(unsigned long) ()
#7  0x00000000009b685a in msvc::thread_helper_t::thread_entry(void*) ()
#8  0x00007f16cd6dcfa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9  0x00007f16cd60deff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7f16c6ffd700 (LWP 899485)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x114bbb8 <vscode::g_thread_pool+304>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x19df7b0, cond=0x114bb90 <vscode::g_thread_pool+264>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x114bb90 <vscode::g_thread_pool+264>, mutex=0x19df7b0) at pthread_cond_wait.c:655
#3  0x00000000009dd15c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#4  0x000000000093c248 in void std::_V2::condition_variable_any::wait<std::unique_lock<std::mutex> >(std::unique_lock<std::mutex>&) ()
#5  0x000000000093bcb3 in msvc::bitset_event_t::wait_any_set(unsigned int) ()
#6  0x000000000093a555 in vscode::thread_pool::do_work(unsigned long) ()
#7  0x00000000009b685a in msvc::thread_helper_t::thread_entry(void*) ()
#8  0x00007f16cd6dcfa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9  0x00007f16cd60deff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7f16c77fe700 (LWP 899484)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x114bbb8 <vscode::g_thread_pool+304>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x19df7b0, cond=0x114bb90 <vscode::g_thread_pool+264>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x114bb90 <vscode::g_thread_pool+264>, mutex=0x19df7b0) at pthread_cond_wait.c:655
#3  0x00000000009dd15c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#4  0x000000000093c248 in void std::_V2::condition_variable_any::wait<std::unique_lock<std::mutex> >(std::unique_lock<std::mutex>&) ()
#5  0x000000000093bcb3 in msvc::bitset_event_t::wait_any_set(unsigned int) ()
#6  0x000000000093a555 in vscode::thread_pool::do_work(unsigned long) ()
#7  0x00000000009b685a in msvc::thread_helper_t::thread_entry(void*) ()
#8  0x00007f16cd6dcfa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9  0x00007f16cd60deff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7f16c7fff700 (LWP 899483)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x114bbb8 <vscode::g_thread_pool+304>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x19df7b0, cond=0x114bb90 <vscode::g_thread_pool+264>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x114bb90 <vscode::g_thread_pool+264>, mutex=0x19df7b0) at pthread_cond_wait.c:655
#3  0x00000000009dd15c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#4  0x000000000093c248 in void std::_V2::condition_variable_any::wait<std::unique_lock<std::mutex> >(std::unique_lock<std::mutex>&) ()
#5  0x000000000093bcb3 in msvc::bitset_event_t::wait_any_set(unsigned int) ()
#6  0x000000000093a555 in vscode::thread_pool::do_work(unsigned long) ()
#7  0x00000000009b685a in msvc::thread_helper_t::thread_entry(void*) ()
#8  0x00007f16cd6dcfa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9  0x00007f16cd60deff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7f16ccce5700 (LWP 899482)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x114bbb8 <vscode::g_thread_pool+304>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x19df7b0, cond=0x114bb90 <vscode::g_thread_pool+264>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x114bb90 <vscode::g_thread_pool+264>, mutex=0x19df7b0) at pthread_cond_wait.c:655
#3  0x00000000009dd15c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#4  0x000000000093c248 in void std::_V2::condition_variable_any::wait<std::unique_lock<std::mutex> >(std::unique_lock<std::mutex>&) ()
#5  0x000000000093bcb3 in msvc::bitset_event_t::wait_any_set(unsigned int) ()
#6  0x000000000093a555 in vscode::thread_pool::do_work(unsigned long) ()
#7  0x00000000009b685a in msvc::thread_helper_t::thread_entry(void*) ()
#8  0x00007f16cd6dcfa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9  0x00007f16cd60deff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f16cd4e6700 (LWP 899481)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0x114bbb8 <vscode::g_thread_pool+304>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1  __pthread_cond_wait_common (abstime=0x0, mutex=0x19df7b0, cond=0x114bb90 <vscode::g_thread_pool+264>) at pthread_cond_wait.c:502
#2  __pthread_cond_wait (cond=0x114bb90 <vscode::g_thread_pool+264>, mutex=0x19df7b0) at pthread_cond_wait.c:655
#3  0x00000000009dd15c in std::condition_variable::wait(std::unique_lock<std::mutex>&) ()
#4  0x000000000093c248 in void std::_V2::condition_variable_any::wait<std::unique_lock<std::mutex> >(std::unique_lock<std::mutex>&) ()
#5  0x000000000093bcb3 in msvc::bitset_event_t::wait_any_set(unsigned int) ()
#6  0x000000000093a555 in vscode::thread_pool::do_work(unsigned long) ()
#7  0x00000000009b685a in msvc::thread_helper_t::thread_entry(void*) ()
#8  0x00007f16cd6dcfa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#9  0x00007f16cd60deff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f16cd4e7740 (LWP 899480)):
#0  futex_abstimed_wait_cancelable (private=0, abstime=0x7fff269a88a0, expected=0, futex_word=0xf23080 <g_eventAbort+80>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
#1  __pthread_cond_wait_common (abstime=0x7fff269a88a0, mutex=0x19dc270, cond=0xf23058 <g_eventAbort+40>) at pthread_cond_wait.c:539
#2  __pthread_cond_timedwait (cond=0xf23058 <g_eventAbort+40>, mutex=0x19dc270, abstime=0x7fff269a88a0) at pthread_cond_wait.c:667
#3  0x000000000045c30d in std::cv_status std::_V2::condition_variable_any::wait_until<std::unique_lock<std::mutex>, std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >(std::unique_lock<std::mutex>&, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > const&) ()
#4  0x0000000000451d07 in bool msvc::event_t::wait_for<long, std::ratio<1l, 1l> >(std::chrono::duration<long, std::ratio<1l, 1l> > const&) ()
#5  0x000000000044d806 in RunServer(unsigned int, char const*) ()
#6  0x000000000044de0b in main ()

The check_and_compile block stands out to me at a glance, but I'm not sure what to make of it.

Originally posted by @justin39

bobbrow commented 1 year ago

@justin39 I think the check_and_compile is intentional recursion. That thread is blocked on the other thread running find_symbol. Are you able to tell if that other thread is stuck in some infinite parsing loop, i.e. is CPU still being used and/or how do the call stacks change over time?

Originally posted by @sean-mcmanus

bobbrow commented 1 year ago

From what I can tell, countless LWP (lightweight processes?) are getting spun up and down. They seem to be spinning over similar operations over different files - for example:

Thread 267 (Thread 0x7fcd8501e700 (LWP 1957276)):
#0  __GI___xstat (vers=<optimized out>, name=0x7fcd45cc1530 "/opt/mitkrb5/include", buf=0x7fcd8501d900) at ../sysdeps/unix/sysv/linux/wordsize-64/xstat.c:35
#1  0x00000000006a825e in compare_dir_names(char const*, char const*, int) ()
#2  0x00000000006a80c6 in remove_duplicate_include_dirs(a_directory_name_entry**, int) ()
#3  0x00000000005692a0 in proc_command_line(int, char**) ()
#4  0x0000000000530012 in cfe_main(int, char**) ()
#5  0x00000000008e3c86 in cfe_main_exception_handler(int, char**) ()
#6  0x0000000000530156 in edg_main(int, char**) ()
#7  0x00000000008dcf9c in edge_compiler_main(int, char const**) ()
#8  0x00000000004b9e96 in preparse(int, char const**, a_scout_store*, edge::translation_unit*) ()
#9  0x000000000047e95f in a_compiler_thread::compiler_thread_routine(a_compiler_thread*) ()
#10 0x00000000009b685a in msvc::thread_helper_t::thread_entry(void*) ()
#11 0x00007fcd8aa1ffa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#12 0x00007fcd8a950eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

A couple seconds later

Thread 288 (Thread 0x7fcd8501e700 (LWP 1961535)):
#0  __GI___xstat (vers=<optimized out>, name=0x7fcd43645060 "/path/to/dependencies/c/include", buf=0x7fcd8501d900) at ../sysdeps/unix/sysv/linux/wordsize-64/xstat.c:35
#1  0x00000000006a825e in compare_dir_names(char const*, char const*, int) ()
#2  0x00000000006a80c6 in remove_duplicate_include_dirs(a_directory_name_entry**, int) ()
#3  0x00000000005692a0 in proc_command_line(int, char**) ()
#4  0x0000000000530012 in cfe_main(int, char**) ()
#5  0x00000000008e3c86 in cfe_main_exception_handler(int, char**) ()
#6  0x0000000000530156 in edg_main(int, char**) ()
#7  0x00000000008dcf9c in edge_compiler_main(int, char const**) ()
#8  0x00000000004b9e96 in preparse(int, char const**, a_scout_store*, edge::translation_unit*) ()
#9  0x000000000047e95f in a_compiler_thread::compiler_thread_routine(a_compiler_thread*) ()
#10 0x00000000009b685a in msvc::thread_helper_t::thread_entry(void*) ()
#11 0x00007fcd8aa1ffa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#12 0x00007fcd8a950eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

All the files seem to correspond to entries under configurations.includePath in .vscode/c_cpp_properties.json. Looks like we have around 200 entries under includePath in this project; not sure how many files we expect to find at each path though.

Originally posted by @justin39

bobbrow commented 1 year ago

That call stack is different from the previous one. Do you know how you're getting it to break repeatedly in remove_duplicate_include_dirs ? Does it seem like the 200 include directories are somehow causing that code to take too long to process? It seems to be creating a lot of preparse threads.

Originally posted by @sean-mcmanus

bobbrow commented 1 year ago

I'm just pausing the cpptools-srv proc at random intervals and grabbing the transient LWP/thread on top - this seems to be what's happening when I leave VS Code to "idle" with 2+ cpp files open, as opposed to while it's actually spinning on a go-to-definition request.

It may well be related to the 200 include directories - I can try stripping them out, but it will likely break IntelliSense :(

Originally posted by @justin39

bobbrow commented 1 year ago

@justin39 The deep nested calls to check_and_compile appears to be something that should not be happening when there's a go to definition...we're looking into what could be going wrong. I'm not sure if that's related to the many preparse threads when idle issue yet.

@justin39 Does the bug repro with just 1 cpp file open or does the repro require 2 files? Do the preparse threads stop launching eventually after being idle for a certain amount of time (when IntelliSense update finishes)?

Originally posted by @sean-mcmanus

bobbrow commented 1 year ago

@sean-mcmanus Did some further testing - we can repro it with only one file, actually! Seems to happen whenever we jump to a header file that's located on our networked artifact storage. Not quite sure why yet, though - in theory, it should be the same as any other header file to the extension.

@sean-mcmanus tested jumping to a header file in a couple different scenarios like keeping the header file on disk or symlinking to a header file also on disk; could only repro the issue when jumping to a header file located on a FUSE file system. Any leads as to why that might be the case? The FUSE file system is a read-only storage used for artifacts.

Originally posted by @justin39

bobbrow commented 1 year ago

@justin39 By jumping to a header file, do you mean using Go to Definition on a #include for the header file? Does the same issue happen if you open the header file the Open File menu option? I don't think we've tested with a FUSE file system so it's possible something is failing we don't expect. But the previously provided call stacks don't give a hint at what the file system issue could be. If the file system is remote, it's possible it's just normal processing that is really slow from reading the remote file.

Originally posted by @sean-mcmanus

bobbrow commented 1 year ago

@sean-mcmanus Ah, the problem seems to be when I close a file open on FUSE:

Originally posted by @justin39

bobbrow commented 1 year ago

@justin39 So the cpp file includes the header, right? When you run C/C++ Log Diagnostics before closing the header do you see just 1 TU being used, shared by the cpp and header (normally you should)? With C_Cpp.loggingLevel set to "Debug" what logging is shown in the C/C++ window after you close the header file without any other interaction -- you should just see a didClose message or no message (since VS Code sometimes keeps the document open in the background). What does Log Diagnostics show after the header is closed (it should appear unchanged)? How do you know IntelliSense locks up? Is hover stuck Loading in the cpp file? Are you able to get a call stack for cpptools or cpptools-srv if that exists? I can't tell which of the previous call stacks are with this repro.

Originally posted by @sean-mcmanus

bobbrow commented 1 year ago

@sean-mcmanus sorry for the delay, follow up:

I can try and grab a new call stack, sure

Originally posted by @justin39

bobbrow commented 1 year ago

Whoops, never got those call stacks to your @sean-mcmanus - sorry!

Here's a couple call stacks:

cpptools-stack.txt cpptools-srv-spinning.txt cpptools-srv-idle.txt

All of these were taken while VS Code was hung - 'cpptools-stack' is for the cpptools process, 'cpptools-srv-spinning' is for the cpptools-srv process that is eating most of the CPU (and appears to be repeatedly spawning new LWPs), and 'cpptools-srv-idle' is for the second cpptools-srv process that doesn't appear to be doing much but I included for good measure.

Originally posted by @justin39

bobbrow commented 1 year ago

@justin39 The cause is the deeply nested check_and_compile in cpptools-srv -- same as the call stack you reported back in September. Another users reported that at https://github.com/microsoft/vscode-cpptools/issues/7333 .

We're not sure yet what can cause this. Is this 100% reproducible with particular files or is it random?

Originally posted by @sean-mcmanus

bobbrow commented 1 year ago

Ah, https://github.com/microsoft/vscode-cpptools/issues/7333 is probably the same structure 😅

I'm fairly confident it's 100% reproducible with our particular files + dependency setup at this point. As long as I close a FUSE header file the issue seems to occur, but I haven't tested on arbitrary FUSE setups, just different files on the FUSE we happen to be using.

Originally posted by @justin39

bobbrow commented 1 year ago

@justin39 Oh, I didn't realize this was the FUSE file system issue. Have you tried setting the C_Cpp.intelliSenseCacheSize to 0 already? There might be some issue with that. Otherwise, if you can identify some minimal code sample that repros it that might help.

Originally posted by @sean-mcmanus

justin39 commented 1 year ago

Quick summary to collect my thoughts....

The issue manifests when closing a C/C++ file under the following conditions:

Before any files are closed, Intellisense behaves normally. Once the file is closed:

If the spinning cpptools-srv process is killed, any hanging Intellisense actions, such as a "Go to definition" that hasn't timed out, will execute.

lhmouse commented 8 months ago

Not sure whether it's related, but I'm getting this exact issue on two separated Linux machines. The extension stops working after a few minutes. After I kill coptools-srv, everything goes back to normal.

On my side, this issue seems new in 1.19.4. I have downgraded to 1.18.5 and the issue is gone.