search

llvm / llvm-project

The LLVM Project is a collection of modular and reusable compiler and toolchain technologies.
http://llvm.org
Other
29.02k stars 11.96k forks source link

[clang-format] Out of memory in clang::format::guessLanguage() #60151

Open danielrparks opened 1 year ago

danielrparks commented 1 year ago

I have a file blis.h which causes the guessLanguage() function to continuously consume memory, possibly indefinitely. My friend @bagel897 , who has more ram than I do, observed that it consumed 25GB before she ran out.

I have clang 15.0.7 on Arch Linux.

This affects clangd when editing the problem file as well as clang-format when formatting it. clangd is able to index the file as long as it is not opened directly in the editor, and clang is able to compile it without problems.

I believe that guessLanguage() is to blame for the issue because if I change the file extension from .h to .hpp, clang-format and clangd work correctly with the file.

Here is a stack trace from a clangd process experiencing the issue:

(gdb) info threads
  Id   Target Id                                           Frame 
* 1    Thread 0x7f692a554dc0 (LWP 30485) "clangd.main"     __GI___libc_read (nbytes=4096, 
    buf=0x55a22f0ebe00, fd=0) at ../sysdeps/unix/sysv/linux/read.c:26
  2    Thread 0x7f691d3ff6c0 (LWP 30486) "clangd.main"     __futex_abstimed_wait_common64 (private=0, 
    cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55a22f13d858) at futex-internal.c:57
  3    Thread 0x7f691cbfe6c0 (LWP 30487) "ground-worker-1" __futex_abstimed_wait_common64 (private=0, 
    cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55a22f13e5a8) at futex-internal.c:57
  4    Thread 0x7f691c3fd6c0 (LWP 30488) "ground-worker-2" __futex_abstimed_wait_common64 (private=0, 
    cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55a22f13e5a8) at futex-internal.c:57
  5    Thread 0x7f691bbfc6c0 (LWP 30489) "ground-worker-3" __futex_abstimed_wait_common64 (private=0, 
    cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55a22f13e5a8) at futex-internal.c:57
  6    Thread 0x7f69133fb6c0 (LWP 30490) "ground-worker-4" __futex_abstimed_wait_common64 (private=0, 
    cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55a22f13e5a8) at futex-internal.c:57
  7    Thread 0x7f6913fff6c0 (LWP 30491) "ground-worker-5" __futex_abstimed_wait_common64 (private=0, 
    cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55a22f13e5a8) at futex-internal.c:57
  8    Thread 0x7f691b3fb6c0 (LWP 30492) "ground-worker-6" __futex_abstimed_wait_common64 (private=0, 
    cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55a22f13e5a8) at futex-internal.c:57
  9    Thread 0x7f691abfa6c0 (LWP 30493) "ground-worker-7" __futex_abstimed_wait_common64 (private=0, 
    cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55a22f13e5a8) at futex-internal.c:57
  10   Thread 0x7f691a3f96c0 (LWP 30494) "ground-worker-8" __futex_abstimed_wait_common64 (private=0, 
    cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55a22f13e5a8) at futex-internal.c:57
  11   Thread 0x7f6918c5f6c0 (LWP 30495) "STWorker:blis.h" 0x00007f69299082ff in llvm::SmallVectorImpl<clang::format::UnwrappedLine>::~SmallVectorImpl () at /usr/include/llvm/ADT/SmallVector.h:587
  12   Thread 0x7f6912bfa6c0 (LWP 30496) "leWorker:blis.h" __futex_abstimed_wait_common64 (private=0, 
    cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x55a22f11cc14) at futex-internal.c:57
(gdb) thread 11
[Switching to thread 11 (Thread 0x7f6918c5f6c0 (LWP 30495))]
#0  0x00007f69299082ff in llvm::SmallVectorImpl<clang::format::UnwrappedLine>::~SmallVectorImpl ()
    at /usr/include/llvm/ADT/SmallVector.h:587
587    if (!this->isSmall())
(gdb) bt
#0  0x00007f69299082ff in llvm::SmallVectorImpl<clang::format::UnwrappedLine>::~SmallVectorImpl ()
    at /usr/include/llvm/ADT/SmallVector.h:587
#1  llvm::SmallVector<clang::format::UnwrappedLine, 0u>::~SmallVector ()
    at /usr/include/llvm/ADT/SmallVector.h:1193
#2  clang::format::UnwrappedLineNode::~UnwrappedLineNode ()
    at /usr/src/debug/clang/clang-15.0.7.src/lib/Format/UnwrappedLineParser.h:336
#3  clang::format::UnwrappedLineParser::pushToken ()
    at /usr/src/debug/clang/clang-15.0.7.src/lib/Format/UnwrappedLineParser.cpp:4340
#4  0x00007f69299a9f54 in clang::format::UnwrappedLineParser::nextToken(int) [clone .constprop.0] ()
    at /usr/src/debug/clang/clang-15.0.7.src/lib/Format/UnwrappedLineParser.cpp:4182
#5  0x00007f6929936572 in clang::format::UnwrappedLineParser::parsePPDefine ()
    at /usr/src/debug/clang/clang-15.0.7.src/lib/Format/UnwrappedLineParser.cpp:1199
#6  0x00007f6929936ab6 in clang::format::UnwrappedLineParser::parsePPDirective ()
    at /usr/src/debug/clang/clang-15.0.7.src/lib/Format/UnwrappedLineParser.cpp:1066
#7  0x00007f6929936d50 in clang::format::UnwrappedLineParser::readToken ()
    at /usr/src/debug/clang/clang-15.0.7.src/lib/Format/UnwrappedLineParser.cpp:4315
#8  0x00007f69299a9f72 in clang::format::UnwrappedLineParser::nextToken(int) [clone .constprop.0] ()
    at /usr/src/debug/clang/clang-15.0.7.src/lib/Format/UnwrappedLineParser.cpp:4185
#9  0x00007f69299339c1 in clang::format::UnwrappedLineParser::parseStructuralElement ()
    at /usr/src/debug/clang/clang-15.0.7.src/lib/Format/UnwrappedLineParser.cpp:1933
#10 0x00007f6929934689 in operator() ()
    at /usr/src/debug/clang/clang-15.0.7.src/lib/Format/UnwrappedLineParser.cpp:506
#11 clang::format::UnwrappedLineParser::parseLevel ()
    at /usr/src/debug/clang/clang-15.0.7.src/lib/Format/UnwrappedLineParser.cpp:602
#12 0x00007f6929931c0f in clang::format::UnwrappedLineParser::parseBlock ()
    at /usr/src/debug/clang/clang-15.0.7.src/lib/Format/UnwrappedLineParser.cpp:890
#13 0x00007f6929932ef8 in clang::format::UnwrappedLineParser::parseStructuralElement ()
    at /usr/src/debug/clang/clang-15.0.7.src/lib/Format/UnwrappedLineParser.cpp:1830
#14 0x00007f6929934689 in operator() ()
    at /usr/src/debug/clang/clang-15.0.7.src/lib/Format/UnwrappedLineParser.cpp:506
#15 clang::format::UnwrappedLineParser::parseLevel ()
    at /usr/src/debug/clang/clang-15.0.7.src/lib/Format/UnwrappedLineParser.cpp:602
#16 0x00007f6929936259 in clang::format::UnwrappedLineParser::parseFile ()
    at /usr/src/debug/clang/clang-15.0.7.src/lib/Format/UnwrappedLineParser.cpp:399
#17 0x00007f6929937471 in clang::format::UnwrappedLineParser::parse ()
    at /usr/src/debug/clang/clang-15.0.7.src/lib/Format/UnwrappedLineParser.cpp:358
#18 0x00007f69299379a8 in clang::format::TokenAnalyzer::process ()
    at /usr/src/debug/clang/clang-15.0.7.src/lib/Format/TokenAnalyzer.cpp:112
#19 0x00007f69298ed00c in clang::format::guessLanguage ()
    at /usr/src/debug/clang/clang-15.0.7.src/lib/Format/Format.cpp:3500
#20 0x00007f69298ed2be in clang::format::getStyle ()
    at /usr/src/debug/clang/clang-15.0.7.src/lib/Format/Format.cpp:3531
#21 0x000055a22d425aca in clang::clangd::getFormatStyleForFile ()
    at /usr/src/debug/clang/clang-15.0.7.src/tools/extra/clangd/SourceCode.cpp:582
#22 0x000055a22d3e7804 in clang::clangd::ParsedAST::build ()
    at /usr/src/debug/clang/clang-15.0.7.src/tools/extra/clangd/ParsedAST.cpp:556
#23 0x000055a22d432cda in generateDiagnostics ()
    at /usr/src/debug/clang/clang-15.0.7.src/tools/extra/clangd/TUScheduler.cpp:1186
#24 0x000055a22d433a0a in clang::clangd::(anonymous namespace)::ASTWorker::updatePreamble(std::unique_ptr<clang::CompilerInvocation, std::default_delete<clang::CompilerInvocation> >, clang::clangd::ParseInputs, std::shared_ptr<clang::clangd::PreambleData const>, std::vector<clang::clangd::Diag, std::allocator<clang::clangd::Diag> >, clang::clangd::WantDiagnostics)::{lambda()#1}::operator()() [clone .part.0] [clone .lto_priv.0] () at /usr/src/debug/clang/clang-15.0.7.src/tools/extra/clangd/TUScheduler.cpp:1119
#25 0x000055a22de8d47d in llvm::unique_function<void ()>::operator()() ()
    at /usr/include/llvm/ADT/FunctionExtras.h:384
#26 llvm::function_ref<void ()>::callback_fn<llvm::unique_function<void ()> >(long) ()
    at /usr/include/llvm/ADT/STLFunctionalExtras.h:45
#27 llvm::function_ref<void ()>::operator()() const () at /usr/include/llvm/ADT/STLFunctionalExtras.h:68
#28 clang::clangd::(anonymous namespace)::ASTWorker::runTask(llvm::StringRef, llvm::function_ref<void ()>) [clone .constprop.0] () at /usr/src/debug/clang/clang-15.0.7.src/tools/extra/clangd/TUScheduler.cpp:1299
#29 0x000055a22d42c41c in run ()
    at /usr/src/debug/clang/clang-15.0.7.src/tools/extra/clangd/TUScheduler.cpp:1432
#30 0x000055a22d582d37 in llvm::unique_function<void ()>::operator()() ()
    at /usr/include/llvm/ADT/FunctionExtras.h:384
#31 operator() () at /usr/src/debug/clang/clang-15.0.7.src/tools/extra/clangd/support/Threading.cpp:100
#32 Apply<clang::clangd::AsyncTaskRunner::runAsync(const llvm::Twine&, llvm::unique_function<void()>)::<lambda()> > () at /usr/include/llvm/Support/thread.h:42
#33 GenericThreadProxy<std::tuple<clang::clangd::AsyncTaskRunner::runAsync(const llvm::Twine&, llvm::unique_function<void()>)::<lambda()> > > () at /usr/include/llvm/Support/thread.h:50
#34 ThreadProxy<std::tuple<clang::clangd::AsyncTaskRunner::runAsync(const llvm::Twine&, llvm::unique_function<void()>)::<lambda()> > >(void) () at /usr/include/llvm/Support/thread.h:60
#35 0x00007f691f49f8fd in start_thread (arg=<optimized out>) at pthread_create.c:442
#36 0x00007f691f521d20 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
llvmbot commented 1 year ago

@llvm/issue-subscribers-clang-format

llvmbot commented 1 year ago

@llvm/issue-subscribers-clangd

kadircet commented 1 year ago

this is same as https://github.com/llvm/llvm-project/issues/48863 and https://github.com/clangd/clangd/issues/719

aqilc commented 3 months ago

This is affecting me too, and clangd is consuming Gigabytes even for the full version of windows.h and some other libraries I have included. Anything to help would be greatly appreciated!

HighCommander4 commented 2 months ago

This is affecting me too, and clangd is consuming Gigabytes even for the full version of windows.h and some other libraries I have included. Anything to help would be greatly appreciated!

This was fixed for clangd in version 19 (not yet released but pre-release builds can be found here) in https://github.com/llvm/llvm-project/pull/84133.

HighCommander4 commented 2 months ago

(Adjusted issue title to reflect that this has been resolved for clangd, at least in the case where you're just opening / reading / navigating a file. clang-format is still affected, and clangd is also still affected if you try to actually format an affected header file.)