Include debug info in clangd release packages

[HighCommander4]

Filing this following a suggestion by @tobiashieta on Discord.

This would allow users of the release packages to include more useful stack traces when reporting a crash.

The debug info could be included in the packages by default, or alternatively supplied in a separate package.

[zyn0217]

The CI script has already enabled -gline-tables-only for both Mac and Linux releases.

https://github.com/clangd/clangd/blob/6e7f5968bae8076cc5a4a751af061e71bbb3e32f/.github/workflows/autobuild.yaml#L183

But sending a fatal signal like SIGABRT by hand to the latest released binary(16.0.2) on my MacOS (without llvm installed) still yields nothing but signature + offsets:

I[22:18:41.886] clangd version 16.0.2 (https://github.com/llvm/llvm-project 18ddebe1a1a9bde349441631365f0472e9693520)
I[22:18:41.886] Features: mac+grpc+xpc
I[22:18:41.886] PID: 7272
I[22:18:41.886] Working directory: /Users/zyounan/clangd_16.0.2/bin
I[22:18:41.886] argv[0]: ./clangd
I[22:18:41.886] Starting LSP over stdin/stdout
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  clangd                   0x000000010478fc2c llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 56
1  clangd                   0x000000010478ee98 llvm::sys::RunSignalHandlers() + 64
2  clangd                   0x0000000104790304 SignalHandler(int) + 344
3  libsystem_platform.dylib 0x00000001a3c042a4 _sigtramp + 56
4  libsystem_c.dylib        0x00000001a3ac6978 __sread + 24
5  libsystem_c.dylib        0x00000001a3aa1b8c _sread + 32
6  libsystem_c.dylib        0x00000001a3aa1b20 __srefill1 + 36
7  libsystem_c.dylib        0x00000001a3ab6c84 fgets + 112
8  clangd                   0x000000010566701c clang::clangd::(anonymous namespace)::readLine(__sFILE*, llvm::SmallVectorImpl<char>&) + 132
9  clangd                   0x0000000105665fc0 clang::clangd::(anonymous namespace)::JSONTransport::loop(clang::clangd::Transport::MessageHandler&) + 688
10 clangd                   0x0000000105558b50 clang::clangd::ClangdLSPServer::run() + 44
11 clangd                   0x00000001046bc614 main + 10216
12 dyld                     0x00000001a38abe50 start + 2544

In contrast, if we do have llvm-symbolizer installed (Use my Linux as an example):

I[22:37:41.498] clangd version 16.0.2 (https://github.com/llvm/llvm-project 18ddebe1a1a9bde349441631365f0472e9693520)
I[22:37:41.498] Features: linux+grpc
I[22:37:41.498] PID: 8343
I[22:37:41.498] Working directory: /tmp/clangd_16.0.2/bin
I[22:37:41.498] argv[0]: ./clangd
I[22:37:41.498] Starting LSP over stdin/stdout
 #0 0x00000000005398bb llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /__w/clangd/clangd/llvm-project/llvm/lib/Support/Unix/Signals.inc:567:13
 #1 0x00000000005378bc llvm::sys::RunSignalHandlers() /__w/clangd/clangd/llvm-project/llvm/lib/Support/Signals.cpp:105:18
 #2 0x0000000000539f66 SignalHandler(int) /__w/clangd/clangd/llvm-project/llvm/lib/Support/Unix/Signals.inc:412:1
 #3 0x00007efe28094f60 __restore_rt sigaction.c:0:0
 #4 0x00007efe27e5974e read /root/glibc-2.33/io/../sysdeps/unix/sysv/linux/read.c:26:0
 #5 0x00007efe27ded25f _IO_file_underflow@@GLIBC_2.2.5 /root/glibc-2.33/libio/fileops.c:519:0
 #6 0x00007efe27dee4c2 _IO_default_uflow /root/glibc-2.33/libio/genops.c:363:0
 #7 0x00007efe27de15d2 _IO_getline_info /root/glibc-2.33/libio/iogetline.c:61:0
 #8 0x00007efe27de0316 _IO_fgets /root/glibc-2.33/libio/iofgets.c:57:0
 #9 0x000000000148008b retryAfterSignalUnlessShutdown<(lambda at /__w/clangd/clangd/llvm-project/clang-tools-extra/clangd/JSONTransport.cpp:211:22), char *> /__w/clangd/clangd/llvm-project/clang-tools-extra/clangd/support/Shutdown.h:79:16
#10 0x000000000148008b clang::clangd::(anonymous namespace)::readLine(_IO_FILE*, llvm::SmallVectorImpl<char>&) /__w/clangd/clangd/llvm-project/clang-tools-extra/clangd/JSONTransport.cpp:210:10
#11 0x000000000147e80e readStandardMessage /__w/clangd/clangd/llvm-project/clang-tools-extra/clangd/JSONTransport.cpp:234:9
#12 0x000000000147e80e readRawMessage /__w/clangd/clangd/llvm-project/clang-tools-extra/clangd/JSONTransport.cpp:149:50
#13 0x000000000147e80e clang::clangd::(anonymous namespace)::JSONTransport::loop(clang::clangd::Transport::MessageHandler&) /__w/clangd/clangd/llvm-project/clang-tools-extra/clangd/JSONTransport.cpp:112:11
#14 0x00000000013418f1 operator bool /__w/clangd/clangd/llvm-project/llvm/include/llvm/Support/Error.h:236:21
#15 0x00000000013418f1 clang::clangd::ClangdLSPServer::run() /__w/clangd/clangd/llvm-project/clang-tools-extra/clangd/ClangdLSPServer.cpp:1649:12
#16 0x00000000004acc24 main /__w/clangd/clangd/llvm-project/clang-tools-extra/clangd/tool/ClangdMain.cpp:1028:18
#17 0x00007efe27d955e0 __libc_start_main /root/glibc-2.33/csu/../csu/libc-start.c:366:0
#18 0x0000000000412a3a _start (./clangd+0x412a3a)

Thus I guess the crash report without line number is due to the absence of llvm-symbolizer, if the binary is obtained from the release page.

[HighCommander4]

Interesting, thanks for checking. I think the part about needing llvm-symbolizer is expected.

Reviewing the Discord conversation with Tobias, it was the packages on releases.llvm.org that he said didn't contain debug symbols; if our Github releases do, then maybe there's nothing further to do here.

[HighCommander4]

Though I guess the awkward part is, to get a local stack trace with line numbers, user needs to combine our binary from a Github package, with llvm-symbolizer from a package from releases.llvm.org?

Maybe we should include llvm-symbolizer in our Github packages?

[zyn0217]

if our Github releases do, then maybe there's nothing further to do here.

Not for Windows. Triggering a crash on Windows still yields nothing for me:

I[12:02:34.368] Built preamble of size 233276 for file F:\clangd-16.0.2\crash.cpp version null in 0.01 seconds
I[12:02:34.369] Building AST...
Exception Code: 0xC0000005
 #0 0x00007ff7028105a3 (F:\clangd-16.0.2\bin\clangd.exe+0x2105a3)
 #1 0x00007ff7027c276e (F:\clangd-16.0.2\bin\clangd.exe+0x1c276e)
 #2 0x00007ff7027c1d1b (F:\clangd-16.0.2\bin\clangd.exe+0x1c1d1b)
 #3 0x00007ff7027c1eb8 (F:\clangd-16.0.2\bin\clangd.exe+0x1c1eb8)
 #4 0x00007ff7027c1d1b (F:\clangd-16.0.2\bin\clangd.exe+0x1c1d1b)
 #5 0x00007ff7027c1eb8 (F:\clangd-16.0.2\bin\clangd.exe+0x1c1eb8)
 #6 0x00007ff7027c1d1b (F:\clangd-16.0.2\bin\clangd.exe+0x1c1d1b)
 #7 0x00007ff7027c1a31 (F:\clangd-16.0.2\bin\clangd.exe+0x1c1a31)
 #8 0x00007ff7030092bf (F:\clangd-16.0.2\bin\clangd.exe+0xa092bf)
 #9 0x00007ff702ff3705 (F:\clangd-16.0.2\bin\clangd.exe+0x9f3705)
#10 0x00007ff702ff0aa8 (F:\clangd-16.0.2\bin\clangd.exe+0x9f0aa8)
#11 0x00007ff7043cdc33 (F:\clangd-16.0.2\bin\clangd.exe+0x1dcdc33)
#12 0x00007ff7043cf008 (F:\clangd-16.0.2\bin\clangd.exe+0x1dcf008)
#13 0x00007ff7043ce86b (F:\clangd-16.0.2\bin\clangd.exe+0x1dce86b)
#14 0x00007ff7043c925c (F:\clangd-16.0.2\bin\clangd.exe+0x1dc925c)
#15 0x00007ff7043cbc60 (F:\clangd-16.0.2\bin\clangd.exe+0x1dcbc60)
#16 0x00007ff70434f79f (F:\clangd-16.0.2\bin\clangd.exe+0x1d4f79f)
#17 0x00007ff704394b94 (F:\clangd-16.0.2\bin\clangd.exe+0x1d94b94)
#18 0x00007ff704395662 (F:\clangd-16.0.2\bin\clangd.exe+0x1d95662)
#19 0x00007ff704392b03 (F:\clangd-16.0.2\bin\clangd.exe+0x1d92b03)
#20 0x00007ff704378c35 (F:\clangd-16.0.2\bin\clangd.exe+0x1d78c35)
#21 0x00007ff70434e3bd (F:\clangd-16.0.2\bin\clangd.exe+0x1d4e3bd)
#22 0x00007ff704353514 (F:\clangd-16.0.2\bin\clangd.exe+0x1d53514)
#23 0x00007ff704348ec2 (F:\clangd-16.0.2\bin\clangd.exe+0x1d48ec2)
#24 0x00007ff70429aed7 (F:\clangd-16.0.2\bin\clangd.exe+0x1c9aed7)
#25 0x00007ff70429a9cb (F:\clangd-16.0.2\bin\clangd.exe+0x1c9a9cb)
#26 0x00007ff70338cee3 (F:\clangd-16.0.2\bin\clangd.exe+0xd8cee3)
#27 0x00007ff7026b05e7 (F:\clangd-16.0.2\bin\clangd.exe+0xb05e7)
#28 0x00007ff7026b2177 (F:\clangd-16.0.2\bin\clangd.exe+0xb2177)
#29 0x00007ff7026a74ba (F:\clangd-16.0.2\bin\clangd.exe+0xa74ba)
#30 0x00007ff7041c5af0 (F:\clangd-16.0.2\bin\clangd.exe+0x1bc5af0)
#31 0x00007ffad4e07614 (C:\WINDOWS\System32\KERNEL32.DLL+0x17614)
#32 0x00007ffad6d426a1 (C:\WINDOWS\SYSTEM32\ntdll.dll+0x526a1)

F:\clangd-16.0.2>where llvm-symbolizer
C:\Program Files\LLVM\bin\llvm-symbolizer.exe

I think we can add extra PDB file for the Windows release, in the way where -DLLVM_ENABLE_PDB is set ON.

Reviewing the Discord conversation with Tobias, it was the packages on releases.llvm.org that he said didn't contain debug symbols;

Fedora maintains a nightly llvm package with debug symbols that can be found here. I suspect other RedHat distros like CentOS could also benefit from it, but I didn't give it a shot. In addition, the well-known official deb package release provides <bin>-dbgsym and .debug for debugging, see build logs here.

Unfortunately, we don't have one for Windows. But since the releases.llvm.org points to the release page on Github, I think it's better to submit the issue/request to the main repository?

Maybe we should include llvm-symbolizer in our Github packages?

Sounds reasonable to me. It is convenient for users without llvm-toolchain installed to obtain a glibc-independent and static-linked single binary and putting that into somewhere is rather portable than installing the whole llvm-toolchain, especially if the developing environment is based on a docker image. But what's concerning me is that, given the fact that llvm-symbolizer is distributed in various ways across different platforms, providing another binary release appears to be redundant and might bring some maintenance issues. (And it seems not to be our responsibility to maintain another binary except for clangd.)

[zyn0217]

Putting PDB file in the binary directory works for me.

Exception Code: 0xC0000005
 #0 0x00007ff6c79a9b43 clang::Decl::getKind D:\a\clangd\clangd\llvm-project\clang\include\clang\AST\DeclBase.h:435:0
 #1 0x00007ff6c79a9b43 clang::CXXRecordDecl::classof D:\a\clangd\clangd\llvm-project\clang\include\clang\AST\DeclCXX.h:1873:0
 #2 0x00007ff6c79a9b43 llvm::isa_impl<clang::CXXRecordDecl,clang::RecordDecl,void>::doit D:\a\clangd\clangd\llvm-project\llvm\include\llvm\Support\Casting.h:64:0
 #3 0x00007ff6c79a9b43 llvm::isa_impl_cl<clang::CXXRecordDecl,clang::RecordDecl const *>::doit D:\a\clangd\clangd\llvm-project\llvm\include\llvm\Support\Casting.h:110:0
 #4 0x00007ff6c79a9b43 llvm::isa_impl_wrap<clang::CXXRecordDecl,clang::RecordDecl const *,clang::RecordDecl const *>::doit D:\a\clangd\clangd\llvm-project\llvm\include\llvm\Support\Casting.h:137:0
 #5 0x00007ff6c79a9b43 llvm::isa_impl_wrap<clang::CXXRecordDecl,clang::RecordDecl const * const,clang::RecordDecl const *>::doit D:\a\clangd\clangd\llvm-project\llvm\include\llvm\Support\Casting.h:127:0
 #6 0x00007ff6c79a9b43 llvm::CastIsPossible<clang::CXXRecordDecl,clang::RecordDecl const *,void>::isPossible D:\a\clangd\clangd\llvm-project\llvm\include\llvm\Support\Casting.h:255:0
 #7 0x00007ff6c79a9b43 llvm::CastInfo<clang::CXXRecordDecl,clang::RecordDecl const *,void>::doCastIfPossible D:\a\clangd\clangd\llvm-project\llvm\include\llvm\Support\Casting.h:493:0
 #8 0x00007ff6c79a9b43 llvm::dyn_cast D:\a\clangd\clangd\llvm-project\llvm\include\llvm\Support\Casting.h:663:0
 #9 0x00007ff6c79a9b43 clang::ASTContext::getASTRecordLayout(class clang::RecordDecl const *) const D:\a\clangd\clangd\llvm-project\clang\lib\AST\RecordLayoutBuilder.cpp:3308:0
#10 0x00007ff6c795a97f clang::ASTContext::getTypeInfoImpl(class clang::Type const *) const D:\a\clangd\clangd\clangd_20230620\tools\clang\include\clang\AST\TypeNodes.inc:35:0
...omitted...