LLVM debug symbols require `--savec` to be useful

[jabraham17]

When using the LLVM backend, we currently have limited support for debug symbols with -g. However, this only seems to work well in a debugger when the code is combined with --savec.

For example, the following just shows the assembly in the debugger and has no debug symbols

chpl -g example/hello.chpl
./hello --lldb # or lldb ./hello

Adding to it --savec tmp allows the debugger to show the code in terms of Chapel source code. You can't really inspect variables, but you can at least see the Chapel code.

chpl -g --savec tmp example/hello.chpl
./hello --lldb # or lldb ./hello

Both binaries have debug symbols (verified with llvm-dwarfdump and file), but only the second one has usable debug symbols.

We should be able to have debug symbols without --savec

$ chpl --version
chpl version 1.34.0 pre-release (89bf06ee4f)
  built with LLVM version 17.0.6
  available LLVM targets: xcore, x86-64, x86, wasm64, wasm32, ve, systemz, sparcel, sparcv9, sparc, riscv64, riscv32, ppc64le, ppc64, ppc32le, ppc32, nvptx64, nvptx, msp430, mips64el, mips64, mipsel, mips, loongarch64, loongarch32, lanai, hexagon, bpfeb, bpfel, bpf, avr, thumbeb, thumb, armeb, arm, amdgcn, r600, aarch64_32, aarch64_be, aarch64, arm64_32, arm64
Copyright 2020-2023 Hewlett Packard Enterprise Development LP
Copyright 2004-2019 Cray Inc.
(See LICENSE file for more details)

$ $CHPL_HOME/util/printchplenv --anonymize
CHPL_TARGET_PLATFORM: darwin
CHPL_TARGET_COMPILER: llvm
CHPL_TARGET_ARCH: arm64
CHPL_TARGET_CPU: native *
CHPL_LOCALE_MODEL: flat
CHPL_COMM: none
CHPL_TASKS: qthreads
CHPL_LAUNCHER: none
CHPL_TIMERS: generic
CHPL_UNWIND: none
CHPL_MEM: jemalloc *
CHPL_ATOMICS: cstdlib
CHPL_GMP: bundled *
CHPL_HWLOC: bundled
CHPL_RE2: bundled *
CHPL_LLVM: system *
CHPL_AUX_FILESYS: none

[mppf]

I'm adding the User Issue label b/c I think Tom W was running into this problem as described on Gitter.

[dlongnecke-cray]

@jabraham17 Do you remember the commands you gave to llvm-dwarfdump? Was it just a basic llvm-dwarfdump -a?

[dlongnecke-cray]

I was not able to get any interesting section info from llvm-dwarfdump when running on a test program such as test/llvm/llvmDebug/llvmDebug_test.chpl. All I see are a bunch of bogus frame info entries. So I'm not sure what you're seeing @jabraham17 or if I'm just goofing up somehow (edit: we both saw the same things, yay!). But after some further investigation, I'm fairly convinced we're not ~generating~ packaging our DWARF properly (this is highly entertaining out of context) on OSX regardless.

OSX

Starting out at ground zero here: https://stackoverflow.com/questions/10044697/where-how-does-apples-gcc-store-dwarf-inside-an-executable

According to this old SO thread, OSX made the decision to not bundle debug symbols into final executables because they're 10x+ the actual size of the binary and bloat link times. (edit: I believe the OSX linker literally ignores DWARF sections and strips them out of the produced executable.) This is where .dSYM archives come in.

If I compile a simple foo.c with a dummy struct definition in it (used in main), and run with clang -g, what I get is a binary and a .dSYM archive. Clang or LLVM is probably invoking dsymutil and pointing it at the intermediate object files before link time to create the .dSYM.

In the Chapel binary's case, I'm fairly positive that LLDB needs to consult the intermediate --savec object files in order to get at the DWARF sections, which is why LLDB reverts to showing assembly if you remove the --savec directory (you can confirm this by running llvm-dwarfdump on <savecdir>/chpl__module.o - you see all sorts of DWARF section entries that you'd expect). I think it's awesome that LLVM/LLDB can know to check the object files even if we're not generating a full on .dSYM archive.

Edit: More fun history: https://wiki.dwarfstd.org/Apple's_%22Lazy%22_DWARF_Scheme.md

Linux

I've confirmed that all the DWARF sections show up in the final Chapel program's binary if you do this same process on Linux. You can run objdump --dwarf --full-contents <chapel-binary> to check. And gdb seems to function just fine if I remove the --savec dir. So I'm happily convinced that we're successfully embedding the DWARF into the final Chapel binary, which is an acceptable practice on Linux.

Nothing to do here.

---

I think there's a couple ways we could proceed here:

• We could just make -g imply --savec under OSX+LLVM, or something to that effect. I think this is the least attractive option because there's a lot of unnecessary stuff in the temp directory. But it would get things to work ASAP.
• We could generate a proper .dSYM archive, hopefully by directing LLVM to do that for us. This means that -g will dump .dSYM folders on Mac, but I think that's acceptable since it's a standard practice. In the tutorials for LLVM's DwarfBuilder API, there are some platform configuration steps that we don't seem to be doing. Perhaps doing those will handle this out-of-the-box.
• We could try to force LLVM to embed the DWARF into the final binary. This would allow us to produce only one artifact when compiling with -g. However, this goes against precedent on OSX, which is why I'm not a fan of the idea, even if it might make our/users lives a bit more simple. As well, it might even be the case that the OSX linker could reject the binary due to the DWARF (who knows).

I'd like to look into having us generate a .dSYM archive on OSX just like Clang does. Hopefully all I have to do is further configure LLVM.

[jabraham17]

Do you remember the commands you gave to llvm-dwarfdump? Was it just a basic llvm-dwarfdump -a

Yes and it looks like you saw the same thing I did, the frame entries.