Document how to debug an application built with uefi-rs

[nicholasbishop]

Splitting this out from https://github.com/rust-osdev/uefi-rs/issues/285:

@timrobertsdev wrote:

I've been attempting to investigate why, but I've been having a little trouble getting rust-gdb and rust-lldb to recognize the symbols in the uefi-test-runner.efi binary. Oddly enough, objdump -freports that it is flagged as HAS_DEBUG, but objdump -t wasn't able to find any symbols. Attemping image dump symtab in rust-lldb results in num-symbols = 0. I've probably got something setup incorrectly on my end, any advice would be greatly appreciated.

I don't know the answer to this, but it would be nice to have a straightforward way to debug an application built with uefi-rs running under qemu. I'm not sure if there is something that currently works and it just needs some documentation, or if there's work needed in the toolchain.

[timrobertsdev]

I found this article, but their approach doesn't seem to work. It finds the base address and address of each section, but still doesn't provide any symbols to gdb. Then I found a uefi_test_runner_xxxx.pdb file alongside a uefi_test_runner_xxxx.efi file in target/{arch}/debug/deps/, but lldb doesn't seem to want to load it, maybe the location of the .pdb isn't in the executable? I'm starting to lean towards it being a toolchain issue. I'll try building EDK2 and OVMF using their LLVM/Clang toolchain and see if it outputs usable debug info first.

[no92]

Previous versions of rustc and/or cargo worked flawlessly as described in the article above, but this doesn't work with recent version. I don't remember whether the .pdb files were always there, but in any case they don't include any symbols now.

IIRC, the spec requires that UEFI apps tick a few boxes, e.g. the subsystem type, but it also states that they ought to be stripped, hence this behavior.

[timrobertsdev]

The executable should be stripped, but applications built in C with EDK2 in DEBUG mode do output .map and .pdb files for debugging. Source: https://edk2-docs.gitbook.io/edk-ii-build-specification/12_build_changes_and_customizations/121_building_for_debug

With that in mind, it sounds like this might be worth filing a bug for the x86_64-unknown-uefi target?

[no92]

I'm in the process of bisecting compiler versions to determine where the old behavior of having symbol tables was lost.

[no92]

The last build that provides symbol tables is nightly-2020-11-14-x86_64-unknown-linux-gnu, while nightly-2020-11-15-x86_64-unknown-linux-gnu doesn't.

I'll try to do some digging as to why that happens, but I don't know the first thing about rustc or cargo code so I don't expect to find the solution TBH.

Methodology

I took uefi-test-runner @ v0.6.0 as a testbed. Rust nightlies tried ranged from 2020-10-20 to 2021-10-04. For getting versions of Rust, I used these commands, substituting the dates accordingly:

rustup toolchain add nightly-2020-11-14 --profile minimal
rustup default nightly-2020-11-14
rustup component add rust-src

For building, ./build.py build was used. In order to detect whether symbol tables are present, the following commands were run in gdb via a .gdbinit file:

add-symbol-file ../target/x86_64-unknown-uefi/debug/uefi-test-runner.efi
info functions

[no92]

Bisecting leads to rust-lang/rust#78959, which is also linked to by rust-lang/rust#87157 where the author of the blog post above reports the same issue.

The solution seems to be to do rustc +nightly -Z unstable-options --print target-spec-json --target x86_64-unknown-uefi | sed -e 's/"is-like-msvc": true/"is-like-msvc": false/g' > x86_64-unknown-uefi-debug.json.

[timrobertsdev]

Can confirm the above solution works, with the additional step of removing "is-builtin" from the target file. build.py would need to be updated to use the custom target as well.

I'll dive into debugging something in QEMU tonight, but this looks like it solves the issue completely as gdb can see all symbols and map them back to the source files.

[nicholasbishop]

I'm not seeing quite the same thing as you with the builtin x86_64-unknown-uefi target. I do get what seems to be a valid pdb file generated under the deps directory. I dumped the symbols in it with the pdb_symbols utility in the pdb crate, and it looks correct. (rustc 1.57.0-nightly (9dbb26efe 2021-10-03.)

So if I'm understanding correctly, rustc used to include DWARF-style debug sections in the executable, and now it outputs a pdb. Which does seem like reasonable behavior.

lldb seems like it supports loading pdb files (either with a Windows-specific loader, or the "native pdb" plugin on other platforms), but in testing on Linux I haven't had any luck getting it to successfully load symbols.

[timrobertsdev]

Yeah, that's the same issue that I was having with the pdb files. I tested on both linux and windows(mingw) and was unable to correctly load any symbols. I'll give MSVC a try.

With @no92's solution, gdb and lldb in linux are able to recognize the symbols enough to say they have successfully set a breakpoint, but they won't actually break with br or hbr. Maybe the addresses aren't correct and need to be fixed up before debugging?

EDIT: Tried to fixup the addresses using the above article, but it segfaulted while running on my machine.

[no92]

@nicholasbishop wrote:

I do get what seems to be a valid pdb file generated under the deps directory. I dumped the symbols in it with the pdb_symbols utility in the pdb crate, and it looks correct.

I had checked it with some pdb tool I frankly can't remember right now, and I didn't find anything resembling symbols. It was, however, a valid pdb file. As it was some random tool pulled from GitHub, it could just be wrong and the Rust crate right. Oops.

@timrobertsdev wrote:

With @no92's solution, gdb and lldb in linux are able to recognize the symbols enough to say they have successfully set a breakpoint, but they won't actually break with br or hbr. Maybe the addresses aren't correct and need to be fixed up before debugging?

As EFI apps are relocatable, the base address changes. This could be read out via the LoadedImage Protocol, but transferring that info from the remote to the debugger host would have to take place somehow. Alternatively, the instructions in the article work for me, I'm not sure why it's segfaulting for you. I use the python script via a .gdbinit in gdb itself as well as some VSCode interface that ultimately uses gdb, too.

[timrobertsdev]

@no92 Nevermind, I got it working. The mistake I was making was attempting to run the dbg gdbscript while QEMU itself was halted and waiting for debugger, instead of the wait_for_debugger() fn outlined in that article.

[vitamiin]

I believe I've taken all the steps mentioned above yet the problem persists -- GDB is not able to recognize any symbols (which I doubt exist). I might've missed something. Any help would be much appreciated.

What I did:

• Created a custom target, and in its .json file, I removed the is_builtin line, as well as set is-like-msvc to true.
• Rolled back the nightly version to nightly-2020-11-14 (used rustup default and updated rust-toolchain.toml)
• Built the project using the custom target (which is specified in my .cargo/config file and was indeed recognized)

[no92]

While I haven't used this in a while, rolling back to an old nightly version should not be necessary. Using the script provided in the blog post linked above is necessary IIRC.

[vitamiin]

Unfortunately, it still doesn't work for me. We're talking about load_symbols.py, right?

[vitamiin]

I have successfully solved the problem by adding the following build (under the [build] section) flag in .cargo/config:

"-C", "link-args=/debug:dwarf"

Thanks to the following video: https://www.youtube.com/watch?v=2YAgDJTs9So

This might be trivial to some however I wasn't aware of this link argument in the first place. All symbols are now recognized by gdb.

[DianaNites]

How did you get it working with gdb? I can't seem to figure it out

[phip1611]

"-C", "link-args=/debug:dwarf"

This enables for example objdump to displays debug symbols. gdb, however, still doesn't find any symbols in the corresponding efi file.

[GabrielMajeri]

I've been digging around this issue recently, here's what I've discovered so far. I've been trying to use LLDB, since it is the only debugger which can read PDB files (in theory) and connect to QEMU using GDB's remote debug protocol.

I've tried to follow along with the instructions from this article on the osdev.org wiki. I've modified the xtask/src/qemu.rs file to pass the -s flag to QEMU (i.e. I've added a line with cmd.arg("-s");) in order to activate the GDB debug port, then I've modified the uefi-test-runner/src/main.rs file to display the address at which the binary is loaded:

{
    let loaded_image = st.boot_services()
        .open_protocol_exclusive::<LoadedImage>(image)
        .expect("Failed to open LoadedImage protocol");
    println!("Image base: {:#X}", loaded_image.info().0 as usize);
}

let wait = true;
while wait { }

When running the app with cargo xtask run, it displays a message similar to Image base: 0xDD89000.

Now, I ran the following commands in LLDB, from the target/x86_64-unknown-uefi/debug/deps directory:

• target create uefi_test_runner-e4bed4b0ac1a4cab.efi — loads the EFI binary and sets it as the default debug target.
• target symbols add uefi_test_runner-e4bed4b0ac1a4cab.pdb — loads the debug symbols from the PDB file.
• gdb-remote localhost:1234 — connects to the running QEMU instance (Note: sometimes I had to run it twice to get it to work).
• process continue — allow the code to run until it gets into the infinite loop we've set up in the main function.
• process interrupt — pause inside the while loop inside the main function.
• image load --file uefi_test_runner-e4bed4b0ac1a4cab.efi .text 0xDD89000 — tell LLDB at which address the EFI binary's .text section starts.

At this point, everything should be set up for debugging. Unfortunately, LLDB just crashes:

PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: lldb
 #0 0x00007f9ba8e3fd01 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0xe3fd01)
 #1 0x00007f9ba8e3da3e llvm::sys::RunSignalHandlers() (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0xe3da3e)
 #2 0x00007f9ba8e40236 (/lib/x86_64-linux-gnu/libLLVM-14.so.1+0xe40236)
 #3 0x00007f9ba7842520 (/lib/x86_64-linux-gnu/libc.so.6+0x42520)
 #4 0x00007f9bb2ca5f75 (/lib/x86_64-linux-gnu/liblldb-14.so.1+0xaa5f75)
 #5 0x00007f9bb2caa78e (/lib/x86_64-linux-gnu/liblldb-14.so.1+0xaaa78e)
 #6 0x00007f9bb2cab262 (/lib/x86_64-linux-gnu/liblldb-14.so.1+0xaab262)
 #7 0x00007f9bb2ca6c91 (/lib/x86_64-linux-gnu/liblldb-14.so.1+0xaa6c91)
 #8 0x00007f9bb2ca6d32 (/lib/x86_64-linux-gnu/liblldb-14.so.1+0xaa6d32)
 #9 0x00007f9bb2caa613 (/lib/x86_64-linux-gnu/liblldb-14.so.1+0xaaa613)
#10 0x00007f9bb2cab2ef (/lib/x86_64-linux-gnu/liblldb-14.so.1+0xaab2ef)
#11 0x00007f9bb2ca6c91 (/lib/x86_64-linux-gnu/liblldb-14.so.1+0xaa6c91)
#12 0x00007f9bb2cab7a0 (/lib/x86_64-linux-gnu/liblldb-14.so.1+0xaab7a0)
#13 0x00007f9bb2cab3d6 (/lib/x86_64-linux-gnu/liblldb-14.so.1+0xaab3d6)
#14 0x00007f9bb2ca6c91 (/lib/x86_64-linux-gnu/liblldb-14.so.1+0xaa6c91)
#15 0x00007f9bb2c88d16 (/lib/x86_64-linux-gnu/liblldb-14.so.1+0xa88d16)
#16 0x00007f9bb2c8936d (/lib/x86_64-linux-gnu/liblldb-14.so.1+0xa8936d)
#17 0x00007f9bb2c8587b (/lib/x86_64-linux-gnu/liblldb-14.so.1+0xa8587b)
#18 0x00007f9bb2c85263 (/lib/x86_64-linux-gnu/liblldb-14.so.1+0xa85263)
#19 0x00007f9bb2c8b078 (/lib/x86_64-linux-gnu/liblldb-14.so.1+0xa8b078)
#20 0x00007f9bb260e6ec (/lib/x86_64-linux-gnu/liblldb-14.so.1+0x40e6ec)
#21 0x00007f9bb25de1ba (/lib/x86_64-linux-gnu/liblldb-14.so.1+0x3de1ba)
#22 0x00007f9bb25de004 (/lib/x86_64-linux-gnu/liblldb-14.so.1+0x3de004)
#23 0x00007f9bb27f5f1e (/lib/x86_64-linux-gnu/liblldb-14.so.1+0x5f5f1e)
#24 0x00007f9bb27f5b90 (/lib/x86_64-linux-gnu/liblldb-14.so.1+0x5f5b90)
#25 0x00007f9bb27e8d82 (/lib/x86_64-linux-gnu/liblldb-14.so.1+0x5e8d82)
#26 0x00007f9bb27ea36f (/lib/x86_64-linux-gnu/liblldb-14.so.1+0x5ea36f)
#27 0x00007f9bb2781b2f (/lib/x86_64-linux-gnu/liblldb-14.so.1+0x581b2f)
#28 0x00007f9bb2783c29 (/lib/x86_64-linux-gnu/liblldb-14.so.1+0x583c29)
#29 0x00007f9bb27ba671 (/lib/x86_64-linux-gnu/liblldb-14.so.1+0x5ba671)
#30 0x00007f9bb274184c (/lib/x86_64-linux-gnu/liblldb-14.so.1+0x54184c)
#31 0x00007f9bb25eb03f (/lib/x86_64-linux-gnu/liblldb-14.so.1+0x3eb03f)
#32 0x00007f9bb26c1912 (/lib/x86_64-linux-gnu/liblldb-14.so.1+0x4c1912)
#33 0x00007f9bb260a83f (/lib/x86_64-linux-gnu/liblldb-14.so.1+0x40a83f)
#34 0x00007f9bb25eb73c (/lib/x86_64-linux-gnu/liblldb-14.so.1+0x3eb73c)
#35 0x00007f9bb26c33d9 (/lib/x86_64-linux-gnu/liblldb-14.so.1+0x4c33d9)
#36 0x00007f9bb23f5eda lldb::SBDebugger::RunCommandInterpreter(bool, bool) (/lib/x86_64-linux-gnu/liblldb-14.so.1+0x1f5eda)
#37 0x0000000000407d4a (/usr/lib/llvm-14/bin/lldb+0x407d4a)
#38 0x0000000000408f85 (/usr/lib/llvm-14/bin/lldb+0x408f85)
#39 0x00007f9ba7829d90 __libc_start_call_main ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#40 0x00007f9ba7829e40 call_init ./csu/../csu/libc-start.c:128:20
#41 0x00007f9ba7829e40 __libc_start_main ./csu/../csu/libc-start.c:379:5
#42 0x0000000000404125 (/usr/lib/llvm-14/bin/lldb+0x404125)
Segmentation fault (core dumped)

I'm guessing it tries to parse the PDB file and the current stack frame, but it triggers some bug somewhere.

I haven't submitted a bug report to LLDB yet, I'm going to play around with this setup for a bit longer.

[stevefan1999-personal]

@GabrielMajeri I managed to replicate your process in VSCode and LLDB shell as well

[lf-]

The pdb files not being copied is a cargo bug: https://github.com/rust-lang/cargo/pull/5179 did not include -uefi as a target to copy pdbs for. I don't feel like filing an actual bug right now; I'm presently testing just patching it and I may file a PR later.

[FuuuOverclocking]

@GabrielMajeri This may be because the virtual address of .text in your image is not 0. You can check the virtual addresses of all sections by using llvm-readobj --sections <efi>. For instance, assumes it prints Image base: 0xDD89000 and

llvm-readobj --sections ./lab/esp/efi/boot/bootx64.efi

File: ./lab/esp/efi/boot/bootx64.efi
Format: COFF-x86-64
Arch: x86_64
AddressSize: 64bit
Sections [
  Section {
    Number: 1
    Name: .text (2E 74 65 78 74 00 00 00)
    VirtualSize: 0x11B66
    VirtualAddress: 0x1000 <-------------------------------
    RawDataSize: 72704
    PointerToRawData: 0x400
    PointerToRelocations: 0x0
    PointerToLineNumbers: 0x0
    RelocationCount: 0
    LineNumberCount: 0
    Characteristics [ (0x60000020)
      IMAGE_SCN_CNT_CODE (0x20)
      IMAGE_SCN_MEM_EXECUTE (0x20000000)
      IMAGE_SCN_MEM_READ (0x40000000)
    ]
  }
......

Then .text shoud be set to 0xDD8A000.

[FuuuOverclocking]

After a week of struggle, I finally realized that using LLDB to debug UEFI applications is far from mature. And I still cannot let the rustc and lld to output dwarf debug symbols. The line below seems to have no effect on me.

"-C", "link-args=/debug:dwarf"

[stemnic]

The following addition in .cargo/config.toml just causes the .pdb file to no longer be generated and from what I can see does not add any debug sections to the .efi file. It seems to give you function names but for example variable names remained unnamed.

[target.x86_64-unknown-uefi]
rustflags = ["-C", "link-args=/debug:dwarf"]

The only solution seems to be the following as mention earlier. Hopefully it will help someone in the future.

The solution seems to be to do rustc +nightly -Z unstable-options --print target-spec-json --target x86_64-unknown-uefi | sed -e 's/"is-like-msvc": true/"is-like-msvc": false/g' > x86_64-unknown-uefi-debug.json.

I also needed the following additions to my .cargo/config.toml

[unstable]
build-std = ["core", "compiler_builtins", "alloc"]
build-std-features = ["compiler-builtins-mem"]

With that the target (x86_64-unknown-uefi-debug.json) can then be built with nightly

cargo +nightly build --target x86_64-unknown-uefi-debug.json

The resulting .efi file will then have debug info which will work with gdb

Sections:
Idx Name          Size      VMA               LMA               File off  Algn
  0 .text         0001a098  0000000140001000  0000000140001000  00000400  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  1 .rdata        00005e5c  000000014001c000  000000014001c000  0001a600  2**4
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  2 .data         00000050  0000000140022000  0000000140022000  00020600  2**4
                  CONTENTS, ALLOC, LOAD, DATA
  3 .eh_fram      00000040  0000000140023000  0000000140023000  00020800  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  4 .reloc        000003bc  0000000140024000  0000000140024000  00020a00  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  5 .debug_abbrev 00009bc1  0000000140025000  0000000140025000  00020e00  2**0
                  CONTENTS, READONLY, DEBUGGING
  6 .debug_aranges 00020bc0  000000014002f000  000000014002f000  0002aa00  2**0
                  CONTENTS, READONLY, DEBUGGING
  7 .debug_info   00172453  0000000140050000  0000000140050000  0004b600  2**0
                  CONTENTS, READONLY, DEBUGGING
  8 .debug_line   0009145e  00000001401c3000  00000001401c3000  001bdc00  2**0
                  CONTENTS, READONLY, DEBUGGING
  9 .debug_loc    000088d9  0000000140255000  0000000140255000  0024f200  2**0
                  CONTENTS, READONLY, DEBUGGING
 10 .debug_pubnames 0006f820  000000014025e000  000000014025e000  00257c00  2**0
                  CONTENTS, READONLY, DEBUGGING
 11 .debug_pubtypes 0009f74d  00000001402ce000  00000001402ce000  002c7600  2**0
                  CONTENTS, READONLY, DEBUGGING
 12 .debug_ranges 00050ec0  000000014036e000  000000014036e000  00366e00  2**0
                  CONTENTS, READONLY, DEBUGGING
 13 .debug_str    001d3c4c  00000001403bf000  00000001403bf000  003b7e00  2**0
                  CONTENTS, READONLY, DEBUGGING