How do I debug with QEMU?

[tsatke]

I know of #258, and I see all the files that are listed in that issue (and the file output matches).

However, no file that I try to load into lldb seems to give me any symbols (both bios and uefi). With the old bootloader, I used lldb target/x86_64-mytarget/debug/mykernel and was good to go. With the current one, the best I can do is use the virtual address that the bootloader jumps to, and step through a ton of assembly, never finding the actual line that I'm searching.

What target file do I need to load? Did I maybe still miss one?

lldb-1403.0.17.64
Apple Swift version 5.8 (swiftlang-5.8.0.124.2 clang-1403.0.22.11.100)

QEMU emulator version 7.2.1
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers

bootloader 0.11.3

[u1f98e]

Finally managed to get this working, bootloader 0.11 packages your kernel into an img and loads it during the boot process rather than it just being loaded as a single binary in 0.9, which means debuggers can't line up the symbols in your kernel binary automatically.

You can get around this by giving them an offset when loading the symbols. If you have frame_buffer_logging enabled in your BootInfo, it'll tell you where the entry point is at during boot, and you can guess where the kernel is being loaded to from that (mine is loading to 0x8000000000, that's probably configurable somewhere). lldb has --slide, which takes an offset that it adds to each symbol:

(lldb) target modules load --file <your_kernel_binary> --slide 0x8000000000

EDIT: The below is irrelevant, you can actually do the same thing with gdb by providing the -o flag when calling add-symbol-file

gdb is a little trickier, it has add-symbol-file which requires the position where the .text section is being loaded rather than an offset, you can find that with readelf and add it to the load offset. The location of .text can change between builds though, so you'd probably want to automate grabbing it if you go that route:

$ readelf -S <your_kernel_binary> | grep '.text' 
   [ 8] .text             PROGBITS         000000000000fae0  0000f830

(gdb) add-symbol-file <your_kernel_binary> 0x800000fae0

# EDIT: Or just
(gdb) add-symbol-file <your_kernel_binary> -o 0x8000000000

My kernel bin is buried at target/x86_64-unknown-none/debug/deps/artifact/kernel-ce7a97fd1ed08090/bin/kernel-ce7a97fd1ed08090

It might be worth putting notes about this in a wiki page or something if it isn't already.

[tsatke]

Alright, for future reference, I'm emitting the path to the kernel binary as an env var in my build.rs file. The runner references that kernel binary and generates a file debug.lldb (per build). After that, the runner starts qemu with -s -S, and I can do lldb -s debug.lldb and start debugging. Thanks again @u1f98e for the solution.

[iTitus]

Alright, for future reference, I'm emitting the path to the kernel binary as an env var in my build.rs file. The runner references that kernel binary and generates a file debug.lldb (per build). After that, the runner starts qemu with -s -S, and I can do lldb -s debug.lldb and start debugging. Thanks again @u1f98e for the solution.

Which command did you use generate the debug build that can be fed to lldb? Which file was used for that, the full build with the bootloader or just the kernel?

[tsatke]

In my build.rs, I have println!("cargo:rustc-env=KERNEL_BINARY={}", kernel.display());.

I use that in my main.rs:

const KERNEL_BINARY: &str = env!("KERNEL_BINARY");
// --- snip ---
if args.debug {
    // create an lldb debug file to make debugging easy
    let content = format!(
        r#"target create {KERNEL_BINARY}
target modules load --file {KERNEL_BINARY} --slide 0x8000000000
gdb-remote localhost:1234
b _start
c"#
    );
    fs::write("debug.lldb", content).expect("unable to create debug file");
    println!("debug file is ready, run `lldb -s debug.lldb` to start debugging");
}

I'm using bootloader 0.11. My root crate has a CLI, so I do cargo run -- --debug in one shell, and lldb -s debug.lldb in the other shell.

[iTitus]

I get this error when trying to start lldb:

warning: failed to set breakpoint site at 0x8000040343 for breakpoint 1.1: error: 34 sending the breakpoint request
Breakpoint 1: where = kernel-71d5f5d4efab7dfb`kernel_main + 19 at main.rs:48:5, address = 0x0000008000040343

kernel_main is the entry point I register with the bootloader. Any ideas?

[tsatke]

Does the breakpoint work? Because it seems like the symbol is being resolved correctly. I also have a kernel_main, however I still break at _start in the script, then I set breakpoints once I'm in lldb. When using kernel_main instead, I don't get the warning. What versions of qemu and lldb do you use? Do you have the source code somewhere?

Maybe it's also a good idea to start a discussion instead of using this issue as a thread @iTitus .

[tsatke]

@u1f98e would you happen to know the answer to the following?

(lldb) bt
* thread #1, stop reason = signal SIGINT
  * frame #0: 0x00000080000b1f91 kernel-e5e8a967c6a90ee3`x86_64::instructions::hlt::h9db1f344477b7582 + 1
    frame #1: 0x00000080000a6791 kernel-e5e8a967c6a90ee3`kernel::arch::x86_64::panic::handle_panic::h3011e5bc69ac17ca + 65
    frame #2: 0x00000080000285a5 kernel-e5e8a967c6a90ee3`rust_begin_unwind(info=0x0000010000020e30) at main.rs:130:5
    frame #3: 0x00000080000e40a5 kernel-e5e8a967c6a90ee3`core::panicking::panic_fmt::h6b9e024da7da2cd6 at panicking.rs:72:14
    frame #4: 0x0000008000052bc2 kernel-e5e8a967c6a90ee3`kernel::mem::virt::foo::ha8557ba09293e47a + 50
    frame #5: 0x000000800002764a kernel-e5e8a967c6a90ee3`kernel::m1::h9e2e98de3695b670 at main.rs:57:5
    frame #6: 0x0000008000027636 kernel-e5e8a967c6a90ee3`kernel::m2::h6d172a2f36304da0 at main.rs:53:5
    frame #7: 0x0000008000027626 kernel-e5e8a967c6a90ee3`kernel::m3::h7f2bed2691a764e5 at main.rs:49:5
    frame #8: 0x0000008000027616 kernel-e5e8a967c6a90ee3`kernel::m4::h34eaf05cea436fdc at main.rs:45:5
    frame #9: 0x00000080000275d6 kernel-e5e8a967c6a90ee3`kernel::kernel_main::h0f9738fd899772c2(boot_info=0x0000028000000000) at main.rs:41:5
    frame #10: 0x00000080000285e7 kernel-e5e8a967c6a90ee3`_start(boot_info=0x0000028000000000) at lib.rs:131:17

Why does lldb not find the file and offset information to frame #4: 0x0000008000052bc2 kernel-e5e8a967c6a90ee3_kernel::mem::virt::foo::ha8557ba09293e47a + 50, but all the others?

[tsatke]

Hey @tsatke , I figured out the answer to your question. Add -g to your RUSTFLAGS, either RUSTFLAGS='-g' cargo build, or in the .cargo/config.toml file as

rustflags = [
    "-g"
]

[sauravdeshpande]

@u1f98e I tried following the steps to add the symbol table but it didn't work for me. I still could not set break-point at kernel_main.

[tsatke]

@sauravdeshpande how are you setting a breakpoint?

Do you have --slide set to the correct value for your kernel?

[sauravdeshpande]

@tsatke I tried setting breakpoint by b kernel_main i.e. our entry point to the kernel and also by b _start , but it still says kernel_main is not defined and for _start it takes bootloader_api symbol. I am using gdb and setting the correct offset by -o.

Edit: I referred your repo and understood where I made the mistake. I had to set the #[no_mangle] for the desired function and I did not add target in .config/config.toml because of which my other files were not getting compiled for debug.

[target.x86_64-unknown-none]
rustflags = ["-g"]

Thank you.

[hfytr]

Hello, I've basically copied @tsatke setup: https://github.com/hfytr/my_os, but it seems that bootloader doesn't generate an object file for the kernel

❯ lldb -s debug.lldb                                                                                                                         nix-shell-env
(lldb) command source -s 0 'debug.lldb'
Executing commands in '/home/fbwdw/docs/os/debug.lldb'.
(lldb) target create target/x86_64-unknown-none/debug/deps/artifact/kernel-d051de5109deb413/bin/kernel-d051de5109deb413
Current executable set to '/home/fbwdw/docs/os/target/x86_64-unknown-none/debug/deps/artifact/kernel-d051de5109deb413/bin/kernel-d051de5109deb413' (x86_64).
(lldb) target modules load --file target/x86_64-unknown-none/debug/deps/artifact/kernel-d051de5109deb413/bin/kernel-d051de5109deb413 --slide 0x80000063a0
error: no object file for module 'target/x86_64-unknown-none/debug/deps/artifact/kernel-d051de5109deb413/bin/kernel-d051de5109deb413'
error: either the "--file <module>" or the "--uuid <uuid>" option must be specified.
(lldb) c
error: Command requires a current process.

Versions:

❯ qemu-system-x86_64 --version                                                                                                               nix-shell-env
QEMU emulator version 9.1.0
Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers
❯ lldb --version                                                                                                                             nix-shell-env
lldb version 18.1.8

bootloader 11.7

[Freax13]

Does target/x86_64-unknown-none/debug/deps/artifact/kernel-d051de5109deb413/bin/kernel-d051de5109deb413 exist?

[tsatke]

I would assume not.

    const KERNEL_BINARY: &str = "target/x86_64-unknown-none/debug/deps/artifact/kernel-d051de5109deb413/bin/kernel-d051de5109deb413";
    let content = format!(
        r#"target create {KERNEL_BINARY}
target modules load --file {KERNEL_BINARY} --slide 0x8000005ec0
gdb-remote localhost:1234
b _start
c"#
    );

@hfytr you have to set the KERNEL_BINARY from an env var that is set in your build.rs. cargo won't generate the same exact name everytime.

The slide address also looks off to me. Make sure that that is the actual location where your kernel is loaded by the bootloader. This has to be the same address as in your bootloader config (if you have one - if not, I'd recommend making one) - see https://github.com/tsatke/devos/blob/8cf915d30d99de409c75a1303419bf3b7b0baf90/kernel/src/lib.rs#L54 . For me, I need to set the start and end range. Please be aware that I use a recursive mapping, not an offset mapping.

[Speak2Erase]

In my build.rs, I have println!("cargo:rustc-env=KERNEL_BINARY={}", kernel.display());.

I use that in my main.rs:

const KERNEL_BINARY: &str = env!("KERNEL_BINARY"); // --- snip --- if args.debug { // create an lldb debug file to make debugging easy let content = format!( r#"target create {KERNEL_BINARY} target modules load --file {KERNEL_BINARY} --slide 0x8000000000 gdb-remote localhost:1234 b _start c"# ); fs::write("debug.lldb", content).expect("unable to create debug file"); println!("debug file is ready, run lldb -s debug.lldb to start debugging"); }

I'm using bootloader 0.11. My root crate has a CLI, so I do cargo run -- --debug in one shell, and lldb -s debug.lldb in the other shell.

This was a lifesaver! I was struggling to get this working for over an hour.