rust-lldb finds wrong source map (from build directory)

[n8henrie]

Describe the bug

When debugging rust projects with lldb on x86_64 NixOS, the project's source info is set to the temporary build directory -- which no longer exists -- and therefore requires manual intervention to get the source code to appear in the debugger.

This doesn't seem to be the case for gdb, so I'm wondering if this is a fixable bug, though perhaps as a consequence of the build sandbox this is unfixable.

Discussed in greater length for aarch64-darwin in https://github.com/NixOS/nixpkgs/issues/262131, though there seem to be other issues at play there.

Steps To Reproduce

$ nix shell nixpkgs#cargo nixpkgs#lldb nixpkgs#rustc
$ cargo new foo
$ cd foo
$ cargo update
$ 
$ cat <<'EOF' > flake.nix
{
    inputs.nixpkgs.url = "github:nixos/nixpkgs";
    outputs = {nixpkgs, ...}: let
        system = "x86_64-linux";
        pkgs = import nixpkgs {inherit system;};
    in {
        packages.x86_64-linux.default = pkgs.rustPlatform.buildRustPackage {
        name = "foo";
        src = ./.;
        cargoLock.lockFile = ./Cargo.lock;
        buildType = "debug";
        dontStrip = true;
        };
    };
}
EOF
$
$ git add .
$ nix build
$
$ rust-lldb -o 'break set --name foo::main' -o run result/bin/foo
(lldb) command script import "/nix/store/dfb6kic3xdp2467l5mjk61dfmmclx6jy-rustc-1.73.0/lib/rustlib/etc/lldb_lookup.py"
(lldb) command source -s 0 '/nix/store/dfb6kic3xdp2467l5mjk61dfmmclx6jy-rustc-1.73.0/lib/rustlib/etc/lldb_commands'
Executing commands in '/nix/store/dfb6kic3xdp2467l5mjk61dfmmclx6jy-rustc-1.73.0/lib/rustlib/etc/lldb_commands'.
(lldb) type synthetic add -l lldb_lookup.synthetic_lookup -x ".*" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)String$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^&(mut )?str$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^&(mut )?\\[.+\\]$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(std::ffi::([a-z_]+::)+)OsString$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)Vec<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)VecDeque<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)BTreeSet<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)BTreeMap<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(std::collections::([a-z_]+::)+)HashMap<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(std::collections::([a-z_]+::)+)HashSet<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)Rc<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)Arc<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(core::([a-z_]+::)+)Cell<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(core::([a-z_]+::)+)Ref<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(core::([a-z_]+::)+)RefMut<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(core::([a-z_]+::)+)RefCell<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^core::num::([a-z_]+::)*NonZero.+$" --category Rust
(lldb) type category enable Rust
(lldb) target create "result/bin/foo"
Current executable set to '/tmp/tmp.8AsZiWTLKL/foo/result/bin/foo' (x86_64).
(lldb) break set --name foo::main
Breakpoint 1: where = foo`foo::main::h71a2af43dbbc2e1a + 4 at main.rs:2:5, address = 0x0000000000008c24
(lldb) run
Process 974805 stopped
* thread #1, name = 'foo', stop reason = breakpoint 1.1
    frame #0: 0x000055555555cc24 foo`foo::main::h71a2af43dbbc2e1a at main.rs:2:5
Process 974805 launched: '/tmp/tmp.8AsZiWTLKL/foo/result/bin/foo' (x86_64)
(lldb) 

Expected behavior

As per below, which starts with the same steps as above, but once at a breakpoint runs source info. At this point it prints out /build/dii8zb390806spk2k9bkd5qasv9hwzsg-source/src/main.rs:2:5 -- the build directory in the nix store (?) -- and from here I can remap that directory to the current directory: settings set target.source-map /build/dii8zb390806spk2k9bkd5qasv9hwzsg-source ., and upon a subsequent run, we see the source code showing up at breakpoints as anticipated.

$ rust-lldb -o 'break set --name foo::main' -o run result/bin/foo
(lldb) command script import "/nix/store/dfb6kic3xdp2467l5mjk61dfmmclx6jy-rustc-1.73.0/lib/rustlib/etc/lldb_lookup.py"
(lldb) command source -s 0 '/nix/store/dfb6kic3xdp2467l5mjk61dfmmclx6jy-rustc-1.73.0/lib/rustlib/etc/lldb_commands'
Executing commands in '/nix/store/dfb6kic3xdp2467l5mjk61dfmmclx6jy-rustc-1.73.0/lib/rustlib/etc/lldb_commands'.
(lldb) type synthetic add -l lldb_lookup.synthetic_lookup -x ".*" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)String$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^&(mut )?str$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^&(mut )?\\[.+\\]$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(std::ffi::([a-z_]+::)+)OsString$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)Vec<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)VecDeque<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)BTreeSet<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)BTreeMap<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(std::collections::([a-z_]+::)+)HashMap<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(std::collections::([a-z_]+::)+)HashSet<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)Rc<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(alloc::([a-z_]+::)+)Arc<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(core::([a-z_]+::)+)Cell<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(core::([a-z_]+::)+)Ref<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(core::([a-z_]+::)+)RefMut<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^(core::([a-z_]+::)+)RefCell<.+>$" --category Rust
(lldb) type summary add -F lldb_lookup.summary_lookup  -e -x -h "^core::num::([a-z_]+::)*NonZero.+$" --category Rust
(lldb) type category enable Rust
(lldb) target create "result/bin/foo"
Current executable set to '/tmp/tmp.8AsZiWTLKL/foo/result/bin/foo' (x86_64).
(lldb) break set --name foo::main
Breakpoint 1: where = foo`foo::main::h71a2af43dbbc2e1a + 4 at main.rs:2:5, address = 0x0000000000008c24
(lldb) run
Process 1062773 stopped
* thread #1, name = 'foo', stop reason = breakpoint 1.1
    frame #0: 0x000055555555cc24 foo`foo::main::h71a2af43dbbc2e1a at main.rs:2:5
Process 1062773 launched: '/tmp/tmp.8AsZiWTLKL/foo/result/bin/foo' (x86_64)
(lldb) source info
Lines found in module `foo
[0x000055555555cc24-0x000055555555cc45): /build/dii8zb390806spk2k9bkd5qasv9hwzsg-source/src/main.rs:2:5
(lldb) settings set target.source-map /build/dii8zb390806spk2k9bkd5qasv9hwzsg-source .
(lldb) r
There is a running process, kill it and restart?: [Y/n] y
Process 1062773 exited with status = 9 (0x00000009) 
Process 1062990 launched: '/tmp/tmp.8AsZiWTLKL/foo/result/bin/foo' (x86_64)
Process 1062990 stopped
* thread #1, name = 'foo', stop reason = breakpoint 1.1
    frame #0: 0x000055555555cc24 foo`foo::main::h71a2af43dbbc2e1a at main.rs:2:5
   1    fn main() {
-> 2        println!("Hello, world!");
   3    }

Additional context

rust-gdb does not seem to require this workaround. Perhaps there is some way in which they are configured differently?

$ rust-gdb result/bin/foo
GNU gdb (GDB) 13.2
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from result/bin/foo...
(gdb) break foo::main
Breakpoint 1 at 0x8c24: file src/main.rs, line 2.
(gdb) r
Starting program: /nix/store/8mynxj3dh9wl6ay4aqc75srxnlrda6s4-foo/bin/foo 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/nix/store/j6mwswpa6zqhdm1lm2lv9iix3arn774g-glibc-2.38-27/lib/libthread_db.so.1".

Breakpoint 1, foo::main () at src/main.rs:2
warning: Source file is more recent than executable.
2           println!("Hello, world!");
(gdb) 

Notify maintainers

@llvm_meta.maintainers @teams.rust.members

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.6.11, NixOS, 23.11 (Tapir), 23.11.20240120.c5b6c17`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.19.2`
 - channels(root): `""`
 - nixpkgs: `/nix/store/6fsjig9qcdw9wds81hx5yn90b4lcpmdc-source`

---

Add a :+1: reaction to issues you find important.

[n8henrie]

demo: https://github.com/n8henrie/nix-rust-debug-info

TLDR: a function that extracts the proper source directory and wraps rust-lldb:

#!/usr/bin/env bash

set -Eeuf -o pipefail

errexit() {
  echo "$*" >&2
  exit 1
}

main() {
  local binary=$1
  local builddir=$(
    local plat=$(uname -s)
    case "${plat}" in
      Linux)
        readelf --debug-dump=info "${binary}" |
          awk '/DW_AT_comp_dir/ { print $NF; exit }'
        ;;
      Darwin)
        dsymutil --symtab "${binary}" |
          awk -F"'" '
            /N_OSO/ {
              buildfile=$(NF-1)
              gsub("/target/.*", "", buildfile);
              print buildfile
              exit
            }
          '
        ;;
      *)
        errexit "unknown platform: ${plat}"
        ;;
    esac
  )

  [[ -z "${builddir}" ]] && errexit "could not find builddir"

  rust-lldb -o "settings set target.source-map ${builddir} ." "${binary}"
}

main "$@"

[n8henrie]

It looks like something for this is already in rust:

• https://github.com/rust-lang/rust/pull/87320
• https://doc.rust-lang.org/rustc/command-line-arguments.html#--remap-path-prefix-remap-source-names-in-output

...and this has already been incorporated into nix! https://github.com/NixOS/nixpkgs/blob/ba6e3dc802f4035a3346fc747c2436198ecfa9e9/pkgs/build-support/rust/build-rust-crate/build-crate.nix#L19

Why isn't this working?

[n8henrie]

Hmmm....

For my reference:

• split-debuginfo=off doesn't seem to provide usable debug data for macos
• split-debuginfo=packed creates a separate .dSYM file and works
  • RUSTFLAGS='-C split-debuginfo=packed' cargo build; ls -ld target/debug/*.dSYM
  • The .dSYM file is a symlink to something in target/debug/deps/
• split-debuginfo=unpacked also works, seems to keep the debug info into compiled artifacts in target/debug/deps/ (such as .rcgu.o files)

Confusingly, on macOS cargo sets the default for split-debuginfo to unpacked whereas rustc sets it to packed

• https://doc.rust-lang.org/cargo/reference/profiles.html#split-debuginfo
• https://doc.rust-lang.org/rustc/codegen-options/index.html#split-debuginfo

Inspecting with buildType = "debug"; and cargoBuildFlags = [ "--verbose" ];, I can confirm I see -C debuginfo=2 -C split-debuginfo=unpacked .

So with unpacked, I think the debug info ends up in the deps/ files which don't end up in $out. To confirm:

$ dsymutil --symtab result/bin/nix-rust-debug-info | grep N_OSO
[   813] 0000fb19 66 (N_OSO        ) 00     0001   0000000000000000 '/private/tmp/nix-build-nix-rust-debug-info.drv-2/xdiv1729xdhznr5wbwmfqm4gnvknn6f9-source/target/aarch64-apple-darwin/debug/deps/nix_rust_debug_info-71
8d3ceb29e315fb.18che7f54zm6gqso.rcgu.o'
[   821] 0000fc38 66 (N_OSO        ) 00     0001   0000000000000000 '/private/tmp/nix-build-nix-rust-debug-info.drv-2/xdiv1729xdhznr5wbwmfqm4gnvknn6f9-source/target/aarch64-apple-darwin/debug/deps/nix_rust_debug_info-71
8d3ceb29e315fb.1oa4gpzp0zfcdeft.rcgu.o'
[   842] 0000fe65 66 (N_OSO        ) 00     0001   0000000000000000 '/private/tmp/nix-build-nix-rust-debug-info.drv-2/xdiv1729xdhznr5wbwmfqm4gnvknn6f9-source/target/aarch64-apple-darwin/debug/deps/nix_rust_debug_info-71
8d3ceb29e315fb.1u92galu87f7lcau.rcgu.o'
[   850] 0000ff8b 66 (N_OSO        ) 00     0001   0000000000000000 '/private/tmp/nix-build-nix-rust-debug-info.drv-2/xdiv1729xdhznr5wbwmfqm4gnvknn6f9-source/target/aarch64-apple-darwin/debug/deps/nix_rust_debug_info-71
8d3ceb29e315fb.2msraqt397prfqsg.rcgu.o'
[   866] 000100cc 66 (N_OSO        ) 00     0001   0000000000000000 '/private/tmp/nix-build-nix-rust-debug-info.drv-2/xdiv1729xdhznr5wbwmfqm4gnvknn6f9-source/target/aarch64-apple-darwin/debug/deps/nix_rust_debug_info-71
8d3ceb29e315fb.2oelg7ywzuj60ugz.rcgu.o'
[   878] 00010210 66 (N_OSO        ) 00     0001   0000000000000000 '/private/tmp/nix-build-nix-rust-debug-info.drv-2/xdiv1729xdhznr5wbwmfqm4gnvknn6f9-source/target/aarch64-apple-darwin/debug/deps/nix_rust_debug_info-71
8d3ceb29e315fb.47i1c26r66vm3xf8.rcgu.o'

Also, of note, RUSTFLAGS="--remap-path-prefix ${PWD}=/build" cargo build seems to work as expected to remap $PWD to /build:

$ cargo clean
$ RUSTFLAGS="--remap-path-prefix ${PWD}=/build" cargo build
$ rust-lldb --batch -o 'break set --name foo::main' -o 'process launch' -o 'source info' target/debug/foo
...
(lldb) source info
Lines found in module `foo
[0x0000000100004810-0x000000010000482c): /build/src/main.rs:2:5

$ RUSTFLAGS="--remap-path-prefix ${PWD}=/highly-unlikely-string" cargo build
$ rg -uuul -F '/highly-unlikely-string' target/
target/debug/deps/foo-104d16c40a5761e7.2yejoq0akawcb9o2.rcgu.o
target/debug/deps/foo-104d16c40a5761e7.55foaeum9yrlg7s2.rcgu.o
target/debug/deps/foo-104d16c40a5761e7.35cl6so44x3231dy.rcgu.o
target/debug/deps/foo-104d16c40a5761e7.37hqk34l78jf1qs2.rcgu.o
target/debug/deps/foo-104d16c40a5761e7.ympa61ftupvfkb1.rcgu.o
target/debug/deps/foo-104d16c40a5761e7.3kncu59apmxfb44i.rcgu.o
target/debug/.fingerprint/foo-104d16c40a5761e7/bin-foo.json
target/debug/incremental/foo-3upiklm9rf188/s-gxx4d27taq-1hdm7r1-340r2c74hidhclhgbloj1lps/ympa61ftupvfkb1.o
target/debug/incremental/foo-3upiklm9rf188/s-gxx4d27taq-1hdm7r1-340r2c74hidhclhgbloj1lps/35cl6so44x3231dy.o
target/debug/incremental/foo-3upiklm9rf188/s-gxx4d27taq-1hdm7r1-340r2c74hidhclhgbloj1lps/3kncu59apmxfb44i.o
target/debug/incremental/foo-3upiklm9rf188/s-gxx4d27taq-1hdm7r1-340r2c74hidhclhgbloj1lps/2yejoq0akawcb9o2.o
target/debug/incremental/foo-3upiklm9rf188/s-gxx4d27taq-1hdm7r1-340r2c74hidhclhgbloj1lps/55foaeum9yrlg7s2.o
target/debug/incremental/foo-3upiklm9rf188/s-gxx4d27taq-1hdm7r1-340r2c74hidhclhgbloj1lps/37hqk34l78jf1qs2.o
target/debug/deps/foo-104d16c40a5761e7
target/debug/foo

Also of note, testing seems complicated by macOS hiding $TMPDIR in /private:

$ bash -c 'cd $(mktemp -d); pwd; realpath .'
/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.xCCkjfeE7H
/private/var/folders/kb/tw_lp_xd2_bbv0hqk4m0bvt80000gn/T/tmp.xCCkjfeE7H

So I have to use --remap-path-prefix /private${PWD}=, and even that seems to only work if empty?

It seems that $NIX_BUILD_TOP evaluates to /private/tmp/nix-build-* so hopefully that is taken care of.

For some reason the grep N_OSO on dsymutil output doesn't show the "new" directory specified by remap-path-prefix; it seems like N_SO might be better:

$ dsymutil --symtab ./target/debug/foo | grep -F highly-unlikely-string
[   466] 0000be20 64 (N_SO         ) 00     0000   0000000000000000 '/highly-unlikely-string/src/main.rs/@/'
[   474] 0000bf0f 64 (N_SO         ) 00     0000   0000000000000000 '/highly-unlikely-string/src/main.rs/@/'
[   482] 0000bfa6 64 (N_SO         ) 00     0000   0000000000000000 '/highly-unlikely-string/src/main.rs/@/'
[   502] 0000c03d 64 (N_SO         ) 00     0000   0000000000000000 '/highly-unlikely-string/src/main.rs/@/'
[   514] 0000c0d4 64 (N_SO         ) 00     0000   0000000000000000 '/highly-unlikely-string/src/main.rs/@/'
[   526] 0000c16b 64 (N_SO         ) 00     0000   0000000000000000 '/highly-unlikely-string/src/main.rs/@/'