Debuginfo tests don't control for ambient tools

rust-lang / rust

Empowering everyone to build reliable and efficient software.

https://www.rust-lang.org

Other

95.24k stars 12.28k forks source link

Debuginfo tests don't control for ambient tools #126092

Open workingjubilee opened 1 month ago

workingjubilee commented 1 month ago

The debuginfo tests should probably use a specific debugger, like maybe an lldb we build with the compiler, so that they work consistently. In practice, they have been effectively broken on user's machines by just trying to use whatever is installed.

In other words: just because I use Linux doesn't mean I have gdb installed.

workingjubilee commented 1 month ago

The debuginfo test suite should also be insulated against gdb changing how it prints things, Because It Does.

Nilstrieb commented 1 month ago

Some points:

compiletest supports cdb, gdb and lldb, running the test suite for all the ones that are available (either via config.toml or via PATH).
our LLVM builds do currently not contain lldb.

workingjubilee commented 1 month ago

That's... informative, thank you!

It does seem like we should reassess our strategy if we're gonna run a test suite that isn't guaranteed to pass even between different versions of a debugger on "any ol' random debugger lying around".

bjorn3 commented 1 month ago

On macOS we can't ship our own debugger. Only an Apple signed lldb is allowed to use the debugging apis of the kernel.

workingjubilee commented 1 month ago

Wait, gdb is useless on macOS?

bjorn3 commented 1 month ago

Supposedly you can use gdb on macOS jf you sign it with a self-signed code signing certificate which you add to the system keychain, mark as trusted for code signing and set the right entitlements when signing gdb. I don't know if you need to disable SIP for this to work though. We can't disable SIP on CI. https://dev.to/jasonelwood/setup-gdb-on-macos-in-2020-489k

workingjubilee commented 1 month ago

Hmm, okay. I think on Linux it is still desirable to control for the sheer amount of variability that Linux distros have, but I can believe that the debuginfo tests have to draw on the environment sometimes.

jieyouxu commented 1 month ago

I think it's fine if we can't necessarily control for the debugger used, but we should at least document it and/or report the debugger used when running the debuginfo test suite.

workingjubilee commented 4 weeks ago

yeah, that was also reported as https://github.com/rust-lang/rust/issues/126628