Compiler selects invalid `--lldb-python` path

[Dante-Broggi]

For my mac (macOS Mojave), the --lldb-python path is being set to /usr/bin/python3, which doesn't exist. Furthermore, /usr/bin is immutable, so I cannot simply symlink to /usr/local/bin/python3, the result of which python3.

This is causing the debuginfo test suite to fail.

Code

./x.py -i test src/test/debuginfo

Meta

rust version: Top of master, specifically: https://github.com/rust-lang/rust/commit/3dbade652ed8ebac70f903e01f51cd92c4e4302c

Error output

running 115 tests
iFFFFFFFiFFFFFiFFFFFFFFFFFiFFFFiFFiiFFiFFiFFiiFiFFFFFFFFFFFFFiFiFiiFFFFFFFFiFFFFFiFiiFFFF.iFFFFiFFiF 100/115
FFFFiiFiFFFFFFF
failures:

---- [debuginfo-lldb] debuginfo/basic-types-globals-metadata.rs stdout ----
NOTE: compiletest thinks it is using LLDB version 1100
NOTE: compiletest thinks it is using LLDB without native rust support

error: Failed to setup Python process for LLDB script: No such file or directory (os error 2)
[ERROR compiletest::runtest] fatal error, panic: "Failed to setup Python process for LLDB script: No such file or directory (os error 2)"
thread 'main' panicked at 'fatal error', src/tools/compiletest/src/runtest.rs:2133:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
// …

Backtrace

``` running 115 tests iFFFFFFFiFFFFFiFFFFFFFFFFFiFFFFiFFiiFFiFFiFFiiFiFFFFFFFFFFFFFiFiFiiFFFFFFFFiFFFFFiFiiFFFF.iFFFFiFFiF 100/115 FFFFiiFiFFFFFFF failures: ---- [debuginfo-lldb] debuginfo/basic-types-globals-metadata.rs stdout ---- NOTE: compiletest thinks it is using LLDB version 1100 NOTE: compiletest thinks it is using LLDB without native rust support error: Failed to setup Python process for LLDB script: No such file or directory (os error 2) [ERROR compiletest::runtest] fatal error, panic: "Failed to setup Python process for LLDB script: No such file or directory (os error 2)" thread 'main' panicked at 'fatal error', src/tools/compiletest/src/runtest.rs:2133:9 stack backtrace: 0: ::fmt 1: core::fmt::write 2: std::io::Write::write_fmt 3: std::io::impls::>::write_fmt 4: std::sys_common::backtrace::print 5: std::panicking::default_hook::{{closure}} 6: std::panicking::default_hook 7: std::panicking::rust_panic_with_hook 8: std::panicking::begin_panic 9: compiletest::runtest::TestCx::fatal 10: compiletest::runtest::TestCx::cmd2procres 11: compiletest::runtest::TestCx::run_revision 12: compiletest::runtest::run 13: core::ops::function::FnOnce::call_once{{vtable.shim}} 14: as core::ops::function::FnOnce>::call_once 15: as core::ops::function::FnOnce>::call_once 16: __rust_maybe_catch_panic 17: std::panicking::try 18: test::run_test_in_process 19: test::run_test::run_test_inner 20: test::run_test 21: test::run_tests 22: test::console::run_tests_console 23: compiletest::main 24: std::rt::lang_start::{{closure}} 25: std::panicking::try::do_call 26: __rust_maybe_catch_panic 27: std::panicking::try 28: std::rt::lang_start_internal 29: main note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. ```

I seem to recall having problems with these tests, even before #68863.

Here's an overview of how I understand the problem:

The lldb that ships with Xcode 11 uses python3. Since 10.14 doesn't include python3, AFAIK it is not possible to use xcode's Python lldb module on Mojave (using an external interpreter). Using homebrew doesn't work either, because it is not compatible with the _lldb.so module.

EDIT: Oh, and there is a Python2 version of the lldb module at /Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/Python. The only way to discover this path would be to run lldb -P and see if it ends with "Python3" and modify the path. That might work, but seems fragile.

One option is to build lldb yourself by editing config.toml and enabling lldb. This will build lldb with python2. The bootstrap code would have to also be changed to use /usr/bin/python instead.

Another option is to use an older version of xcode (and also change the bootstrap hard-coded python3 path).

I think those options are kinda terrible. My preference would be to disable the debuginfo tests on macos if the system version is older than 10.15. I don't know how feasible that is.

@pietroalbini Does that sound like a reasonable summary and possible solution?

It's also not clear whether or not python3 will be removed in 10.16. The release notes just say python will be removed in "Future versions of macOS", but don't say which future versions. If /usr/bin/python3 is removed, I'm not sure how we'll be able to run the tests. It looks like the lldb_batchmode.py script fundamentally needs to use the lldb python module outside of lldb itself. If Apple removes python, then I think the only solution to continue running the debuginfo tests is to build lldb locally and not use the one that ships with xcode. It could also be possible that future Xcode versions will install its own python3?

[pietroalbini]

That explaination sounds plausible based on my limited understanding of the situation (not having any Apple system made writing and debugging that PR so much harder).

cc @Mark-Simulacrum on what the best approach for bootstrap would be

[Mark-Simulacrum]

I'm pretty much down for any solution in bootstrap, detecting the macOS version sounds finicky but also reasonable. I am unlikely to invest time into fixing this myself, though I would be happy to review a PR.

[golddranks]

I'm also encountering this bug. Building rustc fails on Mojave 10.14 with the same error message.

Just to get unstuck with developing rustc, is there a quick option to disable the affected tests?

[ehuss]

@golddranks What are you working on? I think most people do not run all tests, but instead run the specific test-suites that are relevant to what you are doing. I think you can also pass the --exclude flag to skip over the offending suite. Or you can try one of the solutions outlined above if you're actually working on debug-related stuff, the changes should be relatively small.

[golddranks]

@ehuss On stdlib. --exclude helped, and now I feel stupid for not finding it myself :D Thanks.