[lldb][rfc] rebuilding `_lldb`/`liblldb.so` for different Python versions quickly

[arcivanov]

The LLDB's Python package is (generally) as follows:

$ pwd
/usr/lib64/python3.12/site-packages/lldb
$ tree -I __pycache__
.
├── embedded_interpreter.py
├── formatters
│   ├── attrib_fromdict.py
│   ├── cache.py
│   ├── cpp
│   │   ├── gnu_libstdcpp.py
│   │   ├── __init__.py
│   │   └── libcxx.py
│   ├── __init__.py
│   ├── Logger.py
│   ├── metrics.py
│   └── synth.py
├── __init__.py
├── lldb-argdumper -> ../../../../bin/lldb-argdumper
├── _lldb.cpython-312-x86_64-linux-gnu.so -> ../../../liblldb.so <**************
├── _lldb.so -> ../../../liblldb.so.18.1.8
├── plugins
│   ├── __init__.py
│   ├── operating_system.py
│   ├── scripted_platform.py
│   └── scripted_process.py
└── utils
    ├── in_call_stack.py
    ├── __init__.py
    └── symbolication.py

5 directories, 21 files

Out of all, the _lldb is the only native extension and it is actually the liblldb.so. Which Python is used is determined by setting PYTHON_HOME during the initial cmake run.

I'd like to rebuild that extension, i.e. liblldb.so, quickly for different versions of Python, since if I just reconfigure cmake with a different version of Python it'll make me rebuild the whole clang and lldb too and it will take hours and even worse if I have a 2-stage build then it'll take days.

Any suggestions?

[llvmbot]

@llvm/issue-subscribers-lldb

Author: Arcadiy Ivanov (arcivanov)

The LLDB's Python package is (generally) as follows: ```bash $ pwd /usr/lib64/python3.12/site-packages/lldb $ tree -I __pycache__ . ├── embedded_interpreter.py ├── formatters │   ├── attrib_fromdict.py │   ├── cache.py │   ├── cpp │   │   ├── gnu_libstdcpp.py │   │   ├── __init__.py │   │   └── libcxx.py │   ├── __init__.py │   ├── Logger.py │   ├── metrics.py │   └── synth.py ├── __init__.py ├── lldb-argdumper -> ../../../../bin/lldb-argdumper ├── _lldb.cpython-312-x86_64-linux-gnu.so -> ../../../liblldb.so <************** ├── _lldb.so -> ../../../liblldb.so.18.1.8 ├── plugins │   ├── __init__.py │   ├── operating_system.py │   ├── scripted_platform.py │   └── scripted_process.py └── utils ├── in_call_stack.py ├── __init__.py └── symbolication.py 5 directories, 21 files ``` Out of all, the `_lldb` is the only native extension and it is actually the `liblldb.so`. Which Python is used is determined by setting `PYTHON_HOME` during the initial `cmake` run. I'd like rebuild that extension, i.e. `liblldb.so`, quickly for different versions of Python, since if I just reconfigure `cmake` with a different version of Python it'll make me rebuild the whole clang and lldb too and it will take hours and even worse if I have a 2-stage build (then it'll take days). Any suggestions?

[bulbazord]

There are 2 ways of building LLDB. The first way is likely what you're doing, which is building LLVM, Clang, and LLDB together in one unified build. The second way is to build LLVM and Clang separately and then use them as external libraries to build LLDB. The easiest way to do what you want today is configure several LLDB builds, each for a different version of python.

[arcivanov]

The second way is to build LLVM and Clang separately and then use them as external libraries to build LLDB.

Is this the "standalone" build? https://lldb.llvm.org/resources/build.html#standalone-builds

This requires two build trees to begin with. Am I missing something? Also, this process doesn't seem to be compatible with a 2-stage build.

I so far found that using heavy CCACHEing reduces the lldb rebuilds in a 2-stage build and 1 llvm + core + lldb * 5 pythons takes less than 2 hours on native. That said, it'll mean that it will take about 12 hours emulating aarch64, which is going to suck. :cry:

[bulbazord]

The second way is to build LLVM and Clang separately and then use them as external libraries to build LLDB.

Is this the "standalone" build? https://lldb.llvm.org/resources/build.html#standalone-builds

Yes.

This requires two build trees to begin with. Am I missing something? Also, this process doesn't seem to be compatible with a 2-stage build.

I so far found that using heavy CCACHEing reduces the lldb rebuilds in a 2-stage build and 1 llvm + core + lldb * 5 pythons takes less than 2 hours on native. That said, it'll mean that it will take about 12 hours emulating aarch64, which is going to suck. 😢

I'm not sure why you need a 2-stage build for LLDB? If you built LLDB standalone, you could be performing a 2-stage build with LLVM+Clang and then use that to build LLDB. LLDB is not set up to build against multiple versions of python, standalone builds are as close as you're going to get at this time. Technically only a small subset of LLDB itself uses Python directly, but some of the parts use Python headers (which would need a rebuild per version) and then the final shared object/dylib would need to link against a specific python library.

That being said, if you felt sufficiently motivated, you could modify lldb's build to support building against multiple versions of Python. I don't know how much work that would be to do though.