Breakpoints work with built-in lldb but are not getting hit when using custom lldb

[juananpe]

OS: Darwin arm64 22.5.0 (macOS Ventura, M2) VSCode version: 1.81.1 CodeLLDB version: v1.9.2 Compiler: Apple clang version 14.0.3 (clang-1403.0.22.14.1) Debuggee: Mach-O 64-bit executable arm64 (a trivial hello_world application in C)

CodeLLDB is working flawlessly when using provided LLDB against the following trivial hello_world C program:

lldb version 16.0.0-custom
  rust-enabled

but trying to debug exactly the same hello_world program with a custom lldb , breakpoints are not getting hit.

I'm using a custom LLDB:

version
lldb version 18.0.0git (https://github.com/llvm/llvm-project.git revision ba818c4019c550e1a413e1563a05b241b508defd)
  clang revision ba818c4019c550e1a413e1563a05b241b508defd
  llvm revision ba818c4019c550e1a413e1563a05b241b508defd

(I replaced the contents of bin and lib in $HOME/.vscode/extensions/vadimcn.vscode-lldb-1.9.2/lldb with my llvm-project/build/bin/ and build/lib/ folders)

I also tried to debug the same program via CLI with my custom lldb and it works as expected (breakpoints included):

With the hello world C application I turned ON the verbose logging in the extension settings, to no avail (there is no output, neither using built-in lldb nor using my custom lldb):

But when trying to debug another binary (an Ada program, that's why I built my own lldb, to support Ada debugging), the logs are shown. This is the log that I get when trying to debug a simple Ada program with my custom lldb (breakpoints are not getting hit... or looking at the logging file, it seems that I don't receive a breakpoint callback from lldb):

https://gist.github.com/juananpe/6fb8ef7f1a8931d972df8ae24aaf2ae0

Again, the same Ada program can be correctly debugged from CLI using my custom lldb:

I know that Ada is not supported, thus I wanted to replicate the same weird behavior using a trivial C program.

I built my own version of lldb using this recipe: https://gist.github.com/juananpe/3dbf52bc5dcd1e28e7d14c461530bb6d

The bin and lib folders of my custom lldb are here: https://www.dropbox.com/scl/fi/qairf2wbnvfx9mcz8aqec/custom-lldb.tgz?rlkey=8pc6rfszxkx8g5tifwyf0jhc8&dl=0

Any help or hint about how to proceed or what else to test will be much appreciated. I was wondering if you could share the recipe used for building the built-in rust-enabled lldb version 16.0.0 Thanks.

[vadimcn]

Hard to say... I would try stopping at program entry and checking breakpoint state with break list to see if they got resolved.

Any differences between CodeLLDB and what people do at in CLI lldb, usually come down to them setting breakpoints by file name only, whereas CodeLLDB uses a full path: https://github.com/vadimcn/codelldb/wiki/Breakpoints-are-not-getting-hit. Try setting "breakpointMode":"file".

I was wondering if you could share the recipe used for building the built-in rust-enabled lldb version 16.0.0

The Rust-related changes to lldb can be seen here: https://github.com/vadimcn/llvm-project/tree/codelldb/15.x

As for the build pipeline, it's rather complicated and under-documented, and I'd rather not have to provide support for that. It would be required only for building lldb with broad platform compatibility, which likely isn't what you are after.

[juananpe]

@vadimcn Thanks for your response.

I tried to display the list of breakpoints following your advice. Not sure where to stop the program, though. I selected Run / Start Debugging and then tried the break list command from the command input, obtaining the following:

In text format:

break list
Current breakpoints:
1: file = '/opt/adatutorial/small.adb', line = 12, exact_match = 0, locations = 1, resolved = 1, hit count = 1
  1.1: where = small`_ada_small + 66 at small.adb:12:24, address = 0x0000000100001172, resolved, hit count = 1 

So, the breakpoint seems to get resolved (and hit)

I also included the "breakpointMode": "file" setting in launch.json. But removing the "breakPointMode" line from the file AFAICT I got the same result ?:-|

I also get the message

error: Command requires a process, which is currently stopped.

whenever I try to use the v command:

It's weird because the debugger is running but the process has stopped...

What I don't really understand is why I should change any configuration when the debugger works perfectly with a program that already has the breakpoints set correctly. If all I do is swap the lldb binary (and lib/ folder) and restart VSCode, everything should still work the same, right? That's why I thought that my next step should be to rebuild LLDB using the exact recipe that you used for building the built-in rust-enabled lldb version 16.0.0-custom. If it works, I will apply my Ada patch to your lldb fork directly. If not, well, I'll have run out of ideas.

Thanks again for your time,

[vadimcn]

Not sure where to stop the program, though.

Try setting "stopOnEntry": true. This should stop the program before the first instruction gets executed.
Alternatively, you could cause your program to send SIGSTOP to itself, or just intentionally cause a segfault; these will stop the process and allow you it inspect its state.

If all I do is swap the lldb binary (and lib/ folder) and restart VSCode, everything should still work the same, right?

Not necessarily. You are adding your changes to top-of-the-tree LLDB. There could have been changes between 16.0.0 and TOT, that broke CodeLLDB. I guess you could try applying your changes on top of 16.x branch.

That's why I thought that my next step should be to rebuild LLDB using the exact recipe that you used for building the built-in rust-enabled lldb version 16.0.0-custom.

It is very unlikely that my build recipe impacts breakpoints functionality in any way.

[juananpe]

I guess you could try applying your changes on top of 16.x branch.

Good news! Building against that branch did the trick!

I have two additional questions, and I'd greatly appreciate your assistance in understanding them. In your bundled LLDB, there are bin/ and lib/ folders containing just 3 files and 2 files with 2 directories, respectively.

$ ls -al lib/
-rwxr-xr-x@   1 juanan  staff  87194856 Aug 21 09:55 liblldb.dylib
-rwxr-xr-x@   1 juanan  staff  10445726 Aug 21 09:55 libpython39.dylib
drwxr-xr-x@   3 juanan  staff        96 Aug 21 09:55 lldb-python
drwxr-xr-x@ 199 juanan  staff      6368 Aug 21 09:55 python3.9

However, after building my custom LLDB, I found 377 files in lib/. Surprisingly, there's no libpython* or lldb-python present.

$ ls -al lib/ | wc -l
     377

An excerpt:

-rwxrwxrwx  1 juanan  staff     105704 Aug 22 16:08 liblldbPluginSymbolFileBreakpad.a
-rwxrwxrwx  1 juanan  staff    1078288 Aug 22 16:08 liblldbPluginSymbolFileDWARF.a
-rwxrwxrwx  1 juanan  staff     797696 Aug 22 16:08 liblldbPluginSymbolFileNativePDB.a
-rwxrwxrwx  1 juanan  staff     225512 Aug 22 16:08 liblldbPluginSymbolFilePDB.a
-rwxrwxrwx  1 juanan  staff      36456 Aug 22 16:08 liblldbPluginSymbolFileSymtab.a
-rwxrwxrwx  1 juanan  staff       9720 Aug 22 16:08 liblldbPluginSymbolVendorELF.a
-rwxrwxrwx  1 juanan  staff      20632 Aug 22 16:08 liblldbPluginSymbolVendorMacOSX.a
-rwxrwxrwx  1 juanan  staff       9808 Aug 22 16:08 liblldbPluginSymbolVendorPECOFF.a
-rwxrwxrwx  1 juanan  staff      10208 Aug 22 16:08 liblldbPluginSymbolVendorWasm.a
-rwxrwxrwx  1 juanan  staff     201384 Aug 22 16:08 liblldbPluginSystemRuntimeMacOSX.a
-rwxrwxrwx  1 juanan  staff      32264 Aug 22 16:08 liblldbPluginTraceExporterCTF.a
-rwxrwxrwx  1 juanan  staff      65080 Aug 22 16:08 liblldbPluginTraceExporterCommon.a
-rwxrwxrwx  1 juanan  staff     351496 Aug 22 19:41 liblldbPluginTypeSystemClang.a
-rwxrwxrwx  1 juanan  staff     114416 Aug 22 16:08 liblldbPluginTypeSystemRust.a
-rwxrwxrwx  1 juanan  staff      46680 Aug 22 16:08 liblldbPluginUnwindAssemblyInstEmulation.a
-rwxrwxrwx  1 juanan  staff      66624 Aug 22 16:08 liblldbPluginUnwindAssemblyX86.a
-rwxrwxrwx  1 juanan  staff    1005600 Aug 22 16:08 liblldbSymbol.a
-rwxrwxrwx  1 juanan  staff    2571664 Aug 22 16:08 liblldbTarget.a
-rwxrwxrwx  1 juanan  staff     756400 Aug 22 16:08 liblldbUtility.a
-rwxrwxrwx  1 juanan  staff       3064 Aug 22 16:08 liblldbVersion.a
drwxrwxrwx@ 1 juanan  staff    1048576 Aug 22 16:09 python3.11

For every libXXXX.a file, there's also a corresponding ._libXXXX.a file. In fact, I had to remove all those ._lib files to make CodeLLDB function properly.

• Did you have the same number of libs and then removed the unneded? Or any hint about how can I reduce the number of lib* files to the minimum so CodeLLDB still works?
• Do you know how can I get rid of those peculiar ._libXXXX.a files?

Thank you once again for your assistance and the time you've dedicated to answering my questions.

[vadimcn]

These are static libs produced by the LLDB build. CodeLLDB uses dynamic linking, so I bundle only the dylib. Generally, a build directory of any project will contain lots of intermediate build artifacts, which aren't needed at runtime. What you put in the installation package depends on use case. If you do a default build of LLDB, it will depend on whatever version of Python you have installed on your machine. Which is fine for local use, but since I want a "portable" package, I bundle a minimal Python runtime as well.