LLDB: Unexpected scripting behaviour

[fritzlb]

While executing debug commands in a file works fine from interactive lldb mode (command source -s 0 "myfile.txt"), trying to start lldb with "lldb -s "myfile.txt" doesn't work correctly. It seems like lldb executes the first command and then somehow stops.

The used file contains the following debug commands:

process connect --plugin gdb-remote connect://[fde0:5d53:fb4d::1]:52213
settings set target.memory-module-load-level minimal
attach -p 1881
detach

(the script seems useless but actually by connecting a debugger to an iOS app for a short period of time the app gets privileges that are needed for JIT compiling)

This happens on both 17.06 as well as on 18.1.0 RC2.

output with lldb -s "myfile.txt":

(lldb) command source -s 0 C:\Users\User\Desktop\myfile.txt
Executing commands in 'C:\Users\User\Desktop\myfile.txt'.
(lldb) process connect --plugin gdb-remote connect://[fde0:5d53:fb4d::1]:52213

(then hangs there forever)

trying to solve the issue by using lldb -o "command source -s 0 C:\Users\User\Desktop\batch-file.txt" doesn't work either.

also, by adding "quit" to the bottom of myfile.txt lldb doesn't quit but instead crashes (sometimes) when typing another command with the following errors:

PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: C:\\Users\\Frederik\\Downloads\\llvm-mingw-20240207-msvcrt-x86_64\\llvm-mingw-20240207-msvcrt-x86_64\\bin\\lldb.exe
1.      HandleCommand(command = "command source -s 0 C:/Users/Frederik/Desktop/batch-file.txt")
2.      HandleCommand(command = "attach -p 1881")
3.      HandleCommand(command = "process attach -p 1881")
LLDB diagnostics will be written to C:\Users\Frederik\AppData\Local\Temp\diagnostics-8625d3
Please include the directory content when filing a bug report
Exception Code: 0xC0000005
 #0 0x00007ffd866166d6 PyInit__lldb (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\liblldb.dll+0x3166d6)
 #1 0x00007ffd867c1c3e PyInit__lldb (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\liblldb.dll+0x4c1c3e)
 #2 0x00007ffd8665a81c PyInit__lldb (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\liblldb.dll+0x35a81c)
 #3 0x00007ffd8692ef82 PyInit__lldb (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\liblldb.dll+0x62ef82)
 #4 0x00007ffd865c501a PyInit__lldb (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\liblldb.dll+0x2c501a)
 #5 0x00007ffd865b9cac PyInit__lldb (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\liblldb.dll+0x2b9cac)
 #6 0x00007ffd8698b17a PyInit__lldb (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\liblldb.dll+0x68b17a)
 #7 0x00007ffd865c53c1 PyInit__lldb (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\liblldb.dll+0x2c53c1)
 #8 0x00007ffd865b9cac PyInit__lldb (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\liblldb.dll+0x2b9cac)
 #9 0x00007ffd865bd96f PyInit__lldb (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\liblldb.dll+0x2bd96f)
#10 0x00007ffd868a35e2 PyInit__lldb (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\liblldb.dll+0x5a35e2)
#11 0x00007ffd865436bc PyInit__lldb (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\liblldb.dll+0x2436bc)
#12 0x00007ffd865bb10a PyInit__lldb (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\liblldb.dll+0x2bb10a)
#13 0x00007ffd868f4303 PyInit__lldb (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\liblldb.dll+0x5f4303)
#14 0x00007ffd865c501a PyInit__lldb (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\liblldb.dll+0x2c501a)
#15 0x00007ffd865b9cac PyInit__lldb (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\liblldb.dll+0x2b9cac)
#16 0x00007ffd865bd96f PyInit__lldb (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\liblldb.dll+0x2bd96f)
#17 0x00007ffd868a35e2 PyInit__lldb (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\liblldb.dll+0x5a35e2)
#18 0x00007ffd8654348e PyInit__lldb (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\liblldb.dll+0x24348e)
#19 0x00007ffd865bf4a0 PyInit__lldb (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\liblldb.dll+0x2bf4a0)
#20 0x00007ffd86346259 lldb::SBDebugger::RunCommandInterpreter(bool, bool) (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\liblldb.dll+0x46259)
#21 0x00007ff635b23f3b (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\lldb.exe+0x3f3b)
#22 0x00007ff635b24a41 (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\lldb.exe+0x4a41)
#23 0x00007ff635b21311 (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\lldb.exe+0x1311)
#24 0x00007ff635b21366 (C:\Users\Frederik\Downloads\llvm-mingw-20240207-msvcrt-x86_64\llvm-mingw-20240207-msvcrt-x86_64\bin\lldb.exe+0x1366)
#25 0x00007ffdd910257d (C:\Windows\System32\KERNEL32.DLL+0x1257d)
#26 0x00007ffddad0aa58 (C:\Windows\SYSTEM32\ntdll.dll+0x5aa58)

System: latest version of Win11

[mstorsjo]

Is this issue specific to connecting to remote targets somehow? Because I do use script files successfully in some cases - see e.g. https://github.com/mstorsjo/llvm-mingw/blob/master/run-lldb-tests.sh. This does run successfully as part of each build. However I don't do quit at the end of the script file there, I invoke lldb with the option -b, to make it quit when done.

Can you provide instructions for a way to reproduce this issue that doesn't involve connecting to exotic remote targets; is it possible to test by starting up a separate process with lldb-server locally, then connecting to it with LLDB?

[fritzlb]

yes, it seems like that. Running lldb-server g localhost:9090 and then attaching using interactive lldb (even by loading commands from a file) works, using a file directly by launching lldb with -s doesn't.

Terminal 1: .\lldb-server.exe g localhost:9090

Terminal 2: .\lldb.exe -b -s "C:\Users\User\bugfile.txt"

doesn't work while starting lldb in interactive mode and then typing command source -s 0 C:\Users\User\bugfile.txt does.

bugfile.txt contains the following: process connect --plugin gdb-remote connect://localhost:9090 attach error

ofc error isn't a valid process name, but it throws an error so that I can see if the line is reached at all.

[mstorsjo]

yes, it seems like that. Running lldb-server g localhost:9090 and then attaching using interactive lldb (even by loading commands from a file) works, using a file directly by launching lldb with -s doesn't.

Terminal 1: .\lldb-server.exe g localhost:9090

Terminal 2: .\lldb.exe -b -s "C:\Users\User\bugfile.txt"

doesn't work while starting lldb in interactive mode and then typing command source -s 0 C:\Users\User\bugfile.txt does.

bugfile.txt contains the following: process connect --plugin gdb-remote connect://localhost:9090 attach error

ofc error isn't a valid process name, but it throws an error so that I can see if the line is reached at all.

Thanks for the reproducer steps. Unfortunately, I didn't manage to reproduce this. I tried with the latest 18.1.0 RC2 builds, and whenever I try to execute the process connect ... command, I get this error:

(lldb) process connect --plugin gdb-remote connect://localhost:9090 attach error
error: 'process connect' takes exactly one argument:
Usage: process connect <remote-url>

[fritzlb]

Sorry, my bad. I did not format the text correctly.attach error has to be in a new line of the file

[mstorsjo]

Thanks, now I've managed to reproduce it, somewhat, I think. When doing this, it looks like the lldb-server process hangs, consuming 100% cpu.

[mstorsjo]

@alvinhochun As you've poked LLDB a bit, do you have time to look into this? Otherwise I can try to look at it at some later point when I have time.

FWIW, this sounds like an issue which is a clear upstream bug, so if you wanted to, you could file it as a bug at https://github.com/llvm/llvm-project/issues, especially if it is reproducible with the LLDB builds from their official builds. (LLDB is a bit tricky to use there, though, since it requires you to have the same version of Python installed as was used for building it.)

[mstorsjo]

Thanks, now I've managed to reproduce it, somewhat, I think. When doing this, it looks like the lldb-server process hangs, consuming 100% cpu.

Actually, lldb-server seems to be spinning, consuming 100% cpu even in the working case, once the lldb client has connected. That seems clearly suboptimal, but unrelated.

It does look like LLDB deadlocks here, or something like that. It's reproducible with upstream MSVC builds as well, so this issue should ideally be filed upstream in https://github.com/llvm/llvm-project/issues. I can try to file a bug there in a moment.

[mstorsjo]

I reported this issue upstream now, at https://github.com/llvm/llvm-project/issues/106282.

[fritzlb]

Thank you very much!

[alvinhochun]

Sorry, I don't think I can help. At a first glance it does look like https://github.com/llvm/llvm-project/issues/79430 linked by David in the upstream report is related, so perhaps not specific to Windows.

The crash might be a separate thing though.

[mstorsjo]

Sorry, I don't think I can help. At a first glance it does look like llvm/llvm-project#79430 linked by David in the upstream report is related, so perhaps not specific to Windows.

Indeed, that seems to be exactly it. That bug report also links to https://stackoverflow.com/a/77869259/1617325 which contains a good explanation of the issue, and a workaround.