Open lambdageek opened 5 months ago
/cc @hoyosjs
This looks like a runtime issue not SOS's. CorUnix::CPalThread::EnableMachExceptions()'s call to thread_swap_exception_ports in the runtime's PAL. I'm not sure what SOS can do about it.
yea, it's a runtime issue. Just wanted to bring it up here in case anyone else runs into it
I brought it up because there is nothing we can do in the diagnostics repo to fix it that maybe it should be a runtime repo issue.
I can reproduce as well.
If I export PAL_MachExceptionMode=7
first, lldb doesn't crash anymore, but libsos doesn't work:
(lldb) plugin load /Users/rolf/.dotnet/sos/libsosplugin.dylib
(lldb) setsymbolserver -ms
Error: Fail to initialize coreclr 80070008
Maybe move this issue to the runtime repo if that's where it should be?
@rolfbjarne I was going to open a new issue for dotnet/runtime with just a barebones LLDB plugin that hosts CoreCLR so that it's easier to reproduce. it's still on my todo list - I just need to clean up my repro.
Not sure if it's related, but the JVM folks discovered that pthread_write_protect_np(0)
sometimes causes macOS Sonoma 14.4 to kill a process when it's trying to access unrelated mmaped memory with a SIGKILL. Previous versions of macOS issued a SIGBUS.
https://bugs.java.com/bugdatabase/view_bug?bug_id=8327860 (see in particular the repro from 12-03-2024, reproduced below:
#include <stdio.h>
#include <sys/mman.h>
#include <pthread.h>
int main() {
pthread_jit_write_protect_np(0);
char* mem = (char*)mmap(0, 16 * 1024, 0, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0);
fprintf(stderr, "addr = %p\n", mem);
char value = *mem;
fprintf(stderr, "value = %c\n", value);
return 0;
}
macOS 14.3: SIGBUS macOS 14.4: SIGKILL
I opened dotnet/runtime#99977 to track potential runtime work. I think it's worthwhile to keep this diagnostics issue open since this is probably where folks will look for SOS issues.
A workaround is to use a non-Apple lldb
, i.e. one from homebrew
as it won't be as locked down. This doesn't help for debugging from Xcode, but at least can unblock dotnet-sos
for those on 14.4. (brew install llvm
; /opt/homebrew/Cellar/llvm/*/bin/lldb
)
Have you successfully loaded a x64 libsosplugin.dylib under the brew installed lldb? We tried recently and plugin load libsosplugin.dylib
failed with error: this file does not represent a loadable dylib
.
/cc: @mikelle-rogers
A workaround is to use a non-Apple
lldb
, i.e. one fromhomebrew
as it won't be as locked down. This doesn't help for debugging from Xcode, but at least can unblockdotnet-sos
for those on 14.4. (brew install llvm
;/opt/homebrew/Cellar/llvm/*/bin/lldb
)Have you successfully loaded a x64 libsosplugin.dylib under the brew installed lldb? We tried recently and
plugin load libsosplugin.dylib
failed witherror: this file does not represent a loadable dylib
./cc: @mikelle-rogers
The issue with using lldb from homebrew is not an arm64/x64 mismatch (at least on my arm64 mac - everything is an arm64 binary).
The issue is that libsosplugin.dylib
is dynamically linked against @rpath/LLDB.framework/Versions/A/LLDB
while the homebrew lldb doesn't provide a framework.
I had to do the following workaround:
# backup SOS plugin in case I mess up
$ cp ~/.dotnet/sos/libsosplugin.dylib ~/.dotnet/sos/libsosplugin.dylib.orig
# change the sos plugin to link against liblldb from homebrew
$ install_name_tool -change "@rpath/LLDB.framework/Versions/A/LLDB" "@rpath/../lib/liblldb.dylib" ~/.dotnet/sos/libsosplugin.dylib#4694
# ad-hoc codesign libsosplugin.dylib since I just invalidated the original signature
$ codesign -s - -f ~/.dotnet/sos/libsosplugin.dylib
After this I can run dotnet under lldb and examine the debugger
$ /opt/homebrew/Cellar/llvm/18.1.6/bin/lldb /tmp/HelloWorld/bin/Debug/net9.0/HelloWorld
(lldb) b coreclr_execute_assembly
(lldb) r
(lldb) clrthreads
ThreadCount: 3
UnstartedThread: 0
BackgroundThread: 3
PendingThread: 0
DeadThread: 0
Hosted Runtime: no
...
(lldb) verifyheap
42 objects verified, 0 errors.
No heap corruption detected.
Ah yep, that would work. Another option if homebrew is not desired is to keep using the Apple lldb, but strip its permissions:
mkdir ~/tmp # or whatever
cp /Applications/Xcode.app/Contents/Developer/usr/bin/lldb .
install_name_tool -add_rpath /Applications/Xcode.app/Contents/SharedFrameworks lldb
cat <<EOF > ent.xml
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>com.apple.security.get-task-allow</key>
<true/>
<key>com.apple.security.cs.debugger</key>
<true/>
<key>com.apple.security.cs.disable-library-validation</key>
<false/>
</dict>
</plist>
codesign --force --entitlements ~/tmp/ent.xml --sign - lldb
and then use ~/tmp/lldb
Description
After installing
dotnet-sos
and adding then installing the plugin usingdotnet-sos install
, I get a crash as soon as the plugin tries to initialize the coreclr host.Note this is different from https://github.com/dotnet/diagnostics/issues/4259 where the coreclr host fails to initialize but at least it doesn't kill the lldb session.
The new behavior is that lldb completely dies. Looking in
Console.app
, I see the following stack trace:Note
CorUnix::CPalThread::EnableMachExceptions()
crashing in a call tothread_swap_exception_ports
due to anEXC_GUARD
GUARD_TYPE_MACH_PORT
mach exception. Looking around it seems like (see this StackOverflow answer) this is because the exception port for the LLDB process is guarded now. Possibly this is new in macOS 14.4.