Mixed C#/Swig/C++ debugging in vscode

[m17kea]

Initially filed on https://github.com/microsoft/vscode-cpptools/issues/5292 and referred here by @pieandcakes.

• OS and Version: OSX Remote SSH Connection to Amazon Linux 2 (RHEL7 equivalent)
• VS Code Version: 1.44.0
• Dotnet Runtime: 2.2.8
• C/C++ Extension Version: 0.27.0
• Swig 4.0.1
• GCC Version: 9.2
• GDB Version: 8.3
• Other extensions you installed (and if the issue persists after disabling them): C#, CMake, CMake Tools, Test Explorer UI, .Net Core Test Explorer

I am trying to implement mixed-mode debugging in vscode.

To Reproduce I am running a test with the following command: VSTEST_HOST_DEBUG=1 && dotnet vstest testdll.dll --Tests:classname.testname --Parallel --logger:trx;LogFileName=output.trx

This spins up three dotnet processes of which i attach to the third with:

{
    name: '.NET Core Attach',
    type: 'coreclr',
    request: 'attach',
    processId: "${command:pickProcess}", 
    env: {
             "LD_LIBRARY_PATH": "/media/psf/Home/Documents/dev.nosync/Derivitec/symbols-2.2.8;/home/linuxbrew/.linuxbrew/lib/gcc/9/;/media/psf/Home/Documents/dev.nosync/Derivitec/src/build/Debug/Derivitec/;/media/psf/Home/Documents/dev.nosync/Derivitec/src/build/Debug/Derivitec.Wrap/;$LD_LIBRARY_PATH",
            "LD_DEBUG":"all",
            "LD_DEBUG_OUTPUT":"/tmp/testrunner.log"
    }
}

I successfully reach a break point in C# before the C++ I'm interested in.

I attach again with:

{
     "name": "(gdb) Attach",
    "type": "cppdbg",
    "request": "attach",
    "program": "/usr/share/dotnet/dotnet",
    "processId": "${command:pickProcess}",
    "additionalSOLibSearchPath": "${workspaceFolder}/symbols-2.2.8;/home/linuxbrew/.linuxbrew/lib/gcc/9/;${workspaceFolder}/src/build/Debug/Derivitec/;${workspaceFolder}/src/build/Debug/Derivitec.Wrap/",
    "MIMode": "gdb",
    "miDebuggerPath": "/home/linuxbrew/.linuxbrew/bin/gdb",
    "setupCommands": [
         {
             "description": "Enable pretty-printing for gdb",
              "text": "-enable-pretty-printing",
              "ignoreFailures": true,
         }
    ],
    "logging": { 
         "trace": true, 
         "traceResponse": true,
         "engineLogging": true   
    }
}

The gdb debugger attaches successfully and my cpp breakpoints become active in the Breakpoints window. I then press play on the C# debugger. In the call stack the gdb debugger the pauses on an exception as follows:

After continuing the debugger a couple of times it crashes the dotnet process running the test.

I tried adding the libcoreclr source using dotnet-symbol --recurse-subdirectories /usr/share/dotnet/shared/Microsoft.NETCore.App/2.2.8/libcoreclr.so -o symbols-2.2.8 and referencing it in both additionalSOLibSearchPath properties but it does not seem to load it.

GDB logs attached:

gdb.log

Any insights would be greatly appreciated.

Many thanks in advance!

[danmoseley]

You mention you're trying to implement a debugging feature. What specific assistance are you looking for? Do you believe there is a bug in the CoreCLR debugging infrastructure? It's difficult based on this info to guess at where the issue might be.

[m17kea]

Hi @danmosemsft, I initially posted this on the vscode-cpptools repo and they directed me here. Based on the log attached it would seem that the CoreCLR is generating SIGTRAP signals from libcoreclr.so when in debug mode that ultimately crash the application and then the debug session. Whether this is a bug or intended behaviour is beyond my current knowledge of the code base I'm afraid.

I managed to cure the crash by adding the following to the setup commands in the attach config for the debugger:

{
    "ignoreFailures": true,
    "text": "handle SIGTRAP pass noprint nostop"
}

This does now hit my cpp breakpoint but seems to cover up a symptom rather than be a cure? I can also hit the cpp breakpoint if I detach the .Net Core debugger completely. Does this shed anymore light?

I'm happy to commit all of my time to providing you the information you would need, if you can point me in the right direction.

[m17kea]

The stack trace in the call stack image is probably of most use if your trying to tie the problem to the code base.

[danmoseley]

@armitagemderivitec thanks that helps. I'll move this to the place where libcoreclr.so is owned.

[danmoseley]

@tommcdon (BTW @tommcdon , I am happy to get you or someone else set up to get tagged on area-diagnostics issues - just LMK)

[tommcdon]

@tommcdon (BTW @tommcdon , I am happy to get you or someone else set up to get tagged on area-diagnostics issues - just LMK)

@danmosemsft Thanks. I'll let you know if we need to add someone else.

[danmoseley]

OK, I've added you.

[m17kea]

Hi all,

My work around:

{
    "ignoreFailures": true,
    "text": "handle SIGTRAP pass noprint nostop"
}

No longer seems to help. Could you shed any more light on the possible fix for this or any other work around I can implement. We have now switched development to VS Code on Linux and this means we can no longer debug the cpp code from c#.

I've tried various things now and even tried other Linux distributions such as Ubuntu and I'm now out of ideas.

Can I contribute in some way? We're willing to fork and fix for internal use until something is more generally available but obviously waiting until November for .net 5 is going to impede our business.

We'd rather not have to revert back to Windows since our application is running happily on Linux.

Many thanks

Mike

[danmoseley]

Ouch, I hope we can get you working again. cc @tommcdon

[m17kea]

Thanks @danmosemsft I managed to debug into the cpp by detaching the .Net Core debugger. I cannot however reattach to go back into the c#. Any suggestions very welcome. Happy to help in any way I can.

[hoyosjs]

@armitagemderivitec sorry for not responding. I thought I'd sent my answer here and looks like it's now lost.

On windows, mixed mode debugging is a heavy collaboration between the runtime and the native debugger. This was never implemented that way on Linux as the signals and exceptions behave pretty differently from Windows debug events. On Windows, the managed debugger and native debugger collaborate heavily as you can't attach two debuggers; the attach and triage is done first by the managed debugger and then the native debugger handles the events that the managed runtime has no idea of (like the native breakpoints you've set). There's also some nuances here about the native threads of the runtime.

On Linux, the C# debugger doesn't act like a native debugger, hence you can attach a native debugger after it all started. Indeed the SIGTRAP is something you see is an expected behavior of stopping/stepping under a debugger. The native debugger will intercept these at higher priority and you get what you saw; if you pass them then the c# runtime will see them and handle the step, but then you'll not have a very stable (there's plenty more signals and you'll have to be able to handle all the exception/signal semantics we usually own).

I don't know if @mikem8361 or @noahfalk might have any ideas on how to help you improve your inner loop experience on Linux for interop scenarios. @sdmaclea also suggested maybe you can build the runtime with the gdbjit flag, and you'd get a passable c# source mapping experience in GDB (I've never tried breakpoints or variable inspection here) at the expense of pretty expensive in-memory tables that need to be built.

[m17kea]

@hoyosjs thanks for your thorough response. I'll look into the gdbjit flag and see if I can get a workable example.

Are there any plans to enhance the Linux native debug experience? Is this a rare workflow these days?

[hoyosjs]

Are there any plans to enhance the Linux native debug experience? Is this a rare workflow these days?

It's been a long standing conversation to enable interop debugging in Linux, but we are currently pursuing other projects in the production diagnostics space.

On Windows having interop debugging was critical as there was a lot of COM usage for projects migrating to core, and the original architecture made this pretty straightforward. The main scenario For Linux - which is pretty relevant and the heavy argument to enable this - is PInvokes. Perhaps it'd be good to get the conversation rolling over. @janvorli, do you have any ideas to help do interop debugging easier for these scenarios?

fyi @tommcdon

[sdmaclea]

/cc @AaronRobinsonMSFT @jkoritzinsky for interop debugging experience.

[m17kea]

I'm replicating the above setup on Mac OS. I have everything building and running but I cannot attach a lldb or gdb debugger to dotnet because it is runtime hardened:

   0    kernel: (AppleMobileFileIntegrity) macOSTaskPolicy: (com.apple.debugserver) may not get the taskport of (dotnet) (pid: 4859): (dotnet) is hardened, (dotnet) doesn't have get-task-allow, (com.apple.debugserver) is a declared debugger

Current dotnet entitlements:

Executable=/usr/local/share/dotnet/dotnet
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>com.apple.security.cs.allow-jit</key>
      <true/>
    <key>com.apple.security.cs.allow-unsigned-executable-memory</key>
      <true/>
    <key>com.apple.security.cs.allow-dyld-environment-variables</key>
      <true/>
    <key>com.apple.security.cs.disable-library-validation</key>
      <true/>
  </dict>
</plist>

How do I safely bypass the security without disabling SIP entirely. Is there any documentation on this anywhere? Is it just a case of re-signing dotnet with:

<key>com.apple.security.get-task-allow</key>  
    <true/>  

Any advice very welcome!

I'd also like to keep this discussion moving. We are financial risk company serving many brokers and hedge funds and have a significant mixed c#/c++ code base. We've successfully migrated all the code and infrastructure to run our application on the Linux Kernel and do not wish to revert just to enable mixed mode debugging. We'd be happy to collaborate on a linux/MacOS mixed mode debugging project.

[m17kea]

Re-signing with the additional entitlement com.apple.security.get-task-allow seemed to do the trick. I can successfully debug a test in c#, attach the lldb debugger, activate the cpp breakpoints and then detach the .Net Core debugger. However although the program states that it's running the cpp debug breakpoint is never hit and the application appears stalled. Logs attached:

lldb.log

Is anyone able to assist with this?

I'm forced to use clang/lldb over gcc/gdb on MacOS currently because of an outstanding Vcpkg issue:

https://github.com/microsoft/vcpkg/issues/9344

[sdmaclea]

the cpp debug breakpoint is never hit

I think that is a side affect to the handle SIGTRAP pass noprint nostop or its equivalent in lldb

You could add a breakpoint in C# before the failing SWIG call. Run the code, wait for it to break in C#. While code is at C# break. Stop the code in lldb w/ ^C? (optional?) Set the breakpoint in lldb Disable the handle SIGTRAP pass noprint nostop in lldb. Remove/Disable all C# breakpoints Hit go in the C# debugger. Hit go in lldb

We'd be happy to collaborate on a linux/MacOS mixed mode debugging project.

It is something on our backlog. It is definitely too late for 5.0 release, but might be something we could consider fully supporting for the 6.0 release (2021).

Is anyone able to assist with this?

You could also try debugging a coredump. See https://github.com/dotnet/diagnostics/blob/master/documentation/debugging-coredump.md

Linux experience is slightly better than Mac as we don't yet have Mach-O support in all our tools.

If your code gets lost, you could generate a core dump and look at the call stacks in lldb, lldb w/ SOS debugger, or dotnet-dump analyze.

[m17kea]

Just so I can set expectations internally, is debugging from c# to cpp potentially going to be available in .net 6? Our preference would be for Linux but Mac OS would be really useful also. We are happy to be guinea pigs in this effort and help wherever we can.

[sdmaclea]

Native mixed debugging is a major feature development effort. While I would love to see full support here, I won't get time to consider working on this in the .NET 6 time frame. We do not get a lot of requests for this, so it may take a while for it to be prioritized.

We have been working on cross platform debugging support in VS. I am not sure if we support the full set of features you need.

/cc @clguiman is there a good public summary of the VS cross platform debugging support currently/planned released.

[clguiman]

Currently Visual Studio supports the following two scenarios:

1. Attaching to a Linux debugger that's debugging a native Linux core dump:https://devblogs.microsoft.com/cppblog/debug-linux-core-dumps-in-visual-studio/
2. Debugging (on Windows) a managed core dump: https://devblogs.microsoft.com/visualstudio/linux-managed-memory-dump-debugging/

Mixed debugging is not a high priority right now and I don't have a timeframe on when that's going to be supported, but the priority will definitely increase if there's a feature request with lots of votes on the VS Developer Community website.

[emmenlau]

This would be quite helpful to have!

[m17kea]

Hi all, just wondering if this has made it into the backlog to be worked upon yet?

[rodrigo455]

vsdbg handles .pdb files for debug... and it should support C/C++ debug no? if you convert codeview/dwarf debug information to a .pdb file and somehow load that file into vsdbg, maybe it would be able to step into your C++ code? any easy way to do that in linux?

[johndaintree]

Mixed mode debugging on non-Windows platforms is becoming increasingly important to my projects. Is there any sign of happening?

[m17kea]

We're still incredibly keen for this as well

[Jinjinov]

I would also like to have mixed mode debugging on non-Windows platforms

[m17kea]

Hi, is there any update on this?

Also, is it available on Arm64 now Microsoft has Windows 11 and VS running on Arm?

[m17kea]

Any movement here?