Closed WheretIB closed 4 years ago
My ComponentLevel is 1999010.
For additional info, if I move FindDocuments handler to remote component with ComponentLevel of 40500, throwing an exception doesn't break the debugger. But PythonTools project has this handler at level 1999000, so I don't understand what is expected here.
Seems like SymbolProviderId Filter in the vsdconfigxml file is the way to go.
I've added a filter, but only received a single request to FindDocuments for my module against a .cpp file.
No matter if I throw an exception or pretend to return a DkmResolvedDocument, my other component callbacks are not getting called (IDkmSymbolQuery.GetSourcePosition) and since VS debugger hasn't requested any info, raw addresses and 'Frame not in module' are displayed:
Even though the 0889D35A address is within my custom module (0889B938-088C1D87)
Yup, you would definitely want a SymbolProviderId
filter so you are only taking over finding symbols in your custom module. You can called DkmModule.Create and DkmModuleInstance.SetModule(DkmModule) (or just created a module instance with the DkmModule already set) to associate your symbol provider with this custom module? What does your .vsdconfigxml file look?
Oh, and I should add - FindDocuments/FindSymbols is going to be used for binding breakpoints (and similar), NOT for call stack frame formatting. Are you getting called for breakpoint binding?
If you want to do frame formatting, you want to implement IDkmLanguageFrameDecoder.
I now add a SymbolProviderId that matches guidNullcSymbolProviderFilterId
in my vsdconfigxml file:
processData.moduleId = new DkmModuleId(Guid.NewGuid(), NullcDebuggerHelpers.NullcSymbolProviderGuid);
processData.compilerId = new DkmCompilerId(NullcDebuggerHelpers.NullcCompilerGuid, NullcDebuggerHelpers.NullcLanguageGuid);
processData.module = DkmModule.Create(processData.moduleId, "nullc.embedded.code", processData.compilerId, process.Connection, null);
processData.moduleInstance = DkmCustomModuleInstance.Create("nullc", "nullc.embedded.code", 0, process.GetNativeRuntimeInstance(), null, /*symbolFileId*/null, DkmModuleFlags.None, DkmModuleMemoryLayout.Unknown, moduleBase, 1, moduleSize, "nullc embedded code", false, null, null, null);
processData.moduleInstance.SetModule(processData.module, false);
What does your .vsdconfigxml file look?
<?xml version="1.0" encoding="utf-8" ?> <Configuration xmlns="http://schemas.microsoft.com/vstudio/vsdconfig/2008">
<DefineGuid Name="guidNullcLocalSymbolProviderId" Value="E8E22514-AFF8-4A82-BA16-32BC4E91C8E5"/>
<DefineGuid Name="guidNullcSymbolProviderFilterId" Value="BF13BE48-BE1A-4424-B961-BFC40C71E58A"/>
<ManagedComponent ComponentId="guidNullcLocalSymbolProviderId" ComponentLevel="1999010" AssemblyName="nullc_debugger_component">
<Class Name="nullc_debugger_component.DkmDebugger.NullcLocalSymbolProvider" WorkerProcessSupported="true">
<Implements>
<InterfaceGroup>
<Filter>
<SymbolProviderId RequiredValue="guidNullcSymbolProviderFilterId"/>
</Filter>
<Interface Name="IDkmSymbolCompilerIdQuery"/>
<Interface Name="IDkmSymbolDocumentCollectionQuery"/>
<Interface Name="IDkmSymbolDocumentSpanQuery"/>
<Interface Name="IDkmSymbolQuery"/>
</InterfaceGroup>
</Implements>
</Class>
</ManagedComponent>
> Are you getting called for breakpoint binding?
Yes, I get requests for breakpoint locations through `IDkmSymbolDocumentCollectionQuery.FindDocuments` (first argument is my custom module and the second one is the path to a .cpp where I have a breakpoint set), and VS no longer breaks when I return `return new DkmResolvedDocument[0];` or throw an exception with the new filter.
> If you want to do frame formatting, you want to implement IDkmLanguageFrameDecoder.
Thank you, I will look into that.
In general, I want to provide all the data that .pdb files provide to the debugger (functions, source locations, types, argument/variable locations on stack/in registers) but from a custom just-in-time compiled language (x86/x64) with a custom debug data format.
That's why I was looking into 'symbol provider' requests.
You should definitely look at the symbol provider interfaces, but keep in mind that is necessary, but not sufficient -- you will need some sort of at least basic expression evaluator, and based on your description it sounds like you will need a runtime DM for handling source-level stepping.
I now fail to correctly setup a filter for IDkmLanguageFrameDecoder.
I've tried filtering by RuntimeId:
<InterfaceGroup>
<Filter>
<RuntimeId RequiredValue="3AF14FEA-CB31-4DBB-90E5-74BF685CA7B8"/>
</Filter>
<Interface Name="IDkmLanguageFrameDecoder"/>
</InterfaceGroup>
by creating a custom runtime instance:
processData.runtimeId = new DkmRuntimeInstanceId(new Guid("3AF14FEA-CB31-4DBB-90E5-74BF685CA7B8"), 0);
processData.runtimeInstance = DkmCustomRuntimeInstance.Create(process, processData.runtimeId, null);
and passing it to my custom module:
processData.moduleInstance = DkmCustomModuleInstance.Create("nullc", "nullc.embedded.code", 0, processData.runtimeInstance, null, /*symbolFileId*/null, DkmModuleFlags.None, DkmModuleMemoryLayout.Unknown, moduleBase, 1, moduleSize, "nullc embedded code", false, null, null, null);
processData.moduleInstance.SetModule(processData.module, true);
but my frame decoder doesn't get called.
I also tried filtering by LanguageId since my custom module already specifies a custom language, but that also doesn't work.
Later, just to see what it may look like, I've removed the filter to respond to all requests with completionRoutine(new DkmGetFrameNameAsyncResult("FakeTestResultFunction"));
and found that Visual Studio still displays that frame is not in a module:
It looks like another interface is missing (I always try to check vsdconfig.xsd, but can't yet find an interface that might be called for this information).
I also want to note that the language is displayed as 'Unknown', and I have created my custom module with a language that is registered in .pkgdef file
[$RootKey$\AD7Metrics\ExpressionEvaluator\{9221BA37-3FB0-483A-BD6A-0E5DD22E107E}\{A7CB5F2B-CD45-4CF4-9CB6-61A30968EFB5}]
"Language"="nullc"
"Name"="nullc"
[$RootKey$\AD7Metrics\ExpressionEvaluator\{9221BA37-3FB0-483A-BD6A-0E5DD22E107E}\{A7CB5F2B-CD45-4CF4-9CB6-61A30968EFB5}\Engine]
"0"="{449EC4CC-30D2-4032-9256-EE18EB41B62B}"
"1"="{92EF0900-2251-11D2-B72E-0000F87572EF}"
"2"="{3B476D35-A401-11D2-AAD4-00C04F990171}"
Seems like it would be fine to skip the component filter method and use method chaining for requests where I can't provide information.
Still trying to figure out the problem with my custom module. I've added a response to 'IDkmModuleUserCodeDeterminer.IsUserCode' to mark module as user code:
But when IDkmLanguageFrameDecoder.GetFrameName
method is called, it could be seen that the instruction address wasn't recognized as part of my module, so NonuserCode flag is set and the ModuleInstance is null:
I was able to provide required stack frame information using IDkmCallStackFilter.FilterNextFrame
. When I provide info there, IDkmLanguageFrameDecoder.GetFrameName
isn't even called and I get correct stack frame language and double click navigates to file/line returned by IDkmSymbolQuery.GetSourcePosition
.
Sorry for getting out of scope of the original issue with these comments, hope I have enough information now to proceed with other components (expression evaluator/runtime stepper) even if some of the questions above don't have a clear answer.
@WheretIB A few notes in case these are helpful -
[Frames below may be incorrect and/or missing]
entry, but x64 has no issues with the help of RtlAddFunctionTable.After simple expression evaluator I was checking breakpoints and disassembly view and Visual Studio behavior is different depending on custom/native module and runtime.
For example in my original implementation with DkmCustomModuleInstance and DkmCustomRuntimeInstance disassembly view cannot be opened (Disassembly cannot be displayed for the source location. There is no executable code at this location in the source code.
) and breakpoint can't be set (Error while processing breakpoint.
)
OnException: filter exception.
System.ArgumentException: Value does not fall within the expected range.
at XapiExceptionProcessing.ThrowHR(Int32 code)
at Microsoft.VisualStudio.Debugger.Symbols.DkmInstructionSymbol.Bind(DkmModuleInstance ModuleInstance)
at VSDebugEngine.BreakpointManager.BMBreakpointNodeGroup.CreateBoundBreakpoints(BMAsyncBindContext asyncContext, BMPendingBreakpoint pendingBreakpoint, BMInstructionContainer instructionContainer, IList`1 symbols, IList`1 symbolLocations)
at VSDebugEngine.BreakpointManager.BMBreakpointNodeGroup.CreateBoundBreakpoints(BMAsyncBindContext asyncContext, BMPendingBreakpointNode node)
I figure it's because the debugger expects me to handle that manually even when JiT code is 'native' since the module and runtime are 'custom'. I think custom types might make more sense for an interpreter or for example a GPU shader debugger.
I have then switched to DkmNativeRuntimeInstance (with the default native runtime as 'parent runtime') and DkmNativeModuleInstance.
Call stack still contains a single [Frames below may be incorrect and/or missing]
entry on x86, so a custom unwinder might still be required.
Disassembly view can now be opened (even though it's an assembly-only view without source code lines) but breakpoints still fail with Error while processing breakpoint.
OnException: filter exception.
System.NotImplementedException: The method or operation is not implemented.
at XapiExceptionProcessing.ThrowHR(Int32 code)
at Microsoft.VisualStudio.Debugger.Breakpoints.DkmRuntimeBreakpoint.Enable(DkmWorkList WorkList, DkmCompletionRoutine`1 CompletionRoutine)
at VSDebugEngine.BreakpointManager.BMBoundBreakpoint.Initialize(BMAsyncBindContext asyncContext, BindingCancellationToken cancellationToken, BMBindHandler bindHandler)
at VSDebugEngine.BreakpointManager.BMBoundBreakpoint.CreateBoundBreakPointHelper(BMAsyncBindContext asyncContext, BMPendingBreakpoint pendingBreakpoint, BindingCancellationToken cancellationToken, DkmInstructionAddress instructionAddress, BMBindHandler bindHandler, DkmSourcePosition sourcePosition, DkmInstructionSymbol dkmInstructionSymbol, DkmInspectionSession inspectionSession)
...
Maybe I have to implement IDkmPendingFileLineBreakpointCallback.GetCurrentSourcePosition
.
I have also tried to remove DkmNativeRuntimeInstance and use the default native runtime. It fixes the [Frames below may be incorrect and/or missing]
entry in x86 call stack, but disassembly view fails with Disassembly cannot be displayed for the source location. An argument was out of its legal range.
Breakpoints don't display an error message but don't seem to work, and I can no longer step out of a C++ function up one frame into a JiT function (worked before even without a custom stepper).
But even with these problems, progress is still being made in other places so I'm sure it will come together after a while.
Implementing IDkmInstructionAddressProvider fixed the disassembly view when DkmNativeRuntimeInstance /DkmNativeModuleInstance are used.
So the reason DkmRuntimeBreakpoint.Enable
throws System.NotImplementedException: The method or operation is not implemented.
is that there is no IDkmRuntimeMonitorBreakpointHandler
component that wishes to handle the breakpoint coming from a native module in a non-default native runtime (everyone gets filtered out by RuntimeId).
Something to keep in mind when such an exception is thrown.
Haven't found out yet while breakpoints don't work when native module is created in a default native runtime.
I've tried providing IDkmRuntimeMonitorBreakpointHandler
myself using InvisibleWriteMemory to write int 3
and restore it back. Breakpoints are working but only once, since someone removes the custom int 3
(maybe some kind of a default response to IDkmDebugMonitorExceptionNotification.OnDebugMonitorException
in NativeDM::CNativeDebugMonitor
).
And sadly, the breakpoint is displayed as an exception in the VS debugger.
I have attempted to catch both cases in IDkmRuntimeBreakpointReceived.OnRuntimeBreakpointReceived
and IDkmEmbeddedBreakpointHitNotification.OnEmbeddedBreakpointHit
but those are not getting called even though my component level is 40500. Logs show that the first is handled by NativeDM::CNativeDebugMonitor
and the second one by ad7::CALEventReceiver
The original idea was to provide enough high-level symbol information to allow the default debugger to break and step through code, but now with these manual breakpoints and custom stepper it fells like I'm re-implementing the debugger myself (and I don't have enough experience, I'm afraid how I'm going to implement 'Step In' if there is an indirect call, will I have to decode x86 assembly and lookup the addresses in registers? Can't I get the instance of CNativeDebugMonitor and ask it directly to make that step?).
To implement breakpoints, I had to provide IDkmRuntimeMonitorBreakpointHandler
interface.
To enable the breakpoint I place an 'int 3' instruction at the target address.
To disable it, I restore the original byte value at the target address.
When the breakpoint is hit, I handle the breakpoint exception in IDkmDebugMonitorExceptionNotification.OnDebugMonitorException
by suppressing the event , storing the break location for future reference and calling Thread.OnEmbeddedBreakpointHit
to break in IDE.
When process is resumed by IDE, in a IDkmProcessExecutionNotification.OnProcessResume
I schedule a single step using DkmSingleStepRequest.EnableSingleStep
.
When the step is completed, in a IDkmSingleStepCompleteReceived.OnSingleStepCompleteReceived
handler I restore the breakpoint instruction.
Original instruction at breakpoint location is automatically restored by some VS component on x86.
The steps above work in x86 debugger, but on x64 even though the single step is performed and IDkmSingleStepCompleteReceived.OnSingleStepCompleteReceived
is called, the instruction pointer doesn't actually change and the breakpoint instruction is hit again (even if the breakpoint instruction is removed in IDkmDebugMonitorExceptionNotification.OnDebugMonitorException
manualy)
To write a custom symbol provider, as part of the implementation, I have registered for IDkmSymbolDocumentCollectionQuery, but VS asks me to find documents for all modules and source files that are not part of my module.
Documentation tells that
I've tried different return values and they all break the VS debugger:
return null;
- Exception in DkmDebugger, debugger stops working.return return new DkmResolvedDocument[0];
- C++ breakpoints stop working (my extension is not even for C++)return new DkmResolvedDocument[1] { null };
- Visual Studio crashes completelyI've looked at PythonTools, they use
throw new NotSupportedException();
That also leads to C++ breakpoints not working with a message "The breakpoint will not currently be hit. Unexpected symbol reader error while processing console.exe"So is there a way to tell the debugger to 'look elsewhere' if my symbol provider doesn't handle specific modules?