Is it possible to skip IDkmSymbolDocumentCollectionQuery.FindDocuments?

[WheretIB]

To write a custom symbol provider, as part of the implementation, I have registered for IDkmSymbolDocumentCollectionQuery, but VS asks me to find documents for all modules and source files that are not part of my module.

Documentation tells that

        /// Returns document objects from search parameters contained in the document query.
        /// If the symbol file does not contain a reference to this document the returned
        /// document object will be NULL (S_FALSE return code in native). The returned
        /// document objects must be explicitly closed by the caller when the caller is done
        /// with the document.

I've tried different return values and they all break the VS debugger:

return null; - Exception in DkmDebugger, debugger stops working. return return new DkmResolvedDocument[0]; - C++ breakpoints stop working (my extension is not even for C++) return new DkmResolvedDocument[1] { null }; - Visual Studio crashes completely

I've looked at PythonTools, they use throw new NotSupportedException(); That also leads to C++ breakpoints not working with a message "The breakpoint will not currently be hit. Unexpected symbol reader error while processing console.exe"

So is there a way to tell the debugger to 'look elsewhere' if my symbol provider doesn't handle specific modules?

[WheretIB]

My ComponentLevel is 1999010.

For additional info, if I move FindDocuments handler to remote component with ComponentLevel of 40500, throwing an exception doesn't break the debugger. But PythonTools project has this handler at level 1999000, so I don't understand what is expected here.

[WheretIB]

Seems like SymbolProviderId Filter in the vsdconfigxml file is the way to go. I've added a filter, but only received a single request to FindDocuments for my module against a .cpp file. No matter if I throw an exception or pretend to return a DkmResolvedDocument, my other component callbacks are not getting called (IDkmSymbolQuery.GetSourcePosition) and since VS debugger hasn't requested any info, raw addresses and 'Frame not in module' are displayed:

Even though the 0889D35A address is within my custom module (0889B938-088C1D87)

[gregg-miskelly]

Yup, you would definitely want a SymbolProviderId filter so you are only taking over finding symbols in your custom module. You can called DkmModule.Create and DkmModuleInstance.SetModule(DkmModule) (or just created a module instance with the DkmModule already set) to associate your symbol provider with this custom module? What does your .vsdconfigxml file look?

[gregg-miskelly]

Oh, and I should add - FindDocuments/FindSymbols is going to be used for binding breakpoints (and similar), NOT for call stack frame formatting. Are you getting called for breakpoint binding?

If you want to do frame formatting, you want to implement IDkmLanguageFrameDecoder.

[WheretIB]

I now add a SymbolProviderId that matches guidNullcSymbolProviderFilterId in my vsdconfigxml file:

processData.moduleId = new DkmModuleId(Guid.NewGuid(), NullcDebuggerHelpers.NullcSymbolProviderGuid);
processData.compilerId = new DkmCompilerId(NullcDebuggerHelpers.NullcCompilerGuid, NullcDebuggerHelpers.NullcLanguageGuid);

processData.module = DkmModule.Create(processData.moduleId, "nullc.embedded.code", processData.compilerId, process.Connection, null);

processData.moduleInstance = DkmCustomModuleInstance.Create("nullc", "nullc.embedded.code", 0, process.GetNativeRuntimeInstance(), null, /*symbolFileId*/null, DkmModuleFlags.None, DkmModuleMemoryLayout.Unknown, moduleBase, 1, moduleSize, "nullc embedded code", false, null, null, null);

processData.moduleInstance.SetModule(processData.module, false);

What does your .vsdconfigxml file look?

<?xml version="1.0" encoding="utf-8" ?>
<Configuration xmlns="http://schemas.microsoft.com/vstudio/vsdconfig/2008">

<DefineGuid Name="guidNullcLocalSymbolProviderId" Value="E8E22514-AFF8-4A82-BA16-32BC4E91C8E5"/>
<DefineGuid Name="guidNullcSymbolProviderFilterId" Value="BF13BE48-BE1A-4424-B961-BFC40C71E58A"/>

<ManagedComponent ComponentId="guidNullcLocalSymbolProviderId" ComponentLevel="1999010" AssemblyName="nullc_debugger_component">
    <Class Name="nullc_debugger_component.DkmDebugger.NullcLocalSymbolProvider" WorkerProcessSupported="true">
        <Implements>
            <InterfaceGroup>
                <Filter>
                    <SymbolProviderId RequiredValue="guidNullcSymbolProviderFilterId"/>
                </Filter>
                <Interface Name="IDkmSymbolCompilerIdQuery"/>
                <Interface Name="IDkmSymbolDocumentCollectionQuery"/>
                <Interface Name="IDkmSymbolDocumentSpanQuery"/>
                <Interface Name="IDkmSymbolQuery"/>
            </InterfaceGroup>
        </Implements>
    </Class>
</ManagedComponent>

> Are you getting called for breakpoint binding?

Yes, I get requests for breakpoint locations through `IDkmSymbolDocumentCollectionQuery.FindDocuments` (first argument is my custom module and the second one is the path to a .cpp where I have a breakpoint set), and VS no longer breaks when I return `return new DkmResolvedDocument[0];` or throw an exception with the new filter.

> If you want to do frame formatting, you want to implement IDkmLanguageFrameDecoder.

Thank you, I will look into that.

In general, I want to provide all the data that .pdb files provide to the debugger (functions, source locations, types, argument/variable locations on stack/in registers) but from a custom just-in-time compiled language (x86/x64) with a custom debug data format.
That's why I was looking into 'symbol provider' requests.

[gregg-miskelly]

You should definitely look at the symbol provider interfaces, but keep in mind that is necessary, but not sufficient -- you will need some sort of at least basic expression evaluator, and based on your description it sounds like you will need a runtime DM for handling source-level stepping.

[WheretIB]

I now fail to correctly setup a filter for IDkmLanguageFrameDecoder.

I've tried filtering by RuntimeId:

<InterfaceGroup>
    <Filter>
        <RuntimeId RequiredValue="3AF14FEA-CB31-4DBB-90E5-74BF685CA7B8"/>
    </Filter>
    <Interface Name="IDkmLanguageFrameDecoder"/>
</InterfaceGroup>

by creating a custom runtime instance:

processData.runtimeId = new DkmRuntimeInstanceId(new Guid("3AF14FEA-CB31-4DBB-90E5-74BF685CA7B8"), 0);

processData.runtimeInstance = DkmCustomRuntimeInstance.Create(process, processData.runtimeId, null);

and passing it to my custom module:

processData.moduleInstance = DkmCustomModuleInstance.Create("nullc", "nullc.embedded.code", 0, processData.runtimeInstance, null, /*symbolFileId*/null, DkmModuleFlags.None, DkmModuleMemoryLayout.Unknown, moduleBase, 1, moduleSize, "nullc embedded code", false, null, null, null);

processData.moduleInstance.SetModule(processData.module, true);

but my frame decoder doesn't get called.

I also tried filtering by LanguageId since my custom module already specifies a custom language, but that also doesn't work.

Later, just to see what it may look like, I've removed the filter to respond to all requests with completionRoutine(new DkmGetFrameNameAsyncResult("FakeTestResultFunction")); and found that Visual Studio still displays that frame is not in a module: It looks like another interface is missing (I always try to check vsdconfig.xsd, but can't yet find an interface that might be called for this information).

I also want to note that the language is displayed as 'Unknown', and I have created my custom module with a language that is registered in .pkgdef file

[$RootKey$\AD7Metrics\ExpressionEvaluator\{9221BA37-3FB0-483A-BD6A-0E5DD22E107E}\{A7CB5F2B-CD45-4CF4-9CB6-61A30968EFB5}]
"Language"="nullc"
"Name"="nullc"

[$RootKey$\AD7Metrics\ExpressionEvaluator\{9221BA37-3FB0-483A-BD6A-0E5DD22E107E}\{A7CB5F2B-CD45-4CF4-9CB6-61A30968EFB5}\Engine]
"0"="{449EC4CC-30D2-4032-9256-EE18EB41B62B}"
"1"="{92EF0900-2251-11D2-B72E-0000F87572EF}"
"2"="{3B476D35-A401-11D2-AAD4-00C04F990171}"

[WheretIB]

Seems like it would be fine to skip the component filter method and use method chaining for requests where I can't provide information.

Still trying to figure out the problem with my custom module. I've added a response to 'IDkmModuleUserCodeDeterminer.IsUserCode' to mark module as user code:

But when IDkmLanguageFrameDecoder.GetFrameName method is called, it could be seen that the instruction address wasn't recognized as part of my module, so NonuserCode flag is set and the ModuleInstance is null:

[WheretIB]

I was able to provide required stack frame information using IDkmCallStackFilter.FilterNextFrame. When I provide info there, IDkmLanguageFrameDecoder.GetFrameName isn't even called and I get correct stack frame language and double click navigates to file/line returned by IDkmSymbolQuery.GetSourcePosition.

Sorry for getting out of scope of the original issue with these comments, hope I have enough information now to proceed with other components (expression evaluator/runtime stepper) even if some of the questions above don't have a clear answer.

[gregg-miskelly]

@WheretIB A few notes in case these are helpful -

1. If you look at the interface definitions in vsdebugeng.h, they are organized by the type of component that implements the interfaces. Which can be helpful in finding what you might want to implement.
2. It sounds like you may have found this already, but you can also turn on method tracing to find what is being called (see the wiki for more info).
3. Using call stack filters to add your custom frames certainly works, at least if the native unwinder is able to unwind through your frames well enough. The other options are:
   • If your native code is in fixed address ranges, you might be able to use DkmNativeModuleInstance.Create, in which case the debugger could automaticially map your addresses to the right module. Though the non-PE and non-ELF code path for native modules is untested, so if you don't have a PE file you could encounter problems with this approach.
   • You could implement a runtime unwinder instead. This would be helpful if the native debugger sometimes gets lost walking through your code.

[WheretIB]

1. Thank you for the info, I've always skipped that section thinking it was just some internal forward declaration section, but now I see how it's structured.
2. Yes, I have logging enabled, don't have to use it often, but it helped a few times. Although in some cases, a single action like switching to disassembly can result in thousands of events and it's hard to know what to look for if there's an issue.
3. I try to follow MS x86 and x64 ABI so the native unwinder will handle them correctly. x86 still has a single [Frames below may be incorrect and/or missing] entry, but x64 has no issues with the help of RtlAddFunctionTable.

After simple expression evaluator I was checking breakpoints and disassembly view and Visual Studio behavior is different depending on custom/native module and runtime.

For example in my original implementation with DkmCustomModuleInstance and DkmCustomRuntimeInstance disassembly view cannot be opened (Disassembly cannot be displayed for the source location. There is no executable code at this location in the source code.) and breakpoint can't be set (Error while processing breakpoint.)

OnException: filter exception.
System.ArgumentException: Value does not fall within the expected range.
   at XapiExceptionProcessing.ThrowHR(Int32 code)
   at Microsoft.VisualStudio.Debugger.Symbols.DkmInstructionSymbol.Bind(DkmModuleInstance ModuleInstance)
   at VSDebugEngine.BreakpointManager.BMBreakpointNodeGroup.CreateBoundBreakpoints(BMAsyncBindContext asyncContext, BMPendingBreakpoint pendingBreakpoint, BMInstructionContainer instructionContainer, IList`1 symbols, IList`1 symbolLocations)
   at VSDebugEngine.BreakpointManager.BMBreakpointNodeGroup.CreateBoundBreakpoints(BMAsyncBindContext asyncContext, BMPendingBreakpointNode node)

I figure it's because the debugger expects me to handle that manually even when JiT code is 'native' since the module and runtime are 'custom'. I think custom types might make more sense for an interpreter or for example a GPU shader debugger.

I have then switched to DkmNativeRuntimeInstance (with the default native runtime as 'parent runtime') and DkmNativeModuleInstance. Call stack still contains a single [Frames below may be incorrect and/or missing] entry on x86, so a custom unwinder might still be required. Disassembly view can now be opened (even though it's an assembly-only view without source code lines) but breakpoints still fail with Error while processing breakpoint.

OnException: filter exception.
System.NotImplementedException: The method or operation is not implemented.
   at XapiExceptionProcessing.ThrowHR(Int32 code)
   at Microsoft.VisualStudio.Debugger.Breakpoints.DkmRuntimeBreakpoint.Enable(DkmWorkList WorkList, DkmCompletionRoutine`1 CompletionRoutine)
   at VSDebugEngine.BreakpointManager.BMBoundBreakpoint.Initialize(BMAsyncBindContext asyncContext, BindingCancellationToken cancellationToken, BMBindHandler bindHandler)
   at VSDebugEngine.BreakpointManager.BMBoundBreakpoint.CreateBoundBreakPointHelper(BMAsyncBindContext asyncContext, BMPendingBreakpoint pendingBreakpoint, BindingCancellationToken cancellationToken, DkmInstructionAddress instructionAddress, BMBindHandler bindHandler, DkmSourcePosition sourcePosition, DkmInstructionSymbol dkmInstructionSymbol, DkmInspectionSession inspectionSession)
...

Maybe I have to implement IDkmPendingFileLineBreakpointCallback.GetCurrentSourcePosition.

I have also tried to remove DkmNativeRuntimeInstance and use the default native runtime. It fixes the [Frames below may be incorrect and/or missing] entry in x86 call stack, but disassembly view fails with Disassembly cannot be displayed for the source location. An argument was out of its legal range. Breakpoints don't display an error message but don't seem to work, and I can no longer step out of a C++ function up one frame into a JiT function (worked before even without a custom stepper).

But even with these problems, progress is still being made in other places so I'm sure it will come together after a while.

[WheretIB]

Implementing IDkmInstructionAddressProvider fixed the disassembly view when DkmNativeRuntimeInstance /DkmNativeModuleInstance are used.

[WheretIB]

So the reason DkmRuntimeBreakpoint.Enable throws System.NotImplementedException: The method or operation is not implemented. is that there is no IDkmRuntimeMonitorBreakpointHandler component that wishes to handle the breakpoint coming from a native module in a non-default native runtime (everyone gets filtered out by RuntimeId). Something to keep in mind when such an exception is thrown. Haven't found out yet while breakpoints don't work when native module is created in a default native runtime.

I've tried providing IDkmRuntimeMonitorBreakpointHandler myself using InvisibleWriteMemory to write int 3 and restore it back. Breakpoints are working but only once, since someone removes the custom int 3 (maybe some kind of a default response to IDkmDebugMonitorExceptionNotification.OnDebugMonitorException in NativeDM::CNativeDebugMonitor). And sadly, the breakpoint is displayed as an exception in the VS debugger. I have attempted to catch both cases in IDkmRuntimeBreakpointReceived.OnRuntimeBreakpointReceived and IDkmEmbeddedBreakpointHitNotification.OnEmbeddedBreakpointHit but those are not getting called even though my component level is 40500. Logs show that the first is handled by NativeDM::CNativeDebugMonitor and the second one by ad7::CALEventReceiver

The original idea was to provide enough high-level symbol information to allow the default debugger to break and step through code, but now with these manual breakpoints and custom stepper it fells like I'm re-implementing the debugger myself (and I don't have enough experience, I'm afraid how I'm going to implement 'Step In' if there is an indirect call, will I have to decode x86 assembly and lookup the addresses in registers? Can't I get the instance of CNativeDebugMonitor and ask it directly to make that step?).

[WheretIB]

To implement breakpoints, I had to provide IDkmRuntimeMonitorBreakpointHandler interface. To enable the breakpoint I place an 'int 3' instruction at the target address. To disable it, I restore the original byte value at the target address.

When the breakpoint is hit, I handle the breakpoint exception in IDkmDebugMonitorExceptionNotification.OnDebugMonitorException by suppressing the event , storing the break location for future reference and calling Thread.OnEmbeddedBreakpointHit to break in IDE.

When process is resumed by IDE, in a IDkmProcessExecutionNotification.OnProcessResume I schedule a single step using DkmSingleStepRequest.EnableSingleStep.

When the step is completed, in a IDkmSingleStepCompleteReceived.OnSingleStepCompleteReceived handler I restore the breakpoint instruction.

Original instruction at breakpoint location is automatically restored by some VS component on x86.

The steps above work in x86 debugger, but on x64 even though the single step is performed and IDkmSingleStepCompleteReceived.OnSingleStepCompleteReceived is called, the instruction pointer doesn't actually change and the breakpoint instruction is hit again (even if the breakpoint instruction is removed in IDkmDebugMonitorExceptionNotification.OnDebugMonitorException manualy)