Debugger engine crashes when creating a TTD memory write timeline

[akaranta]

I am trying to debug an access violation caused by a double free (at least Address Sanitizer reports it as such). I have a time travel debugging trace sized ~450G (created without Address Sanitizer being involved if that makes a difference).

Stepping backwards and forwards around the error works fine. However, after finding out the offending memory address and trying to create a Write access memory timeline on it, the debugging session ends and I get a dialog telling me "The engine has been disconnected unexpectedly.".

Here are the logs:

2022-08-19:01:53:44:770 : Information : DbgX.dll : (E) Executing request: ModelQueryRequest: @$cursession.TTD.Memory(0x000001f44ce086d0, 0x000001f44ce086d8, "w").Select( x => new { MemoryAccessInfo = x, SourceInformationInfo = @$dis(x.Address) } ),1000 2022-08-19:01:54:06:179 : Error : DbgX.dll : (E) Request failed: ModelQueryRequest: @$cursession.TTD.Memory(0x000001f44ce086d0, 0x000001f44ce086d8, "w").Select( x => new { MemoryAccessInfo = x, SourceInformationInfo = @$dis(x.Address) } ),1000 2022-08-19:01:54:06:179 : Information : DbgX.dll : (E) Completed request (21408 ms) : ModelQueryRequest: @$cursession.TTD.Memory(0x000001f44ce086d0, 0x000001f44ce086d8, "w").Select( x => new { MemoryAccessInfo = x, SourceInformationInfo = @$dis(x.Address) } ),1000 2022-08-19:01:54:06:187 : Information : DbgX.Services.dll : Action item published: Error - The engine has been disconnected unexpectedly. 2022-08-19:01:54:06:225 : Information : DbgX.Debugger.dll : Starting EngHost: C:\Program Files\WindowsApps\Microsoft.WinDbg_1.2202.7001.0_neutral8wekyb3d8bbwe\amd64\EngHost.exe npipe:pipe=DbgX_4bcc0264a2874b2b99e3d9c3b16f9c18,password=59734bb534bd "C:\Program Files\WindowsApps\Microsoft.WinDbg_1.2202.7001.0_neutral__8wekyb3d8bbwe\amd64" "C:\ProgramData\Dbg" 2022-08-19:01:54:06:231 : Information : DbgX.dll : (E) Added process EngHost (6500) to job object afbe9fd0-c126-40db-b7b9-4e581e9ccf68 2022-08-19:01:54:06:335 : Information : DbgX.dll : (E) DebugConnect returned S_OK 2022-08-19:01:54:06:337 : Warning : DbgX.dll : (E) Failed to retrieve last session settings : System.InvalidCastException: Unable to cast COM object of type 'System.ComObject' to interface type 'WindowsDebugger.DbgEng.IDebugSettings'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{9D339BE5-30CD-4403-92C3-57EA33799CB1}' failed due to the following error: The call was not transmitted properly; the message queue was full and was not emptied after yielding. (Exception from HRESULT: 0x8001000A (RPC_E_CANTTRANSMIT_CALL)). at System.StubHelpers.StubHelpers.GetCOMIPFromRCW(Object objSrc, IntPtr pCPCMD, IntPtr& ppTarget, Boolean& pfNeedsRelease) at WindowsDebugger.DbgEng.IDebugSettings.StoreSettingsInStream(IDebugOutputStream output) at DbgX.EngineThread.SyncSettingsFromEngine() 2022-08-19:01:54:06:337 : Information : DbgX.dll : (E) Killing process EngHost (7996) for job object 16f27d16-a434-4ba6-9c7c-5e6687dddf41 (HasExited: True) 2022-08-19:01:54:06:337 : Information : DbgX.dll : (E) Killing process EngHost 7996 failed because : Access is denied 2022-08-19:01:54:06:337 : Information : DbgX.dll : (E) Changed client, isPrimaryClient=True 2022-08-19:01:54:06:372 : Information : DbgX.Services.dll : Saving target configuration 2022-08-19:01:54:06:376 : Information : DbgX.Services.dll : Target configuration filename: C:\Users\nakk\AppData\Local\dbg\Targets\OpenTrace0177670446.debugTarget 2022-08-19:01:54:06:384 : Information : DbgX.Services.dll : Target configuration added to recent list 2022-08-19:01:54:06:337 : Information : DbgX.dll : (E) Loading engine settings 2022-08-19:01:54:06:337 : Information : DbgX.dll : (E) Executing secondary thread request: ChangeClientRequest 2022-08-19:01:54:06:338 : Information : DbgX.dll : (E) ChangeSymbolState(DEBUG_CSS.PATHS) 2022-08-19:01:54:06:338 : Information : DbgX.dll : (E) ChangeSymbolState(DEBUG_CSS.PATHS) 2022-08-19:01:54:06:338 : Information : DbgX.dll : (E) Symbol path after loading engine settings: 2022-08-19:01:54:06:339 : Information : DbgX.dll : (E) Symbol path = '', source path = '' (Checking for client refresh) 2022-08-19:01:54:06:343 : Information : DbgX.dll : (E) Completed secondary thread request: ChangeClientRequest

The system has a ridiculous amount of RAM - 96G, of which less than 10G are at use, so this seems unlikely to be out-of-memory issue.

In addition to the C/C++ code I am debugging the application also has Fortran and managed code. I am only trying to debug the C/C++ part(s), but wanted to mention this in case it has some known effect on TTD / WinDBG Preview.

Any ideas on what I could do differently to work around the issue?

WinDBG Preview Debugger client version: 1.2202.7001.0 Debugger engine version: 10.0.22549.1000

[KenSykes]

[Sorry that we let this issue sit, this page hasn't been monitored for awhile]

First .. 450GB is very impressive! I think this may be a record size trace.

My guess is your .Memory() call is generating so many results that the in-memory std::vector<> holding the results grows too large and triggers an exception. If so, the workaround would be to use "ba w8 " and g / g- as appropriate.

[akaranta]

Thanks for the answer! I don't have the said trace anymore and got the issue resolved by other means (although had TTD worked for me it would have been much easier). But I'll keep this in mind for the future.

If this page is not monitored, what is the correct place to send feedback / bug reports & such?

[KenSykes]

This may end up being the right place, we are discussing support channels internally. In the meantime the Feedback Hub in WinDbg Preview's ribbon is the best bet.

[lzybkr]

We are actually monitoring this page a little bit more than in the past.

The signal to noise ratio is unfortunately very low in Feedback Hub so we are considering other options, so I'd say opening issues here is fine.