search

dotnet / diagnostics

This repository contains the source code for various .NET Core runtime diagnostic tools and documents.
MIT License
1.18k stars 355 forks source link

WinDbg SOS Just Won't Give Up On Symbol Files #675

Open Zhentar opened 4 years ago

Zhentar commented 4 years ago

SOS refuses to take "no" for an answer when looking up symbol files, repeatedly trying to acquire them. !U for a single method without a pdb available can take minutes.

Example output:

Windows symbol path parsing FAILED
00007ffd`d1d5064e ffe0            jmp     rax
Windows symbol path parsing FAILED
00007ffd`d1d50650 90              nop
Windows symbol path parsing FAILED
00007ffd`d1d50651 eb0f            jmp     00007ffd`d1d50662
Windows symbol path parsing FAILED
00007ffd`d1d50653 90              nop
Windows symbol path parsing FAILED
00007ffd`d1d50654 eb17            jmp     00007ffd`d1d5066d
Windows symbol path parsing FAILED
00007ffd`d1d50656 90              nop
Windows symbol path parsing FAILED
00007ffd`d1d50657 e997050000      jmp     00007ffd`d1d50bf3
Windows symbol path parsing FAILED
00007ffd`d1d5065c 90              nop
Windows symbol path parsing FAILED
00007ffd`d1d5065d e9f7070000      jmp     00007ffd`d1d50e59
Windows symbol path parsing FAILED
00007ffd`d1d50662 33c0            xor     eax,eax
Windows symbol path parsing FAILED
00007ffd`d1d50664 8945ec          mov     dword ptr [rbp-14h],eax

In this case, I was lucky - because my sympath couldn't be parsed, it was only pinging a the Microsoft symbol server and most lines took a mere 200ms each to print! image

mikem8361 commented 4 years ago

I not sure it is the attempted sympath parsing is taking 200ms. It is probably SOS source/line number lookup searching for the portable PDB and then the windows PDB for the method being disassembled. Can you send me the .sympath you were using? And the output of !setsymbolserver? Thanks.

I am looking into only parsing the .sympath once even on failures. It is just changing where this flag gets set.

Zhentar commented 4 years ago

The sympath parsing was only taking 1-2ms; it's a problem for the output but it's not the performance problem here. I included the screenshot of the Wininet graph to show that in between each line, it's doing three round trips to http://msdl.microsoft.com/download/symbols/ at 60-70ms each. If you have more than one external symbol server in your sympath, or a slower internet connection, it gets much worse.

mikem8361 commented 4 years ago

I'm not sure exactly what we should do here. Most of our tools don't cache "misses" which I assuming is happening in your case.

Zhentar commented 4 years ago

This is literally the only tool I've ever used that re-tries a failed symbol lookup without manual input. Most of them don't even automatically retry after setting a new symbol path.

The frequency at which SOS does it here is crippling; even with my 'best case scenario' of single digit ping time corporate internet using only the Microsoft symbol server, !U on a large method with 1500 lines of assembly would take 5 minutes! On my crappy home internet with one third party server added onto my path and I'd be looking at 30 minutes to disassemble one function, and the operation isn't even cancel-able!