Thread Stack doesn't resolve symbols for .NET Core or .NET 5

[KyleKolander]

I have asked the question on stackoverflow, but haven't gotten an answer. Rather than duplicate everything here, I've provided a link to the post on stackoverflow.com: https://stackoverflow.com/questions/68632054/how-to-get-net-5-symbols-to-show-up-in-the-call-stack-in-process-hacker-or-proc.

[KyleKolander]

Is anyone able to get symbols to resolve correctly in the Thread Stack of a .NET 5 console application? I'm trying to determine if it is a limitation of Process Hacker, or something configured wrong on my machine. Anyone?

[jxy-s]

I'll have to look into if we have any existing facility in Process Hacker to do symbol resolution of managed code for .NET 5.0. It's possible some existing manage code symbol resolution is broken with .NET 5.0. WinDbg Preview has the same problem when dumping the native stack:

   0  Id: 4630.4660 Suspend: 1 Teb: 000000e8`f23b4000 Unfrozen
 # Child-SP          RetAddr               Call Site
00 000000e8`f21ae4f8 00007ffd`57f595be     ntdll!NtDelayExecution+0x14
01 000000e8`f21ae500 00007ffc`c8e1be5e     KERNELBASE!SleepEx+0x9e
02 (Inline Function) --------`--------     coreclr!ClrSleepEx+0xd [D:\workspace\_work\1\s\src\coreclr\src\vm\hosting.cpp @ 259] 
03 000000e8`f21ae5a0 00007ffc`c8e1bd54     coreclr!Thread::UserSleep+0xb2 [D:\workspace\_work\1\s\src\coreclr\src\vm\threads.cpp @ 4187] 
04 000000e8`f21ae5f0 00007ffc`c894c1da     coreclr!ThreadNative::Sleep+0xa4 [D:\workspace\_work\1\s\src\coreclr\src\vm\comsynchronizable.cpp @ 643] 
05 000000e8`f21ae740 00007ffc`69335f03     System_Private_CoreLib!System.Threading.Thread.Sleep(Int32)$##60025CC+0xa
06 000000e8`f21ae770 00007ffc`c8e89413     0x00007ffc`69335f03
07 000000e8`f21ae7b0 00007ffc`c8d6d0ca     coreclr!CallDescrWorkerInternal+0x83 [D:\workspace\_work\1\s\src\coreclr\src\vm\amd64\CallDescrWorkerAMD64.asm @ 100] 
08 000000e8`f21ae7f0 00007ffc`c8df07af     coreclr!MethodDescCallSite::CallTargetWorker+0x3d2 [D:\workspace\_work\1\s\src\coreclr\src\vm\callhelpers.cpp @ 552] 
09 (Inline Function) --------`--------     coreclr!MethodDescCallSite::Call+0xb [D:\workspace\_work\1\s\src\coreclr\src\vm\callhelpers.h @ 458] 
0a 000000e8`f21ae980 00007ffc`c8df057a     coreclr!RunMainInternal+0x11f [D:\workspace\_work\1\s\src\coreclr\src\vm\assembly.cpp @ 1465] 
0b 000000e8`f21aeab0 00007ffc`c8df02d9     coreclr!RunMain+0xd2 [D:\workspace\_work\1\s\src\coreclr\src\vm\assembly.cpp @ 1536] 
0c 000000e8`f21aeb60 00007ffc`c8df0098     coreclr!Assembly::ExecuteMainMethod+0x1cd [D:\workspace\_work\1\s\src\coreclr\src\vm\assembly.cpp @ 1648] 
0d 000000e8`f21aeef0 00007ffc`c8db2932     coreclr!CorHost2::ExecuteAssembly+0x1c8 [D:\workspace\_work\1\s\src\coreclr\src\vm\corhost.cpp @ 384] 
0e 000000e8`f21af060 00007ffc`edad2b26     coreclr!coreclr_execute_assembly+0xe2 [D:\workspace\_work\1\s\src\coreclr\src\dlls\mscoree\unixinterface.cpp @ 431] 
0f (Inline Function) --------`--------     hostpolicy!coreclr_t::execute_assembly+0x2b [D:\workspace\_work\1\s\src\installer\corehost\cli\hostpolicy\coreclr.cpp @ 89] 
10 000000e8`f21af100 00007ffc`edad2d67     hostpolicy!run_app_for_context+0x3be [D:\workspace\_work\1\s\src\installer\corehost\cli\hostpolicy\hostpolicy.cpp @ 246] 
11 000000e8`f21af290 00007ffc`edad39eb     hostpolicy!run_app+0x37 [D:\workspace\_work\1\s\src\installer\corehost\cli\hostpolicy\hostpolicy.cpp @ 275] 
12 000000e8`f21af2d0 00007ffd`05dd399e     hostpolicy!corehost_main+0xfb [D:\workspace\_work\1\s\src\installer\corehost\cli\hostpolicy\hostpolicy.cpp @ 408] 
13 000000e8`f21af490 00007ffd`05dd7210     hostfxr!execute_app+0x1de [D:\workspace\_work\1\s\src\installer\corehost\cli\fxr\fx_muxer.cpp @ 146] 
14 (Inline Function) --------`--------     hostfxr!?A0xd9c3d171::read_config_and_execute+0x10a [D:\workspace\_work\1\s\src\installer\corehost\cli\fxr\fx_muxer.cpp @ 520] 
15 000000e8`f21af580 00007ffd`05dd5a7b     hostfxr!fx_muxer_t::handle_exec_host_command+0x214 [D:\workspace\_work\1\s\src\installer\corehost\cli\fxr\fx_muxer.cpp @ 1001] 
16 000000e8`f21af670 00007ffd`05dd2029     hostfxr!fx_muxer_t::execute+0x39b [D:\workspace\_work\1\s\src\installer\corehost\cli\fxr\fx_muxer.cpp @ 566] 
*** WARNING: Unable to verify checksum for apphost.exe
17 000000e8`f21af7b0 00007ff7`4b50e0f0     hostfxr!hostfxr_main_startupinfo+0x89 [D:\workspace\_work\1\s\src\installer\corehost\cli\fxr\hostfxr.cpp @ 50] 
18 000000e8`f21af8b0 00007ff7`4b50e458     apphost!exe_start+0x620 [D:\workspace\_work\1\s\src\installer\corehost\corehost.cpp @ 236] 
19 000000e8`f21afa90 00007ff7`4b50f9c8     apphost!wmain+0x124 [D:\workspace\_work\1\s\src\installer\corehost\corehost.cpp @ 302] 
1a (Inline Function) --------`--------     apphost!invoke_main+0x22 [D:\agent\_work\10\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 90] 
1b 000000e8`f21afc00 00007ffd`59c77034     apphost!__scrt_common_main_seh+0x10c [D:\agent\_work\10\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl @ 288] 
1c 000000e8`f21afc40 00007ffd`5a262651     KERNEL32!BaseThreadInitThunk+0x14
1d 000000e8`f21afc70 00000000`00000000     ntdll!RtlUserThreadStart+0x21

I'm only familiar with the native symbol resolution in Process Hacker, as of now, so I'll have to investigate why you get an expected result with <.NET 5.0.

[jxy-s]

At this time I'm unsure if this is an existing bug or just an unsupported path which needs enhancement, so I'm labeling it as needs investigation for now.

[KyleKolander]

Thank you @jxy-s! I just recently discovered Process Hacker, but I absolutely love it! It's so much faster and more responsive than Process Explorer. FWIW, I tested with .NET Core 3.1 and get the same result -- doesn't resolve the symbols, either. If I had to wager a guess, I'd say it's likely something to do with .NET Core and .NET 5 using apphost.exe, whereas .NET Framework console applications actually contained the IL code.

[jxy-s]

Happy to hear it! Thanks for the info. I'll update the thread when we have more info on this.

If you find any more info or have a solution, please update us here.

[KyleKolander]

Maybe take a look at a few of these methods for determining whether an executable is .NET Core/CoreBootstrapExe/Standard/Framework: PE.cs.

Specifically the IsDotNetCoreBootstrapExe method checks for apphost.pdb, which is what you find when you extract the pdbPath from the .NET Core 3.1 / .NET 5 exe files.

public bool IsDotNetCore
{
    get
    {
        if (this.IsManaged && this.managedPlatform == ManagedPlatform.Unknown)
        {
            this.managedPlatform = ComputeIsDotNetCore(this.metadataReader);
        }
        return this.managedPlatform == ManagedPlatform.DotNetCore;
    }
}

public bool IsDotNetCoreBootstrapExe =>
        // The .NET core bootstrap exe is a generated native binary that loads
        // its corresponding .NET core application entry point dll.
        !this.IsDotNetCore
            && this.CodeViewDebugDirectoryData.Path != null
            && (this.CodeViewDebugDirectoryData.Path.EndsWith("apphost.pdb", StringComparison.OrdinalIgnoreCase)
                || this.CodeViewDebugDirectoryData.Path.EndsWith("singlefilehost.pdb", StringComparison.OrdinalIgnoreCase));

public bool IsDotNetNativeBootstrapExe
{
    get
    {
        if (this.IsDotNetCore) { return false; }
        if (this.Imports == null || this.Imports.Length != 1) { return false; }

        string correspondingDllName = Path.GetFileNameWithoutExtension(this.Uri.OriginalString) + ".dll";

        return this.Imports[0].Equals(correspondingDllName, StringComparison.OrdinalIgnoreCase);
    }
}

public bool IsDotNetStandard
{
    get
    {
        if (this.IsManaged && this.managedPlatform == ManagedPlatform.Unknown)
        {
            this.managedPlatform = ComputeIsDotNetCore(this.metadataReader);
        }
        return this.managedPlatform == ManagedPlatform.DotNetStandard;
    }
}

public bool IsDotNetFramework
{
    get
    {
        if (this.IsManaged && this.managedPlatform == ManagedPlatform.Unknown)
        {
            this.managedPlatform = ComputeIsDotNetCore(this.metadataReader);
        }
        return this.managedPlatform == ManagedPlatform.DotNetFramework;
    }
}

internal static ManagedPlatform ComputeIsDotNetCore(MetadataReader metadataReader)
{
    if (metadataReader.AssemblyReferences.Count == 0)
    {
        return ManagedPlatform.DotNetFramework;
    }

    foreach (AssemblyReferenceHandle handle in metadataReader.AssemblyReferences)
    {
        AssemblyReference assemblyReference = metadataReader.GetAssemblyReference(handle);
        StringHandle stringHandle = assemblyReference.Name;
        string assemblyName = metadataReader.GetString(stringHandle);

        switch (assemblyName)
        {
            case "aacorlib":
            case "mscorlib":
            {
                return ManagedPlatform.DotNetFramework;
            }

            case "System.Private.CoreLib":
            case "System.Runtime":
            {
                return ManagedPlatform.DotNetCore;
            }

            case "netstandard":
            {
                return ManagedPlatform.DotNetStandard;
            }
        }
    }

    throw new InvalidOperationException("Could not identify managed platform.");
}

[KyleKolander]

Also, could you take a look at this Microsoft dotnet tool: dotnet-stack?

You can see from the output below that this tool is able to resolve symbols for .NET Core stack traces. This seems pretty conclusive that it's a bug with Process Hacker not being able to "understand" .NET Core / 5 thread stacks. Do you agree?

dotnet-stack report --process-id 37692

Thread (0x3124):
  [Native Frames]
  System.Console.il!System.ConsolePal.ReadKey(bool)
  System.Console.il!System.Console.ReadKey()
  SymbolsCoreConsoleApp!SymbolsCoreConsoleApp.UserInteraction.Interact(int64)
  SymbolsCoreConsoleApp!SymbolsCoreConsoleApp.Program.Run()
  SymbolsCoreConsoleApp!SymbolsCoreConsoleApp.Program.Main()

[jxy-s]

Thanks for the resources, I'll dig deeper into these when I have time.

This seems pretty conclusive that it's a bug with Process Hacker

A bug classification would mean we have preexisting code to do this which isn't working - which I don't believe we do. So it's more of an enhancement request in my eyes.

[KyleKolander]

OK, thanks for looking into it. Hopefully you can get it resolved and it's not too complicated given the information I provided.

I get where you're coming from regarding bug vs. enhancement. The counter argument for it being a bug would be that Process Hacker has preexisting code to resolve symbols in Thread Stacks (including .NET), but it doesn't work for some of .NET (i.e., Core). Regardless, it's a great tool, but it would be unfortunate if our team couldn't use it because we develop .NET Core applications. It's tough to get a team to adopt a new tool if it's missing a critical piece of functionality.

Thanks again! Please let me know if there's anything I can do to help.

[dmex]

Fixed in latest nightly build 👍

I've also added inline stack support. You'll now also see the inline frames when symbols are available:

[KyleKolander]

@dmex - Very Cool!!! I just tested it myself and it works as expected! Thank you!