dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.27k stars 4.73k forks source link

Issues with memory dumps and Single File App #107344

Open ppekrol opened 2 months ago

ppekrol commented 2 months ago

Description

When dumps are being created on Windows 11 and Windows Server 2022 they cannot be always debugged correctly. Most of the time. This only happens when Single File App is being used. Here are my findings:

OS Type Behavior
W11 Mini Works
W11 Triage Does not work
W11 Heap Works
W11 Full Works
WS2022 Mini Does not work
WS2022 Triage Does not work
WS2022 Heap Does not work
WS2022 Full Works

Reproduction Steps

  1. Create console app (.NET 8.0.8):
    
    using System.Diagnostics;

using (var process = Process.GetCurrentProcess()) { Console.WriteLine($"PID: {process.Id}"); }

var mre = new ManualResetEvent(false);

var t = new Thread(() => { Thread.CurrentThread.Name = "Background Work";

mre.WaitOne();

});

t.Start();

Console.WriteLine("Press any key to close..."); Console.ReadLine();

mre.Set(); t.Join();

Console.WriteLine("Done");



2. Publish `dotnet publish -c Release /p:PublishSingleFile=true` and run
3. `dotnet-dump collect -p PID --type Triage`
4. Open VS and click `Debug with Managed Only`
![image](https://github.com/user-attachments/assets/06950074-163b-4f71-9c86-d60f35ec99a2)
5. In that area we also found: https://developercommunity.visualstudio.com/t/Parallel-Stacks-do-not-display-threads-f/10738338?port=1025&fsid=efce44a5-7c7a-474d-867d-c812b3de1d99

### Expected behavior

Should work?

### Actual behavior

Does not work

### Regression?

_No response_

### Known Workarounds

_No response_

### Configuration

_No response_

### Other information

_No response_
ppekrol commented 2 months ago

Forgot to add:

dotnet-dump --version
8.0.532401+d98406ff95be0a8398d4c403056c5705389665f3
ppekrol commented 2 months ago

Checked also Linux:

OS Type Behavior
Ubuntu 24.04 Mini Does not work
Ubuntu 24.04 Triage Does not work
Ubuntu 24.04 Heap Does not work
Ubuntu 24.04 Full Does not work

Not sure if the dumps are wrong, the file opens in VS without errors, but 'Debug Managed Code' throws an issue, or there is missing support for something in the VS?

ppekrol commented 2 months ago

In parallel we are testing same thing with dotnet publish -c Release /p:PublishSingleFile=false -r linux-x64 --self-contained and it works fine.

tommcdon commented 2 months ago

Hi @ppekrol, unfortunately only full dumps are supported for single file applications - please see https://learn.microsoft.com/en-us/dotnet/core/diagnostics/collect-dumps-crash#types-of-mini-dumps. It should be possible to workaround this limitation by deploying the matching DAC next to the binary:

Hope this helps!

ppekrol commented 2 months ago

Hi @tommcdon

We have tested your suggestion on WS2022 and Ubuntu 24.04 but it does not have any effect.

image

Situation is as follows: OS Type Behavior
WS2022 Mini Does not work
WS2022 Triage Does not work
WS2022 Heap Does not work
WS2022 Full Works
Ubuntu 24.04 Mini Does not work
Ubuntu 24.04 Triage Does not work
Ubuntu 24.04 Heap Does not work
Ubuntu 24.04 Full Does not work

Which worries me for few reasons:

  1. You stated that Full dumps should work, but they do not on Ubuntu 24.04
  2. Taking a full dump in many situations is a killer, especially when you use mmap
  3. All of those dumps can be opened in WinDBG, so the issue is in VS or in the dump file itself?
tommcdon commented 2 months ago

@hoyosjs @mikem8361

mikem8361 commented 1 month ago

It looks like you are copying the Windows DAC (mscordaccore.dll) next to your app for WSL and Ubuntu 24.04. For Linux, you need to copy libmscordaccore.so. This doesn't explain why Ubuntu 24.04 full dumps do not work. Add the --diag option to your dotnet-dump collect command line should produce some log output that may be helpful.

mikem8361 commented 1 month ago

I assumed when you said "WS2022" you meant WSL (Linux) but you mean Windows Server 2022 which should work with mscordaccore.dll next your app.

mikem8361 commented 1 month ago

Triage dumps may have enough managed state for VS to load them. Windbg/SOS should work for all types of dumps.

On Windows Server 2022, it may not have the support needed to use the DAC that is a resource in your app like Windows 11 does. It probably needs these special registry keys set up to allow it to work:

Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\KnownFunctionTableDlls Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\KnownManagedDebuggingDlls Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\MiniDumpAuxiliaryDlls

arekpalinski commented 1 month ago
  • Can you also clarify what you mean in your tables by "Does not work"?

  • Is the dump generated?

  • Can windbg load the dump?

  • Can VS load it?

It means that the dump is generated, the WinDBG can load it but opening it in VS and trying "Debug with Managed Only" action results in:

image

It doesn't mater whether it's Mini, Heap or Triage.

Regarding the registry keys, we have the following values on Windows Server 2022 (we didn't modify anything there):

image

image

image

The application is using .NET 8.0.8

mikem8361 commented 1 month ago

I'm not sure yet, why the WS2022 dumps are not being loaded by VS. The reason Ubuntu 24.04 dumps are not being loaded by VS is probably that the .NET 8 you installed is a "source-build" version that Ubuntu has built/provided. This means that the necessary debugging binaries are not available on the Microsoft symbol servers for download. This is a common problem now that more and more distro's are building .NET themselves. One solution is to install the Microsoft built .NET 8 SDK on your Ubuntu 24.04 machines. See ubuntu-2404. You need to use the Microsoft package repository.

ppekrol commented 1 month ago

Ok, so I installed Ubuntu 22.04 via Hyper-V Quick Create option, updated all packages and installed dotnet-sdk-8.0 and dotnet-dump from Microsoft package repository (even switched the package repository priority in /etc/apt/preferences).

.NET SDK:
 Version:           8.0.402
 Commit:            70aa751718
 Workload version:  8.0.400-manifests.b6724b7a
 MSBuild version:   17.11.4+37eb419ad

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  22.04
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /usr/share/dotnet/sdk/8.0.402/

.NET workloads installed:
Configured to use loose manifests when installing new manifests.
There are no installed workloads to display.

Host:
  Version:      8.0.8
  Architecture: x64
  Commit:       08338fcaa5

.NET SDKs installed:
  8.0.402 [/usr/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 8.0.8 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 8.0.8 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

WITHOUT libmscordaccore.so and createdump next to the app:

  1. dotnet-dump collect -p PID --type Full --diag => File created => in VS 2022 getting Managed debugging is not available for this minidump
Writing full to /home/ppekrol/Downloads/core_20240927_100714
Complete

Output: full.txt

  1. dotnet-dump collect -p PID --type Triage --diag => File NOT created
Writing triage dump to /home/ppekrol/Downloads/core_20240927_101116
Complete

Output: triage.txt

  1. dotnet-dump collect -p PID --type Heap --diag => File NOT created
Writing dump with heap to /home/ppekrol/Downloads/core_20240927_101321
Complete

Output: heap.txt

  1. dotnet-dump collect -p PID --type Mini --diag => File NOT created
Writing dump to /home/ppekrol/Downloads/core_20240927_101655
Complete

Output: mini.txt

WITH libmscordaccore.so and createdump next to the app:

  1. dotnet-dump collect -p PID --type Full --diag => File created => in VS 2022 getting Managed debugging is not available for this minidump
Writing full to /home/ppekrol/Downloads/core_20240927_102637
Complete

Output: full2.txt

  1. dotnet-dump collect -p PID --type Triage --diag => File created => in VS 2022 getting Managed debugging is not available for this minidump
Writing triage dump to /home/ppekrol/Downloads/core_20240927_103140
Complete

Output: triage2.txt

  1. dotnet-dump collect -p PID --type Heap --diag => File created => in VS 2022 getting Managed debugging is not available for this minidump
Writing dump with heap to /home/ppekrol/Downloads/core_20240927_102936
Complete

Output: heap2.txt

  1. dotnet-dump collect -p PID --type Mini --diag => File created => in VS 2022 getting Managed debugging is not available for this minidump
Writing dump to /home/ppekrol/Downloads/core_20240927_103306
Complete

Output: mini2.txt

ppekrol commented 4 weeks ago

Any update on this?