Support perfmon for core CLR

[preetamshetty1]

AB#1117227 Of this list, things we depend on and would like perf counters for are below. • GC Heap Size • Gen 0 GC Rate • Gen 1 GC Rate • Gen 2 GC Rate • % Time in GC • Gen 0 Heap Size • Gen 1 Heap Size • Gen 2 Heap Size • LOH Heap Size • Allocation Rate • Exception Rate • Monitor Lock Contention Rate Other components of .NET Core also publishes counters: ASP.NET Core Microsoft.AspNetCore.Hosting providerRequests per second • Total Requests Count • Current Requests Count • Failed Requests Count

[benaadams]

Some are already provided by dotnet-counters? https://github.com/dotnet/diagnostics/blob/master/documentation/dotnet-counters-instructions.md

[hoyosjs]

We have dotnet-counters if you want to just see them. Also there's EventListener with the "System.Runtime" provider for in-proc listening. There's also https://www.nuget.org/packages/Microsoft.Diagnostics.NETCore.Client/ to consume them out of proc. However, notice the versioning of this library - it's not on a 1.0 release meaning that there could be breaking changes coming up. @sywhang

[hoyosjs]

Looks like @benaadams beat me to it :)

[preetamshetty1]

My request here was for windows performance counters which we have had in the framework in the past. I understand that these are from Trace counters that just spit out values. Most services have infra to track and upload windows performance counters at scale. If I understand correctly, the tool about is just pushing trace counters out to a CSV? Custom collectors etc. would be needed to do this on a large scale service.

[preethikurup]

Re-opening, since this request came in again.

[sywhang]

If I understand correctly, the tool about is just pushing trace counters out to a CSV? Custom collectors etc. would be needed to do this on a large scale service.

dotnet-counters either displays it in real-time on the console or writes it to a JSON/CSV, but you can export the data into any format you'd like using Microsoft.Diagnostics.NETCore.Client NuGet package: https://www.nuget.org/packages/Microsoft.Diagnostics.NETCore.Client.

While I'm against the idea of having the runtime write these data into Windows performance counter directly, it's possible to write a library that reads the counter values from the runtime using the Microsoft.Diagnostics.NETCore.Client library, and exports them into Windows perfmon using the PerformanceCounter API.

[tommcdon]

Moving this item to "Triaged - Future" because we do not currently plan on exposing .NET Core performance counters though perfmon (https://docs.microsoft.com/en-us/windows-server/administration/windows-commands/perfmon) in .NET 5.

[zeronetworks-devops]

Would love to get this support, this is used by many people in the industry

[indy-singh]

Is there any chance of this getting done? All of our monitoring relies on perfmon/typeperf. dotnet-counters isn't really an option for us, as we only deploy the runtime and not the sdk which is needed to install the tool.

We tried downloading directly from the repo, and it "works". But we feed all of the information from std out into a format that our monitoring software (OMD) understands.

Cheers, Indy

[tommcdon]

we only deploy the runtime and not the sdk which is needed to install the tool.

Hello @indy-singh! Our EXE download option does not have an SDK dependency but does require a runtime install. Please see the "direct download" option on https://learn.microsoft.com/dotnet/core/diagnostics/dotnet-counters. Would this option work for you?

[indy-singh]

Hey @tommcdon!

We already tried that and it isn't a great experience to hook into our existing monitoring seam as we pull info from stdout;

We basically execute this script every few seconds to collect information (see attached script).

param($strArgAppPool)

$perfdata = @{};

foreach($a in "$($strArgAppPool).blue","$($strArgAppPool).green","$strArgAppPool") {    
    $perf = @{
        'WorkingSetSize' = '0bytes;;';
        'Gen0HeapSize' = '-1bytes;;';
        'Gen1HeapSize' = '-1bytes;;';
        'Gen2HeapSize' = '-1bytes;;';
        'LOHSize' = '-1bytes;;';
        'InAllHeaps' = '-1bytes;;';
        'NumberOfPinnedObjects' = '-1objects;;';
        'NumberGen0Collections' = '-1c;;';
        'NumberGen1Collections' = '-1c;;';
        'NumberGen2Collections' = '-1c;;';
        'TimeinGC' = '-1c;;';
        'FinalizationSurvivors' = '-1items;;';
        'PrivateBytes' = '-1bytes;;';
    }   
    $perfdata.Add($a, $perf);
}

$strQueryAppPools = Get-WmiObject -Query "SELECT * FROM Win32_Process WHERE Name='w3wp.exe' AND CommandLine LIKE '%$strArgAppPool%'";

$outputString = ""
$statusString = ""

foreach ($oAppPool in $strQueryAppPools) {
    $appPoolName =  $oAppPool.CommandLine -replace ".*w3wp.exe -ap `"([^`"]+)`".*",'$1'; #`"

    if($perfdata[$appPoolName] -eq $null) {
        continue;
    }

    $statusString +=  "OK: w3wp ($appPoolName PID: $($oAppPool.ProcessId)) ";
    $perfdata[$appPoolName]['WorkingSetSize'] = "$($oAppPool.WorkingSetSize)bytes;;";

    $dotNetQueryJob = Get-WmiObject -Query "SELECT * FROM Win32_PerfRawData_NETFramework_NETCLRMemory WHERE ProcessId=$($oAppPool.ProcessId)" -AsJob;
    $perfCounterQueryJob = Get-WmiObject -Query "SELECT * FROM Win32_PerfRawData_PerfProc_Process WHERE IDProcess=$($oAppPool.ProcessId)" -AsJob;

    Wait-Job -Job $dotNetQueryJob,$perfCounterQueryJob -Timeout 3 | Out-Null;
    $dotNetQuery = Receive-Job -Job $dotNetQueryJob;
    $perfCounterQuery = Receive-Job -Job $perfCounterQueryJob;
    Remove-Job -Force -Job $dotNetQueryJob,$perfCounterQueryJob;

    foreach ($NetMemory in $dotNetQuery) {
        $perfdata[$appPoolName]['Gen0HeapSize'] = "$($NetMemory.Gen0heapsize)bytes;;";
        $perfdata[$appPoolName]['Gen1HeapSize'] = "$($NetMemory.Gen1heapsize)bytes;;";
        $perfdata[$appPoolName]['Gen2HeapSize'] = "$($NetMemory.Gen2heapsize)bytes;;";
        $perfdata[$appPoolName]['LOHSize'] = "$($NetMemory.LargeObjectHeapsize)bytes;;";
        $perfdata[$appPoolName]['InAllHeaps'] = "$($NetMemory.NumberBytesinallHeaps)bytes;;";
        $perfdata[$appPoolName]['NumberOfPinnedObjects'] = "$($NetMemory.NumberofPinnedObjects)objects;;";
        $perfdata[$appPoolName]['NumberGen0Collections'] = "$($NetMemory.NumberGen0Collections)c;;";
        $perfdata[$appPoolName]['NumberGen1Collections'] = "$($NetMemory.NumberGen1Collections)c;;";
        $perfdata[$appPoolName]['NumberGen2Collections'] = "$($NetMemory.NumberGen2Collections)c;;";
        $perfdata[$appPoolName]['TimeinGC'] = "$($NetMemory.PercentTimeinGC)c;;";
        $perfdata[$appPoolName]['FinalizationSurvivors'] = "$($NetMemory.FinalizationSurvivors)items;;";
    }

    foreach($perfCounter in $perfCounterQuery) {
        $perfdata[$appPoolName]['PrivateBytes'] = "$($perfCounter.PrivateBytes)bytes;;";
    }
}

foreach($appPoolName in $perfdata.Keys) {
    foreach($counter in $perfdata[$appPoolName].Keys) {
        $outputString += "$($appPoolName)_$counter=$($perfdata[$appPoolName][$counter]) ";
    }
}

Write-Host "$statusString|$outputString"
Exit 0;

Thanks, Indy

[infraweavers]

Alternatively, is there way of getting the stdout format of the command in a computer-parsable format like JSON or XML? It looks from the flags that you can either: write a JSON/XML file or you can "see" the output from stdout. If there was an output option which was easy to work with, then it seems like that could be a reasonable compromise?

[indy-singh]

Hi,

Is there any news on this?

dotnet-counters doesn't work for us at all in this scenario as we can't easily shell out the data. For now we've had to deploy our own exe using https://www.nuget.org/packages/Microsoft.Diagnostics.NETCore.Client/ which isn't a super great for a collecting diagnostic.

Cheers, Indy

[JeffBarnard]

Alternatively, is there way of getting the stdout format of the command in a computer-parsable format like JSON or XML? It looks from the flags that you can either: write a JSON/XML file or you can "see" the output from stdout. If there was an output option which was easy to work with, then it seems like that could be a reasonable compromise?

You can use MetricListener api or OpenTelemetry.NET to export the data into json format. https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/docs/metrics/extending-the-sdk/README.md#exporter