Memory allocation profiling in .NET

[livegrenier]

Hello,

I have been trying to evaluate pyroscope and as part of my POC I have been using the rideshare dot net example in this repo: https://github.com/grafana/pyroscope/tree/main/examples/language-sdk-instrumentation/dotnet/rideshare

Is it expected that I only get stats on CPU:

I would like to see some memory stats to help identify a memory leak, I was under the impression that was supported for dot net based on what the documentation on the Grafana site shows: https://grafana.com/docs/pyroscope/latest/configure-client/language-sdks/dotnet/

Not sure if this is a bug or if I'm doing something wrong, please let me know.

Thanks.

[bryanhuhta]

👋 Can you share more about the issue you're facing? For example:

• Are you having trouble getting memory profiling to work for the rideshare example or your POC (or both)?
• What is the configuration you're using to profile your app?

Without additional information, I would suggest making sure you've followed the steps outlined in the "Getting Started" guide. Additionally, make sure you have this environment variable set to enable memory profiling:

PYROSCOPE_PROFILING_ALLOCATION_ENABLED=true 

Here is a more complete list of the environment variable settings available to you.

[livegrenier]

Hi, Thanks for taking the time to reply, for right now I'm only using the rideshare example, I wanted to show something to our dev team before they start making changes to our apps, I do believe that the rideshare example has: PYROSCOPE_PROFILING_ALLOCATION_ENABLED=true properly set in the Dockerfile, that's why I'm a bit confused on why it is not working.

[bryanhuhta]

Ah gotcha. What will best hep you is process_cpu:alloc_size. While confusingly grouped under the CPU statistics, this will represent the number of bytes a function allocates. You can also utilize process_cpu:alloc_samples to catch functions that might be allocating a lot of times, but not allocating much memory. Unfortunately, our rideshare example doesn't do a great job of illustrating memory allocations, so the resulting flamegraph looks pretty barren. This is something we should address.

The rideshare example should look something like this:

You can see the bulk of the memory allocations are string manipulations of one sort or another.

Some languages support two additional types of memory profiles:

• Inuse Objects - Number of objects allocated
• Inuse Space - Number of bytes allocated

However, the .NET does not profiler does not have that support yet. We have a matrix that details which profile types each language supports: https://grafana.com/docs/pyroscope/latest/view-and-analyze-profile-data/profiling-types/#available-profiling-types.

[bryanhuhta]

You're likely evaluating just our OSS offering, but I will plug our Cloud offering which has a UI that gets more regular updates. Here we've made it a little clearer with the profile types in .NET.

Though, even here "alloc_size" should but under the "memory" grouping.

[livegrenier]

Oh great, thanks for explaining this to me, would you happen to know if in the case of a memory leak, would alloc_size help identify the problem?

[bryanhuhta]

It would absolutely be helpful in identifying most memory leaks. It's especially effective at memory leaks that result in OOM crashes. You would see the function that's allocating grow dramatically in width in the flame graph. To illustrate, I modified the rideshare OrderService class to allocate 1mb every time FindNearestVehicle is called:

private readonly List<int[]> _myLeakyList = new List<int[]>();

public void FindNearestVehicle(long searchRadius, string vehicle)
{
    // 1mb alloc
    this._myLeakyList.Add(new int[1 << 20]);

    lock (_lock)
    {
        var labels = Pyroscope.LabelSet.Empty.BuildUpon()
            .Add("vehicle", vehicle)
            .Build();
        Pyroscope.LabelsWrapper.Do(labels, () =>
        {
            for (long i = 0; i < searchRadius * 1000000000; i++)
            {
            }

            if (vehicle.Equals("car"))
            {
                CheckDriverAvailability(labels, searchRadius);
            }
        });
    }
}

And here is the corresponding flame graph for alloc_size. When comparing this against the unmodified alloc_size flame graph I showed before, you can see a significant growth in the width to FindNearestVehicle calls.

Now, if you have a more insidious leak where the memory being leaked isn't an ever-increasing amount but rather a critical resource that leaks once, this can be more difficult to view in a flame graph. In most applications, this is rarely the case, though.

Edit I'm technically allocating 8mb at a time, but you get the idea 😄

[livegrenier]

Great, thanks again for all the help, extremely appreciated