search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.47k stars 4.76k forks source link

API Proposal: GetGCInfo #34648

Closed Maoni0 closed 4 years ago

Maoni0 commented 4 years ago

API to monitor per GC Info. This is meant to be used in situations where folks used to use the .NET Memory counters and provides much richer info than the counters used to provide.

Rationale

We'd like to provide the flexibility for folks who monitor the GC heap on a more detailed level. Previously we had the GetGCMemoryInfo API that provides high level memory info including total available/in use memory, high mem threshold, total heap size and fragmentation. Folks have requested to provide more detailed info in this fashion for monitoring purpose. The difference between this and eventing (ETW, eventpipe) is

At the end of the day, it's a tradeoff between simplicity and the amount of info you can get.

Proposed APIs

public readonly struct GCGenerationInfo
{
    public long SizeBytes;
    public long FragmentationBytes;
}

public sealed class GCInfo
{
    public long HighMemoryLoadThresholdBytes { get; }
    public long MemoryLoadBytes { get; }
    public long TotalAvailableMemoryBytes { get; }
    public long HeapSizeBytes { get; }
    public long FragmentedBytes { get; }

    // The index of this GC
    public long Index { get; }
    // The generation this GC collected
    public int Generation { get; }
    // Did it compact
    public bool Compacted { get; }
    // Was it concurrent
    public bool Concurrent { get; }
    // Total committed bytes
    public long CommittedBytes { get; }

    // How much this GC promoted
    public long PromotedBytes { get; }
    // # of pinned handles this GC encountered
    public long PinnedHandlesCount { get; }
    // # of ready for finalization objects this
    // GC encountered
    public long FinalizeObjectCount { get; }

    // This is the STW pause times for this GC
    // For BGC there are 2 STW pauses.
    public TimeSpan[] PauseDurations { get; }
    // This is the running counter for % pause time
    // in GC, calculated based on the pause durations
    // and total elapsed time.
    public double PauseTimePercentage { get; }

    public GCGenerationInfo[] genInfo { get; }
}

class GC
{
    public static GCInfo GetGCInfo();
}

Comments on the API

All the info provided via this API should be low cost to get.

We might want to add more in the future so suggestions to keep the API extensible are very welcome.

CC @terrajobst @mjsabby

scalablecory commented 4 years ago 
public float PauseDurationMSec[2];

Microseconds? Milliseconds? Can this be a TimeSpan?

Maoni0 commented 4 years ago

MSec is millisecond. microsecond is us.

I don't see much of a point to make it a TimeSpan since in general this duration shouldn't be more than hundreds of ms. ms is what we've always returned in tooling. it seems simple and sufficient.

scalablecory commented 4 years ago

I don't see much of a point to make it a TimeSpan since in general this duration shouldn't be more than hundreds of ms

I propose it for consistency and to disambiguate the unit of measure, not due to float's range/precision.

I'm not aware of any float-based time in BCL, only int for milliseconds or otherwise TimeSpan (ignoring specialized things like Stopwatch). If there are other GC APIs in the BCL that I'm not aware of (likely) that use a float that we want to be consistent with, or there's a great case for making this one of those specialized things, then I'm fine keeping it that way.

Maoni0 commented 4 years ago

not due to float's range/precision.

I was more thinking from the other angle, as in I didn't see TimeSpan as a necessary thing to have here since most of what exists on TimeSpan will be unused for this purpose. people will most likely use this as ms and that's it. they will unlikely use it as hours or minutes.

I'm gonna relying on the API folks to make the right decision here as I don't have strong opinions on this.

also including @stephentoub.

jkotas commented 4 years ago

Is there any advantage in Proposal 1?

Proposal 2 looks like more natural API. The general API guideline is to avoid out arguments except for well-established patterns (e.g. TryParse) or exceptional circumstances.

We might want to add more in the future so suggestions to keep the API extensible are very welcome.

The Proposal 2 is more naturally extensible.

How much of this is specific to the current CoreCLR GC implementation? Is Mono going to be able to provide meaningful implementation of this API?

ghost commented 4 years ago

Tagging @Maoni0 as an area owner

Maoni0 commented 4 years ago

@jkotas I have no preference between the 2 proposals. the only reasons why I had the first one were

I dunno how to extend proposal #2 aside from having an overload that just returns a different type (a type that inherit from this one). I'm not sure if you'd call that "extensible" but I'll go with whatever "the norm" is.

Maoni0 commented 4 years ago

if some of this info does not apply to mono I imagine they can just return 0. I would guess most of it does apply.

jkotas commented 4 years ago

I dunno how to extend proposal #2

You can just keep adding more properties to the class GCInfo

Maoni0 commented 4 years ago

wouldn't that be a breaking change?

jkotas commented 4 years ago

Adding new properties to an existing type is not a breaking change.

Maoni0 commented 4 years ago

ah great; proposal #2 is then.

what's the next step? do I need to attend an API review meeting? if so when?

svick commented 4 years ago

I'm a bit confused by the fact that the proposed GetGCInfo() is a method that can be called at any time, but the descriptions of fields talks about "this GC". Is the method actually providing information about the last GC that completed?

This also raises the question: how is this API meant to be used to monitor the state of GC? Do I call it based on some fixed timer, with the understanding that I might miss information on some GCs (if more than one GC occurred between timer ticks) and will have to deal with duplicates (if no GC occurred between the ticks)? Or do I use it together with WaitForFullGCComplete()? But then, as I understand it, I wouldn't get information on concurrent GCs. Or am I missing something?

stephentoub commented 4 years ago

what's the next step? do I need to attend an API review meeting? if so when?

Next step is an API review meeting. @terrajobst can comment on when that will be for this issue.

stephentoub commented 4 years ago

people will most likely use this as ms and that's it

Part of the issue here is that we've frequently heard that lack of units makes working with time-based APIs confusing, even if the unit is in the name of the thing providing the opaque numerical value. There are long-standing issues like https://github.com/dotnet/runtime/issues/14336 about it.

My preference would be to use TimeSpan (unless there's something about TimeSpan that would actually make it inaccurate?) It removes any confusion, enables simplifying the property name to not require including the unit, enables helpers that operate on TimeSpan or otherwise to pass it to other APIs that take TimeSpans without needing to be concerned about ensuring units match, etc.

stephentoub commented 4 years ago

public float PauseDurationMSec[2];

What is this syntax meant to suggest? This isn't valid C#. If we expect it to always be 2, then it could be exposed as a two separate fields, or if not / we want the flexibility for it to be more, it could be exposed as a TimeSpan[] (or potentially Span<TimeSpan> if that bought us anytyhing), but the 2-ness of it wouldn't be part of the public API.

ah great; proposal #2 is then.

Mind editing your post to be the exact public surface area you're hoping to see then? Currently proposal #2 refers to proposal #1, but at least the way I'm reading it I'm having trouble following exactly what the proposed surface area would be. Thanks.

Maoni0 commented 4 years ago

@stephentoub

What is this syntax meant to suggest?

I wanted to express this is an array with 2 elements. but now that I think about, it's better to just have an array because we might have more elements in this array in the future. also just making sure, were you suggesting I should change float to TimeSpan?

I'll get rid of proposal #1

Maoni0 commented 4 years ago

@svick I suppose "this GC" is not a good way to say "the last GC that finished". but that's what it means, I'll change the wording.

this will provide info on concurrent GCs, I'm wondering what made you conclude it wouldn't? I even specifically mentioned BGC:

    // This is the STW pause times for this GC
    // For BGC there are 2 STW pauses.
    public float PauseDurationMSec[2];

I would expect users to call this at the time interval/period of interest. some people might choose to call it for 10 mins at some interval, once every hour; others might call it always at some interval. it's true that this is subject to the weakness of sampling - this is the same problem as perf counters and people have been using counters for ages. this is a better version of the old .NET memory perf counters as you will know which GCs you are getting and which ones you are missing (and it provides way more info). the idea is sampling would be sufficient for the most part. if you want accurate info then you would need to go the tracing or profiling route.

@mjsabby can you please provide concrete examples how your team might want to use this?

mjsabby commented 4 years ago

I've written this gist as a way to inform our teams how to use the existing APIs for counters. We use internal calls to the 5 new counters that were added as EventCounters but not exposed as APIs.

https://gist.github.com/mjsabby/45df7c2375a230edbd8ec449f587330e

Teams moving from .NET Framework to .NET Core bring up the deficiency of GC counters that are available on .NET Framework but missing from .NET Core.

This API provides the data so that teams can implement Windows performance counters sampling the data using these APIs.

When you have many .net core processes (think ~100) running on one computer and each of them is using EventPipe to get counters, the cost of monitoring these counters adds up. A cheap mechanism of monitoring counters is needed. We think the managed APIs provide the most flexibility to implement a solution that is suitable for all needs.

svick commented 4 years ago

@Maoni0

this will provide info on concurrent GCs, I'm wondering what made you conclude it wouldn't?

My assumption (based on what you're saying, an incorrect one) was that you would want to call GetGCInfo() after every GC. But the best way to do that that I could find is WaitForFullGCComplete(), which does not notify about concurrent GCs. So, if you called GetGCInfo() after every WaitForFullGCComplete() notification, you would not get information about concurrent GCs.

If it's understood that this API is not a good way to get information about every GC, then that's okay with me.

mjsabby commented 4 years ago

@svick this API is in the same vein as a few others like ThreadPool.ThreadCount or Process.WorkingSet64 that give you the state of (in this case the last the GC) information at sample/observation time.

So with something like a timer that fires every X seconds, you can infer the general state of things to a reasonable degree.

So my use case is exactly that, a timer that'll fire every 10 seconds and report it to our counter system.

If we need to reflect in the name of the API that the information is only accurate at the time of access then we should strive to do that. Maybe GetLastGCInfo is a better name?

svick commented 4 years ago

@mjsabby Except some of the proposed members don't work like that. For example, if you check GC.GetGCInfo().GeneralInfo.CommittedBytes every X seconds, you will indeed get a good picture of how that value changes over time. But getting a similar picture based on calling e.g. GC.GetGCInfo().GeneralInfo.Generation or GC.GetGCInfo().PromotionInfo.PromotedBytes every X seconds is harder, because they don't give you the state as of the last GC, they give you information about the last GC.

(And again, as long as these limitations are understood, I'm okay with this proposal.)

Maoni0 commented 4 years ago

all info is based on last GC.

stephentoub commented 4 years ago

also just making sure, were you suggesting I should change float to TimeSpan?

For a time value, TimeSpan is my personal preference over float, yes. Obviously we can discuss it as part of the API review.

GSPP commented 4 years ago

Would it be possible to define an event for when a GC has occurred? This would be a natural complement.

The event could receive a GCInfo for that GC cycle. One issue here is that GCInfo is mutable (the arrays in it). Is it possible to make GCInfo completely immutable from the start?

marek-safar commented 4 years ago

/cc @BrzVlad

BrzVlad commented 4 years ago

These look ok from mono perspective. If we export pause time information per collection, I'd personally export also the duration of the total collection or just for the background collection, with an associated running counter. Even if pause times are small, long collections in the background could indicate a problem with the GC and degraded performance in the application for that period.

GrabYourPitchforks commented 4 years ago

If it's a bunch of read-only properties, then also consider sealing the type and giving it an internal ctor.

terrajobst commented 4 years ago

Couple of questions:

Maoni0 commented 4 years ago
jkotas commented 4 years ago

ReadyForFinalizationObjectCount

Is this going to be the same number that we have called "Finalization Survivors" in .NET Framework? (see https://docs.microsoft.com/en-us/dotnet/framework/debug-trace-profile/performance-counters) If yes, should we use the same name for it?

There's no array, just one. It doesn't need to be allocated each time. We could just allocate one and keep using it. That's implementation details though.

It is not an implementation detail. When you are holding an instance of this info, the values cannot be changing underneath you if somebody else in the process happens to call the API too. It may be better to make this a function that returns an information about specific generation, e.g. public GCGenerationInfo GetGenerationInfo(int generation) so that there are no arrays involved at all.

Maoni0 commented 4 years ago

Is this going to be the same number that we have called "Finalization Survivors" in .NET Framework?

yes, so FinalizationSurvivorCount?

It is not an implementation detail. When you are holding an instance of this info, the values cannot be changing underneath you if somebody else in the process happens to call the API too.

oh I see; I misunderstood the question. yeah, when you are holding an instance of this info, it does not change. that would be really odd. so yes this would be new info every time you call it and the info you already got does not change.

bartonjs commented 4 years ago 
internal class GCInfo { ... }

public struct GCMemoryInfo {
    private GCInfo _info;

    public long PinnedHandlesCount => _info?.PinnedHandlesCount ?? 0;
    public long HighMemoryLoadThresholdBytes => _info?.HighMemoryLoadThresholdBytes ?? 0;
    ...
}
bartonjs commented 4 years ago

Final API:

// Existing type, with modifications
public readonly struct GCMemoryInfo
{
    // Existing properties
    public long HighMemoryLoadThresholdBytes { get; }
    public long MemoryLoadBytes { get; }
    public long TotalAvailableMemoryBytes { get; }
    public long HeapSizeBytes { get; }
    public long FragmentedBytes { get; }

    // New properties
    public long Index { get; }
    public int Generation { get; }
    public bool Compacted { get; }
    public bool Concurrent { get; }
    public long CommittedBytes { get; }
    public long PromotedBytes { get; }
    public long PinnedHandlesCount { get; }
    public long FinalizationPendingCount { get; }
    public ReadOnlySpan<TimeSpan> PauseDurations { get; }
    public double PauseTimePercentage { get; }
    public ReadOnlySpan<GCGenerationInfo> GenerationInfo { get; }
}

// New type
public readonly struct GCGenerationInfo
{
    public long SizeBytes { get; }
    public long FragmentationBytes { get; }
}
NickCraver commented 4 years ago

I'm a little confused on Generation - is that basically MaxGeneration? e.g. the last GC run went up to gen 0, 1, etc.? If so, I humbly suggest a slightly clearer name there.

I'd love to know if compactions happen, but it doesn't look like we could do so in a snapshot manner via this API (I'll try to open suggestions on adding some counter metrics around this). The nature of the API definitely suits some scenarios looking at the last one, but there is an inherent monitoring downside we see in approaches like WaitForFullGCComplete, where we have a thread forever in all dumps and affecting performance timings, wait stats, etc. (making a lot of profiling tools less useful because of the noise).

IMO what's still lacking is overall bits, for example our production looks like this: image Gen 2 is growing because compaction didn't happen. That'd be excellent info to be able to chart but unless we wait on every GC run here (checking for a bool each time), it's still not info that (AFAIK, please correct me if I'm wrong) is attainable from an API.

I like that this is being exposed as a snapshot use case "as of the last GC", which I get that the counters and such are always "as of the last GC" that we're monitoring in that graph as well, but there is a clear distinction in usefulness because those counters are cumulative. Checking them now or later or on any interval is more useful in monitoring scenarios because there isn't any requirement to monitor every one. Typically this happens on some snapshot interval: 15 seconds, 1 minute, 5 minutes, etc. If we do that here, we'd have a lot of gaps.

Are we open to adding corresponding counters on this, e.g. for how many compactions ran overall? Akin to how the existing GC counters work today? I'm not trying to conflate this issue, just trying to understand where we'd be aiming to use this API (which use cases), and where we should lobby for a counter addition instead :)

On the overall API, it occurs to me writing above that the previous properties on GCMemoryInfo are cumulative and monitoring them "late" is juzy fuzzy, but mostly correct. The new properties being combined there are for an inherently ephemeral state which isn't fuzzy, it's very point-in-time. Concurrent? Does that mean the GC is running concurrently? Not quite, it means the last run was...or wasn't. Compacted is very similar, same for Generation. I'm not sure combining what is very much "last run" vs. "overall state" is a great idea from the API design standpoint here. IMO, it's confusing.

Things like ReadOnlySpan<GCGenerationInfo> GenerationInfo do make sense - they're the last known global state. They're not the last known state of a run. I believe these properties are 2 distinct groups.

Here are the global state (good!):

    public long CommittedBytes { get; }
    public long PinnedHandlesCount { get; }
    public long FinalizationPendingCount { get; }
    public ReadOnlySpan<GCGenerationInfo> GenerationInfo { get; }

Here are the properties of the run (bad):

    public long Index { get; }
    public int Generation { get; }
    public bool Compacted { get; }
    public bool Concurrent { get; }
    public long PromotedBytes { get; }
    public ReadOnlySpan<TimeSpan> PauseDurations { get; }
    public double PauseTimePercentage { get; }

I get the want to save types, but it seems very strange to combine these properties of fundamentally different things, IMO. I apologize for chiming in late here - only was pointed at this issue today.

Maoni0 commented 4 years ago

thanks for your feedback, @NickCraver!

I'm a little confused on Generation - is that basically MaxGeneration? e.g. the last GC run went up to gen 0, 1, etc.? If so, I humbly suggest a slightly clearer name there.

this is the generation that gets collected, so it would be 0, 1 or 2. perhaps call it CollectedGeneration or CollectedGenerations since younger gen will get collected too?

Are we open to adding corresponding counters on this, e.g. for how many compactions ran overall? Akin to how the existing GC counters work today?

the thing is we can't make every tool do everything - some tools are meant to be very high level (like the GC counters) for general monitoring, some are meant for diagnostics purpose and others are inbetween. if you want to know in general if compactions happen you could sample by calling this API; or you could do real time ETW monitoring (I know you said this is difficult but we should figure out how to make it easy for you because that is the ultimate way of getting GC data continuously, and with details you can dial). @mjsabby talked about the way he envisioned to use this.

Things like ReadOnlySpan GenerationInfo do make sense - they're the last known global state

not sure what you meant? GenerationInfo is per GC. it only reflects what that GC observed. other things you mentioned like CommittedBytes/PinnedHandlesCount are all for that GC. the only thing that's accumulative is PauseTimePercentage (you could also say Index is accumulative since the general trend for it is to go up).

Maoni0 commented 4 years ago

BTW @mjsabby I wanted to answer this comment from you -

But getting a similar picture based on calling e.g. GC.GetGCInfo().GeneralInfo.Generation or GC.GetGCInfo().PromotionInfo.PromotedBytes every X seconds is harder, because they don't give you the state as of the last GC, they give you information about the last GC.

you could first group these by generation, and make statements like "gen1's PromotedBytes kept going up" so that could explain your gen2 count grows faster. you can then group by another counter like here's the data for "all gen2 GCs that compacted"

Maoni0 commented 4 years ago

for my comment about ETW, also looping in @brianrob - @NickCraver if you could tell us what aspect we could improve to make using ETW easier, I'm sure Brian would be interested to know too.

NickCraver commented 4 years ago

if you want to know in general if compactions happen you could sample by calling this API

I guess I don't understand how that'd work - given we see compactions rarely over the course of days, I'm just not sure sampling would work in any meaningful way. To illustrate: if we observe this once per some interval, we'd miss any compactions that happen between samples, e.g.:

...|GC 1|.....|GC 2|....|Sample|.....|GC 3|....|GC 4 (Compacted)|....|GC 5|....|Sample|...

In this we see the stats from GC 2 and 5 (Compacted : false) for each. It's a fundamental problem with "last stats" vs. an incremented counter.

I do understand that not all tools fit all cases (that's why I'm aiming to see which counters and gauges we should request), but in this case I still see it as a mix of APIs. I get that the values are as of the last GC, but they're not attributes of the last GC. The first group I put are basically gauges - they're the last known values for these things. The second group is values about the GC run itself, and as such the only way to use those values overall would be to call this every single time a GC occurs and translate those values into counters.

I'm not at all opposed to that if that's the only API we'd add here and counters aren't an option, but if that's the case, I'd put forth it would be much better if we could also have an event handler or something to do so. Right now, the only solution there (AFAIK, please tell me if there's another) is to call GC.WaitForFullGCComplete on a dedicated thread which has the overhead and downsides to profiling and such noted above (really throws "wall time" diagnostics). If we had such an event, we could call the API in this issue and increment counters at will (we'd add this to our library in a heartbeat).

I know you said this is difficult but we should figure out how to make it easy for you because that is the ultimate way of getting GC data continuously, and with details you can dial

I'm not sure I'm on the same page there - are we envisioning ETW monitoring some way inside the processes, or from outside? Outside is fundamentally far harder to practically setup and run. It's basically "deploy, configure, and run a whole other app to get this one set of stats". If that's the case, it's just an unreasonably high bar to do by default. With a few counters added, we can do this in-process as we already do with most counters. The goal with constantly monitoring is "what is GC doing?" (along with many other process things - thread pool, etc.), if there's a problem, that's when I'd expect to pull out ETW tracing and setup more detailed monitoring to figure out an issue. We're just trying to see is there an issue 99.9% of the time.

In my above example, it's hard to see why memory is still growing even though we eliminated a lot of allocations and know to have knocked many gigs out of gen 2. It's however readily explained by witnessing that compaction isn't happening. Current options are:

Of course this isn't the only scenario and I'm just using it as an example - in this specific case I really do think a compaction counter is a good add but that's another issue :) The main problem I'd like to raise with this API as-is, is that it's per-run, but it's not easily observable per-run...save an always-dedicated thread to do so. It seems like an odd request for every observer to fire up such a thread, so my ask: is an event that fires on completion a reasonable request so we can easily see this per-run and solve the sampling issue?

Maoni0 commented 4 years ago

if we observe this once per some interval, we'd miss any compactions that happen between samples

I view this as more diagnostics, not monitoring because you already suspect that it's due to lack of compaction and you want to verify that theory; it may not be, it could be that your live data size is growing larger and you'd have to get that info from other tools too. you are asking for compaction now because you happen to have an example where you'd want to know about compaction, but I could see that next time you'd want to know something else and all those are already provided by another tool (ETW). more on ETW below.

however, I do see merit of "highlighting things that I care about that's hard to observe". it seems like something like compacting gen2s should just be part of the actual counters?

The first group I put are basically gauges - they're the last known values for these things. The second group is values about the GC run itself, and as such the only way to use those values overall would be to call this every single time a GC occurs and translate those values into counters.

as I mentioned, the 1st group is not gauges (not in the sense you are describing anyway), for example, the pinned count is just for that GC so you would see a much bigger pinned count for a gen2 GC than for a gen0/1 GC. you'd need to do per generation filter.

I'm not sure I'm on the same page there - are we envisioning ETW monitoring some way inside the processes, or from outside? Outside is fundamentally far harder to practically setup and run.

that's interesting - historically I know that many people really preferred perf counters and they always monitored out of proc and part of the reason was exactly they just deploy a tool per machine and they are done without having to change anything else on the machine :smile:

Setup ETW and go through it (and also learn new tooling to determine that, for most users - let's not ignore there's a bar there)

this is exactly what I'm asking how we could make it easier. I'd think collecting ETW should be automatable (I know teams who have done that); the "go through" part is also automatable with our libraries (TraceEvent already does a lot of processing for you, we have tools on top of that that make it very easy to say "given an etw file, tell me this aspect of the GC, eg, how many compacting gen2 GCs we've done).

there are 2 aspects wrt ETW - one is whether it's a technology you can use at all - because it is out of proc, if that imposes a fundamental obstacle for you that's not really something we can do much about; the other aspect is if it's just a matter of making it easier to parse the results, we can definitely do something about that.

is an event that fires on completion a reasonable request so we can easily see this per-run and solve the sampling issue?

the thing is, I don't think sampling is an issue - people use sampling all the time; it's a very effective way of profiling. it sounds to me that you are saying you simply don't want to sample. if you want to have only accumulative monitoring, the actual counters could be what you want and if there's things lacking there we can definitely discuss; and if you want to diagnose issue, we already have technologies like ETW to do that. we should figure out why those are not usable to you. is this the correct understanding?

please note that I'm perfectly happy to have the discussion - I just don't want to be in a place where we keep inventing new things.

also looping in @tommcdon @noahfalk as this is stepping more into general diagnostics space.

noahfalk commented 4 years ago

A couple suggestions that might help?

It sounds like a challenge with this API is that at any given time the most recent GC is likely to be an ephemeral GC. There is potentially a very narrow/rare window of time when the snapshot values correspond to a gen2 GC, yet this is valuable info we'd like to obtain long after the GC may have happened. Using just a small amount of additional memory we could track statistics for different generation GCs separately and provide information about the last GC of each type. For example the last ephemeral GC, the last foreground gen2 GC, and the last background gen2 GC. This would require some adjustment to the managed API, either returning an array of GCMemoryInfo or overloading GetGCInfo() to take an argument indicating which generation of GC is of interest.

is an event that fires on completion a reasonable request so we can easily see this per-run and solve the sampling issue?

@NickCraver Not sure if you are familiar with the EventListener type, but that may be one way to satisfy what you are looking for?

When Maoni was discussing ETW events above its probably useful to distinguish the event payloads from the mechanism that is used to transport/dispatch the event. The runtime currently supports four different ways of dispatching the same events and ETW is but one way:

  1. EventListener - a fully in-proc managed API, works cross platform, no privilege requirements. Supported for the runtime events starting in .Net Core 2.2
  2. EventPipe - runtime implemented, cross platform, in-proc or out-of-proc, no privilege requirements. The runtime serializes all the events to a windows pipe or unix domain socket in a documented runtime defined format (nettrace). Managed NuGet packages (TraceEvent and Microsoft.Diagnostics.NetCore.Client) are provided for connecting to the stream and parsing the data. Supported starting .Net Core 3.0.
  3. Lttng - events are output via Lttng on Linux in the CTF format, intended for out-of-proc usage. The TraceEvent library and PerfView have some limited support for parsing the portions of the format we emit.
  4. ETW - the classic option we've had for ages and I assume everyone is familiar with

I think your original ask @NickCraver was for some event that tells you the GC is occurring and then you query the snapshot API. I want to warn that all of these events are asynchronous so while it is unlikely that another GC will happen faster than you can respond to the event, it isn't a hard guarantee. However each event also carries a payload of information which should include everything you could have gotten from this API, albeit not necessarily with a strongly typed API facade. We do consider the event data contracts to be the same as API contracts, once the data is documented it won't change.

Maoni0 commented 4 years ago

For example the last ephemeral GC, the last foreground gen2 GC, and the last background gen2 GC.

I really like this suggestion and think this would make the sampling more useful yet still keep it as sampling instead of "now you just get every single GC".

mjsabby commented 4 years ago

I agree, this is a pretty good improvement. Should we take this back to api triage?

Maoni0 commented 4 years ago

yes. I'll take it back to API folks. one thing worth discussing that comes with that is whether we should have different data for BGC and blocking GCs because BGC does have more data (eg the PasueDurations would only have one element for blocking GCs but for BGC there are two. also via discussion with @NickCraver I realized that at least for BGC it would be worthwhile to provide in GenerationData the size and fragmentation on entry of the GC. this data would be very useful for determining how efficient BGC is (ie, how much of the frag it made has been used). this class does get quite big...

NickCraver commented 4 years ago

Thanks @noahfalk and @Maoni0! I am indeed familiar with EventListener, but like the diagnostics API without concrete types it's very "messy" (e.g. you have to know the docs and/or often internals to go and cast everything out - IMO it's a stretch to call the payloads themselves an API). We've had similar lengthy discussions with various teams around DiagnosticListener for pretty much the same reasons - the consumption of these APIs is difficult at best. It may be moot anyway though, since I don't see fragmentation on that API anyway (sanity check: am I looking in the right place? - docs here as well)

So many event listener areas are unique in how to correlate them. For example in the GC instance, I'm guessing we're supposed to correlate on the count as that appears to be a sequence? Unless all garbage collections are guaranteed to be serial (I honestly don't know here - are they?) then the blog post doesn't quite handle the correlation piece - to be completely correct, we'd need to store based on the correlation bits then "log" at the end to anything. That kind of code for everything we observe is quite a nuisance to maintain since the correlation bits are a fun and unique approach in every case (and I want to stress: if there is enough into to correlate - we're fighting a lack of this in other areas like ASP.NET now).

I think for our current case, an event counter for fragmentation (per gen?) would be the ideal (what #31951 appears to be after) - that would be almost zero consumer code and we could see fragmentation. If that's not an option, it would be great to be able to observe this API as needed...if that was possible we could observe and synthesize any counters we wanted. There are just inherent flaws to "stop/start" style eventing on the consumer side, because each consumer is responsible for matching the two up. I've found all over the ecosystem for that in itself to be the most painful part of the consumption experience to get any meaningful data out of APIs.

To be clear, witnessing the last fragmentation value wouldn't require event correlation here, if it's available since that's just a "last seen" value to update (a la a gauge). I'm more explaining in general why we tend to not go towards the eventing APIs for information. In the cases where it's correlated by nature (e.g. Entity Framework does this well), it's not so bad but it's still not as simple as most of the examples :)

Overall: I'm still not sure from the above discussions: how would I get fragmentation numbers from Gen 2 reliably? Are we talking about a thread spinning on WaitForFullGCComplete, and calling this API when it hits? Or is there another suggestion? Or are we implicitly talking about adding fragmentation numbers to the payloads? I'm seeing 2 disconnected states thus far: an API we propose to add fragmentation and other bits to, but without a reliable way to witness it...am I missing a piece?

noahfalk commented 4 years ago

Thanks @NickCraver, let me take a crack at clarifying some of this : )

an API we propose to add fragmentation and other bits to, but without a reliable way to witness it...am I missing a piece?

The proposed way to witness the API discussed here is via sampling. You've expressed some concerns about sampling which may mean this API is suited for use-cases that aren't yours. However I am optimistic we do have alternatives that can still help you do what you want:

  1. Use an EventCounter - We are planning to do this and you suggested it would be ideal so maybe the rest of this list gravy? We can discuss any details about the exact counter(s) in #31951.
  2. Use EventPipe - This one gives you access to strongly typed APIs, usable from in-proc. Potential areas of improvement on this one are:
  3. Use EventListener - This one had some issues when I dug into it further. It could be good with some work but as-is I am removing it from my recommendation list. I'd guess the investments you'd want to see in the runtime before you'd find it easily usable are:
  4. Use the EventListener GC event as a trigger to witness the GCInfo API described here (Example). This one is the closest to your suggestion if we could have an event. I'll guess you were probably picturing GC.OnGCEnd += DoStuff rather than a ~15 line EventListener subclass. Aside from the API ease-of-use which we could look into as a separate issue they do accomplish the same task.

For example in the GC instance, I'm guessing we're supposed to correlate on the count as that appears to be a sequence?

Ideally you could follow the EventPipe sample which uses the TraceManagedProcess helper in TraceEvent to handle all the correlation for you. Looking in a little more detail at the EventListener GC events as-is I'd say they are best suited for cases where you didn't need to do correlation or you only needed some simple correlation. If you ever had to do the full correlation manually for whatever reason the source code for TraceManagedProcess is probably a good guide.

I'm still not sure from the above discussions: how would I get fragmentation numbers from Gen 2 reliably?

Does one of the options 1-4 above sound workable and specific enough to move forward? If one of them sounds promising but still needs a little refinement we can dig in (and potentially do that in a different issue if the approach doesn't involve using the GetGCInfo() API).

Hope this helps : )

Maoni0 commented 4 years ago

thanks @noahfalk.

I'd like to add that the counter mentioned in #31951 uses the exact same sampling idea - you get whatever the last value in that interval happens to be (which is updated by the last GC happened in that interval).

mjsabby commented 4 years ago

Do we have any update from API triage on this?

Maoni0 commented 4 years ago

this is the final shape of the API. PR is coming soon:

    public readonly struct GCGenerationInfo
    {
        public long SizeBeforeBytes { get; }
        public long FragmentationBeforeBytes { get; }
        public long SizeAfterBytes { get; }
        public long FragmentationAfterBytes { get; }
    }
    public enum GCKind
    {
        Ephemeral = 0,    // gen0 or gen1 GC
        FullBlocking = 1, // blocking gen2 GC
        Background = 2   // background GC (always gen2)
    };
    public readonly struct GCMemoryInfo
    {
        /// <summary>
        /// High memory load threshold when the last GC occured
        /// </summary>
        public long HighMemoryLoadThresholdBytes;

        /// <summary>
        /// Memory load when the last GC ocurred
        /// </summary>
        public long MemoryLoadBytes;

        /// <summary>
        /// Total available memory for the GC to use when the last GC ocurred.
        /// </summary>
        public long TotalAvailableMemoryBytes;

        /// <summary>
        /// The total heap size when the last GC ocurred
        /// </summary>
        public long HeapSizeBytes;

        /// <summary>
        /// The total fragmentation when the last GC ocurred
        /// </summary>
        public long FragmentedBytes;

        /// <summary>
        /// The index of this GC.
        /// </summary>
        public long Index ;

        /// <summary>
        /// The generation collected for this GC.
        /// </summary>
        public int Generation;

        /// <summary>
        /// Is this a compacting GC or not.
        /// </summary>
        public bool Compacted;

        /// <summary>
        /// Is this a concurrent GC (BGC) or not.
        /// </summary>
        public bool Concurrent;

        /// <summary>
        /// Total committed bytes.
        /// </summary>
        public long TotalCommittedBytes;

        /// <summary>
        /// Promoted bytes for this GC.
        /// </summary>
        public long PromotedBytes;

        /// <summary>
        /// Number of pinned objects this GC observed.
        /// </summary>
        public long PinnedObjectsCount;

        /// <summary>
        /// Number of objects ready for finalization this GC observed.
        /// </summary>
        public long FinalizationPendingCount;

        /// <summary>
        /// Pause durations. For blocking GCs there's only 1 pause; for BGC there are 2.
        /// </summary>
        public ReadOnlySpan<TimeSpan> PauseDurations;

        /// <summary>
        /// This is the % pause time in GC so far. If it's 1.2%, this number is 1.2.
        /// </summary>
        public double PauseTimePercentage;

        /// <summary>
        /// Generation info.
        /// </summary>
        public ReadOnlySpan<GCGenerationInfo> GenerationInfo;
    }

    class GC
    {
        public static GCMemoryInfo GetGCMemoryInfo();
        public static GCMemoryInfo GetGCMemoryInfo(GCKind kind);
    }
Maoni0 commented 4 years ago

PR