Add support for System.Diagnostics.Metrics

[shirhatti]

Background and Motivation

dotnet-monitor currently only supports collection of EventCounter metrics. Given that the dotnet team's recommendation is to use System.Diagnostics.Metrics for new metrics, we should expose via the /metrics endpoint as well

Proposed Feature

There currently isn't an out-of-process consumption mechanism for System.Diagnostics.Metrics and whatever we do will be dependent on a new runtime feature. It will probably be something similar to DiagnosticsSourceEventSource

[darraghjones]

Any idea when dotnet-monitor will support the new metrics APIs?

[dominikjeske]

More then one year and nothing?

[pbolduc]

I am glad to see this issue has not been closed, so there is hope it will be completed. However, in the meantime, does anyone have a workaround? Our project is being deployed in Kubernetes. In our environment, I have to use Prometheus scrape. Right now, this is working great for the dotnet-monitor metrics. However, I am missing a lot of other metrics exposed by our application or 3rd party libraries. I could use another sidecar that does metric aggregation. I have heard that the .NET API Gateway Ocelot can aggregate responses. I was thinking it would be possible expose a /metrics endpoint (using OpenTelemetry) in our application and have Ocelot fetch both dotnet-monitor metrics and the application metrics to the Prometheus scrape endpoint. This a more complicated than it really should be.

Does anyone have an example of exporting both dotnet-monitor metrics and System.Diagnostics.Metrics for Prometheus scrape? The metrics aggregator can use any technology. Preferred approach would not require me building a custom container. A stable container with configuration file / environment variables will minimize deployment effort.

Edit: added above comment to new discussion #2798.

[VitaliiTsilnyk]

@pbolduc I ended up using dotnet-monitor side container to export event counters and OpenTelemetry within the application itself to export everything else. Yes it's two scraping endpoints per single application instance and ideally I would love to have it all outside the application itself, but I couldn't find any better way to do it.

[pbolduc]

@pbolduc I ended up using dotnet-monitor side container to export event counters and OpenTelemetry within the application itself to export everything else. Yes it's two scraping endpoints per single application instance and ideally I would love to have it all outside the application itself, but I couldn't find any better way to do it.

Because I have to use Prometheus scrape and need to annotate the pod with the prometheus.io/scrape: true and prometheus.io/port: 52325 annotations. I do not have control of or have ability adjust the scraping service (sysdig) configuration. From my understanding, the prometheus scraping looks for annotations on the pod, not each container in the pod for scraping. I cannot specify multiple scrape endpoints. Perhaps I am not understanding your setup. Do you run your own OTEL collector?

Edit: perhaps this should be moved to discussions instead of this issue.

[darraghjones]

Another option is to ditch the dotnet-monitor sidecar and just use the OpenTelemetry to export both your 'system' metrics and application metrics....you just need to add a few additional nuget packages and wire them us like this....

{
    mb.AddMeter("demo")
    .AddProcessInstrumentation() // needs OpenTelemetry.Instrumentation.Process 
    .AddRuntimeInstrumentation() // needs OpenTelemetry.Instrumentation.Runtime
    .AddAspNetCoreInstrumentation()
    .AddHttpClientInstrumentation()
    .AddConsoleExporter()
    .AddPrometheusExporter();   
});

var app = builder.Build();

app.UseSwagger();
app.UseSwaggerUI();

app.MapPrometheusScrapingEndpoint();

[pbolduc]

Thank you. I am going to take this approach. I have referenced this answer in discussion #2798.

[kkeirstead]

Just wanted to give an update to everyone following this issue - we've started actively working on adding support for System.Diagnostics.Metrics, but we don't have a timeline for it yet. We'll link back to this issue once we have a PR for it. Thanks!

[dominikjeske]

@kkeirstead thanks for info

[vchirikov]

@kkeirstead any news?

[kkeirstead]

@kkeirstead any news?

Sure! We've just finished adding initial limited support for System.Diagnostics.Metrics, which will be available soon in the upcoming 8.0 Preview 1 release. There are currently some limitations (you can learn more here), so we'll still be making some improvements. I'll continue posting updates to this thread once the feature is available.

[kkeirstead]

Quick update - 8.0 Preview 1 was released earlier this week with our initial limited support for System.Diagnostics.Metrics (you can find our release notes here). We're excited to keep making improvements to it and can't wait to hear your feedback!

[jamescarter-le]

Running a .NET 7.0 app and added Metrics from System.Diagnostic.Metrics assuming dotnet-monitor would support it, but I cannot get dotnet-monitor to report them via /metrics endpoint.

Also unsure what the naming convention for what Metrics should be for injestion via dotnet-monitor, My Counter or my_counter or my-counter.

I am running dotnet-monitor 8.0.0-rc.2 to try and get these metrics scraped by Prometheus in an AWS ECS cluster.

Running dotnet-counter correctly reports the metrics.

Specifying the metrics in the settings.config does not help either:

"Metrics": {
    "IncludeDefaultProviders": true,
    "Providers": [
      { "ProviderName": "System.Net.Http" },
      { "ProviderName": "System.Net.Security" },
      { "ProviderName": "MyMetrics", "MetricType": "Meter" },
      { "ProviderName": "TestMetrics.Data", "MetricType": "Meter" },
      { "ProviderName": "TestMetrics.DataSource", "MetricType": "Meter","CounterNames": ["my_counter"] }
    ]
  }

Any assistance would be appreciated, burnt a lot of time thinking that System.Diagnostics.Metrics was the way to go as it's recommended by dotnet docs.

[jander-msft]

I could be wrong, but the .NET Monitor docs might be missing/outdated for the System.Diagnostics.Metrics specification. Meters are specified separately (using the Metrics:Meters configuration section) from counters (using the Metrics:Providers sections). For now, I'll point you to the data model used for binding configuration: https://github.com/dotnet/dotnet-monitor/blob/main/src/Microsoft.Diagnostics.Monitoring.Options/MetricsOptions.cs#L51C41-L51C48

[kkeirstead]

+1 to Justin's comment - we made a change to our initial schema after writing up the docs, and it looks like updating them slipped through the cracks (I believe this is the relevant PR). I've created a new issue to track this to prevent any further confusion around the docs (see here) - thanks for bringing this to our attention!

I've rewritten out your configuration using the new schema - feel free to give this a try and let me know if you have any issues so that we can get you up and running.

"Metrics": {
    "IncludeDefaultProviders": true,
    "Providers": [
      { "ProviderName": "System.Net.Http" },
      { "ProviderName": "System.Net.Security" }
    ],
    "Meters": [
      { "MeterName": "MyMetrics" },
      { "MeterName": "TestMetrics.Data" },
      { "MeterName": "TestMetrics.DataSource","InstrumentNames": ["my_counter"] }
    ]
  }

One additional note - I don't know how you've named your instrument in the target app, but I tend to do "my-counter", which is then how I would include it in the InstrumentNames

[jamescarter-le]

Perfect thank you guys! I hated having to add the OpenTelemetry assemblies to my app when I have a dotnet-monitor sidecar already for dumps clr metrics.

Using Meters in the config for the sources of System.Diagnostics.Metrics works perfectly.