[QUERY] Is it possible to configure Logging & Metrics without adding many GB of Tracing & Dependencies?

[af4jm]

Library name and version

Azure.Monitor.OpenTelemetry.AspNetCore 1.2.0

Query/Question

we're trying to use OpenTelemetry to push logs & metrics into AppInsights from our isolated process Azure Functions, but if we use the suggested .AddOpenTelemetry().UseAzureMonitor() in addition to the small amount of logging & metrics, we get a large amount of traces and appdependencies. With the Classic SDK there were simple options to turn that off.

Question is how to turn that off with OpenTelemetry? Our current init code is below

        services.ConfigureOpenTelemetryMeterProvider(builder =>
        {
            builder.AddMeter("snip"); // creates 6 Meters, with the same strings we were using as namespaces with the ClassicSDK
        });

        var telemetryBuilder = services.AddOpenTelemetry();
        telemetryBuilder.UseAzureMonitor(options =>
        {
            options.Diagnostics.IsDistributedTracingEnabled = false; // no effect on reducing data ingest
        });

        telemetryBuilder.WithLogging();
        telemetryBuilder.WithMetrics(builder =>
        {
            builder.AddAzureMonitorMetricExporter();
            builder.AddHttpClientInstrumentation();
        });

Paying for several GB of tracing & dependencies data per day to be ingested is not an option for us. The metrics & exception logging is well under 1GB per day, with a high signal to noise ratio while the dependencies & tracing is all noise & no signal & gets in the way of seeing what we need to see (not to mention multiplying our ingest spend by 20-25x)

Environment

No response

[cijothomas]

Can you share a minimal reproducible example that shows too many telemetry being collected?

[af4jm]

I'm not sure I can, because we're not sure what's generating all the data... this app is 4 functions projects with about 2 dozen functions... some only run once per day or hourly, but a couple of them run tens of thousands of times per day (the app collects inventory updates (~40-50k items) & syndicates that out to multiple ad partners, including Bing search)

I can show you this query result from our Insights workspace

we're considering adding this just after the above lines, but it seems VERY hacky to try fishing things out of the service provider (and this seems like it would just get the traces not the dependencies)

        var toRemove = services.Where<ServiceDescriptor>(t =>
            t.ImplementationType == typeof(TracerProvider));
        foreach (var item in toRemove ?? Enumerable.Empty<ServiceDescriptor>())
        {
            services.Remove(item);
        }

[af4jm]

in case it helps, 99% of our logging is either exception logs, info/debug logs (not too many of those), and our Counters & Histograms that we base our alerts off of (e.g. avg inventory count per store, count of items in poison queues)

[cijothomas]

Sorry, I can't really help much without a minimal repro.

If the distro is producing anything more than what is documented here : https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-add-modify?tabs=aspnetcore#automatic-data-collection, then we can help fix that.

Also, I don't think the code snippet shared is doing what the doc shows. For instance, there is no AddAzureMonitorMetricExporter required when using distro. AddHttpClientInstrumentation is also not required. Please ensure you are following the doc correctly.

 telemetryBuilder.WithMetrics(builder =>
    {
        builder.AddAzureMonitorMetricExporter();
        builder.AddHttpClientInstrumentation();
    });

[cijothomas]

Oops! Just realized you are on Azure Functions. Then this distro is not what you want. Check Functions docs here : https://learn.microsoft.com/en-us/azure/azure-functions/opentelemetry-howto?tabs=app-insights&pivots=programming-language-csharp and report issues, if any, to that repo.

[cijothomas]

Also check this https://github.com/Azure/azure-functions-dotnet-worker/issues/2634

[af4jm]

my team tells me that any time one of our Functions makes an HTTP request that 404s, we get a mess of stuff in the logs

[github-actions[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @cijothomas @rajkumar-rangaraj @reyang @TimothyMothra @vishweshbankwar.

[TimothyMothra]

Hi @af4jm, There's a lot of details here, let me know if i've missed something.

1) Azure Functions Azure Functions has historically written their own wrappers around telemetry SDKs to provide extra value. This is something my team can't help with. If you've having a negative experience with regard to functions, please report this on their repo so they can help triage.

my team tells me that any time one of our Functions makes an HTTP request that 404s, we get a mess of stuff in the logs

This is a red flag for me that Functions may be doing something extra. If you were using only Azure Monitor, I would expect a single failed Dependency telemetry item.

2) Is your goal to have zero traces and dependencies, or fewer?

The `UseAzureMonitor()` api applies recommended configurations to OpenTelemetry. You can view that configuration [here](https://github.com/Azure/azure-sdk-for-net/blob/230bb725c55891374d39a66e27cce4a4adb5d6aa/sdk/monitor/Azure.Monitor.OpenTelemetry.AspNetCore/src/OpenTelemetryBuilderExtensions.cs#L78). 

If you want a very custom configuration, you may consider onboarding to our Exporter instead. This would give you finer control over each individual component.

OpenTelemetry has a concept of 3 signals: Traces (Requests and Dependencies), Metrics, and Logs.

FYI Application Insights stores OpenTelemetry Logs in a table called "Traces".

- Regarding Application Insights "Traces" / OpenTelemetry Logs
    If you want zero Application Insights "Traces", you could opt to not configure the OpenTelemetry Logging Provider.
    If you want fewer, take a look and see if either the LogLevel or Log Filter rules can meet your needs. ([ILogger Doc](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/logging/?view=aspnetcore-8.0#default-log-level)).

- Regarding Application Insights "Dependencies" / OpenTelemetry Traces
    Each individual type of dependencie (ie: HTTP or SQL) is collected by a specific Instrumentation Library. These individual libraries have their own Filter delegates which gives you control. ([Instrumentation.Http Readme](https://github.com/open-telemetry/opentelemetry-dotnet-contrib/tree/main/src/OpenTelemetry.Instrumentation.Http#filter-httpclient-api))

3) Your code sample

- The Functions guide has some other code samples that I don't see in your shared code.

Please make sure you're following their best practices. https://learn.microsoft.com/en-us/azure/azure-functions/opentelemetry-howto?tabs=app-insights&pivots=programming-language-csharp#3-enable-opentelemetry-in-your-app

- your code sample has both `UseAzureMonitor` and `AddAzureMonitorMetricExporter`. `UseAzureMonitor` configures all three signals so the `AddAzureMonitorMetricExporter` is not needed.

[TimothyMothra]

It's been a few weeks without any response, so I'm closing this issue. If you have any follow up questions, please feel free to reopen the issue or open a new issue to continue the conversation.