[FEATURE REQ] Value of attribute deployment.environment is not being sent to app insights

[nicolaslattuada]

Library name and version

Azure.Monitor.OpenTelemetry.Exporter version 1.0.0-beta.9

Query/Question

Hello We use open telemetry with app insights, and we would like to use the deployment.environment property in order to discriminate between our environments, and not mix metrics reported by production and test environments.

We use the following code in open telemetry:

                .AddAttributes(new Dictionary<string, object>() { { "deployment.environment", "ppe or prod" } });

But unfortunately in the app insights customMetrics table we cannot find this value anywhere.

Please can you advise on what would be the recommended approach in that case?

Environment

No response

[nicolaslattuada]

Ideally I would like to have the exporter library passing this value in the map of customDimensions in app insights

[github-actions[bot]]

Thank you for your feedback. This has been routed to the support team for assistance.

[navba-MSFT]

@nicolaslattuada Thanks for reaching out to us and reporting this issue. The below sample code, creates an instance of AzureMonitorTraceExporter and passes it to AddAzureMonitorTraceExporter() method to configure and build a TracerProvider.

The AzureMonitorTraceExporter is used to export telemetry data to Azure Monitor. You can configure it with the AzureMonitorExporterOptions object to specify the connection string, export interval, metric namespace, and other options.

The Sdk.CreateTracerProviderBuilder() method returns a TracerProviderBuilder object that allows you to configure and build a TracerProvider. You can use it to add sources, instrumentation, exporters, and other components to the TracerProvider.

In this example, the "deployment.environment" attribute is set using the SetTag method of the Activity object. This will ensure that the attribute is included in the telemetry data sent to App Insights through the Azure Monitor Trace Exporter.

using Azure.Monitor.OpenTelemetry.Exporter;
using OpenTelemetry;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using System;
using System.Diagnostics;

public class Program
{
    public static void Main(string[] args)
    {
        var exporter = new AzureMonitorTraceExporter(new AzureMonitorExporterOptions
        {
            ConnectionString = "<your connection string>",
            ExportProcessorType = ExportProcessorType.Simple,
            ExportInterval = TimeSpan.FromSeconds(30),
            MetricNamespace = "MyNamespace"
        });

        using var tracerProvider = Sdk.CreateTracerProviderBuilder()
            .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("MyService"))
            .AddSource("MySource")
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddSqlClientInstrumentation()
            .AddJaegerExporter()
            .AddZipkinExporter()
            .AddConsoleExporter()
            .AddAzureMonitorTraceExporter(options => options = exporter.Options)
            .Build();

        // Create a new activity representing the telemetry data
        var activity = new Activity("MyActivity");
        activity.SetTag("deployment.environment", "ppe or prod");

        // Start the activity and do some work...
        activity.Start();
        // ...

        // Stop the activity when you're finished with the work
        activity.Stop();
    }
}

[github-actions[bot]]

Hi @nicolaslattuada. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[nicolaslattuada]

hi @navba-MSFT Thanks for taking time to answer, I know that I can add tags to activities. Here we are looking for a global solution, I don't want to have to pass the deployment.environment value to every activities. Do you know if there is a way to configure global tags maybe?

[navba-MSFT]

@nicolaslattuada Thanks for getting back. You can configure global tags using the ResourceBuilder object. The ResourceBuilder allows you to create a Resource object that represents the entity producing telemetry data. You can add attributes to the Resource object to represent global tags that apply to all telemetry data produced by the entity.

Here's an example of how to create a Resource object with global tags. Hope this answers.

[github-actions[bot]]

Hi @nicolaslattuada. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[nicolaslattuada]

yes, this is what we are doing, but my point is the values we set in attributes are not being saved in app insights. Please re-read my initial question.

[nicolaslattuada]

And I think this should be added to the AzureMonitorMetricExporter library

[navba-MSFT]

@nicolaslattuada Thanks for clarifying the ask. I am adding the Service team to look into this further.

[github-actions[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @cijothomas @reyang @rajkumar-rangaraj @TimothyMothra @vishweshbankwar.

[navba-MSFT]

@cijothomas @reyang @rajkumar-rangaraj @TimothyMothra @vishweshbankwar Could you please look into this once you get a chance ? Thanks in advance.

[nicolaslattuada]

Hi ! any update on this? Thanks

[nicolaslattuada]

hi @cakescience ! at this point if you have located where the passing of the values happens, how about going ahead and sending a pull request? and add the values we need :)

[treyjones-ms]

Just submitted PR https://github.com/Azure/azure-sdk-for-net/pull/36027 to hopefully address this issue.

[erichkray]

@navba-MSFT where are we at with this? The PR from May 4th is abandoned and closed, can we get an official update so that the Azure Monitor Exporter for .NET sends Resource attributes correctly?

[TimothyMothra]

This is on our backlog. It was mentioned that there may be a workaround but haven't had time to investigate.

[erichkray]

My current workaround is to use a custom Processor.

public class LogResourceProcessor : BaseProcessor<LogRecord>
{
    readonly Resource? _resource = null;

    public LogResourceProcessor(Resource resource)
    {
        _resource = resource;
    }

    public override void OnEnd(LogRecord logRecord)
    {
        if (logRecord is null)
        {
            return;
        }

        if (_resource is not null && _resource.Attributes.Any())
        {
            List<KeyValuePair<string, object?>> stateValues = new();
            if (logRecord.StateValues is not null)
            {
                stateValues.AddRange(logRecord.StateValues);                
            }

            stateValues.AddRange(_resource.Attributes.ToDictionary(k => k.Key, v=> (object?)v.Value));
            logRecord.StateValues = stateValues;
        }
    }
}

[costinbanu]

The workaround above didn't really work for me (on net7, OpenTelemetry 1.6.0) but this one did:

services.AddLogging(loggingBuilder =>
  {
          loggingBuilder.AddOpenTelemetry(opts =>
          {
              opts.AddProcessor(new LogProcessor(new Dictionary<string, object>
              {
                  //global attributes
              }));
              // other logging config
          });
      }
  });

and then

class LogProcessor : BaseProcessor<LogRecord>
{
    private readonly IEnumerable<KeyValuePair<string, object>> _attributes;

    internal LogProcessor(IEnumerable<KeyValuePair<string, object>> attributes)
    {
        _attributes = attributes;
    }

    public override void OnEnd(LogRecord data)
    {
        if (data is not null)
        {
            var allAttributes = new List<KeyValuePair<string, object>>();
            foreach (var a in data.Attributes)
            {
                if (a.Value is not string val)
                {
                    val = JsonConvert.SerializeObject(a.Value, Formatting.Indented);
                }
                allAttributes.Add(KeyValuePair.Create(a.Key, val as object));
            }
            allAttributes.AddRange(_attributes);
            data.Attributes = allAttributes;
        }

        base.OnEnd(data);
    }
}

[LevYas]

@TimothyMothra Are there any news? It's a pity that custom attributes are getting lost

[TimothyMothra]

No update. The bulk of our feedback has been regarding Resource Attributes. We're still discussing this internally to solve it for all languages.

cc @mattmccleary

[shuowpro]

Any update on this? we can get around with the processor with log and trace, but there is no such way in metrics.

[TimothyMothra]

I'm consolidating all issues regarding exporting custom Resource Attributes into another issue to help with tracking. https://github.com/Azure/azure-sdk-for-net/issues/46020