grafana / grafana-opentelemetry-dotnet

Grafana's distribution of OpenTelemetry instrumentation for .NET
Apache License 2.0
37 stars 6 forks source link

Is it possible to set loki labels here? #100

Closed chinwobble closed 1 month ago

chinwobble commented 5 months ago

Question

Hi,

I am wondering if it is possible to set the loki labels here using this library?

Use Github Discussions.

chinwobble commented 1 month ago

@matt-hensley I've used this library with the following code to ship logs.

services
  .AddOpenTelemetry()
   .WithLogging(o =>
            {
                o.AddConsoleExporter();
                o.AddOtlpExporter((options, rules) =>
                {

                    rules.ExportProcessorType = OpenTelemetry.ExportProcessorType.Batch;
                    var endpoint = configuration["GRAFANA_CLOUD_OTLP_ENDPOINT"];
                    var instanceId = configuration["GRAFANA_CLOUD_INSTANCE_ID"];
                    var apiToken = configuration["GRAFANA_CLOUD_API_TOKEN"];
                    options.Endpoint = new Uri(endpoint + "/v1/logs");
                    options.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.HttpProtobuf;
                    options.Headers = $"Authorization=Basic {EncodeToBase64(instanceId!, apiToken!)}";
                });
            });

I can see that the logs are being sent successfully to grafana open telemetry endpoint. https://otlp-gateway-prod-au-southeast-0.grafana.net/otlp/v1/logs

Response code is 204.

How can I find the logs in my grafana cloud instance? I don't see how labels are set? Should be exploring them in Loki? Do I need a certain version of the grafana cloud stack running for this to work?

chinwobble commented 1 month ago

I was able to figure this issue out. Loki will only promote certain resource attributes to loki labels. These include:

        // Loki promotes these resource attributes to Loki labels and persists others as Loki message metadata: cloud.availability_zone, cloud.region, container.name, deployment.environment, k8s.cluster.name, k8s.container.name, k8s.cronjob.name, k8s.daemonset.name, k8s.deployment.name, k8s.job.name, k8s.namespace.name, k8s.pod.name, k8s.replicaset.name k8s.statefulset.name, service.instance.id, service.name, service.namespace.
``
This means you cannot do anything in this library to control loki labels.
I used this code 
```c#
services
  .AddOpenTelemetry()
   .WithLogging(o =>
            {
                o.AddOtlpExporter((options, rules) =>
                {
                    var resourceBuilder = ResourceBuilder.CreateEmpty();
                    resourceBuilder.AddAttributes(new Dictionary<string, object>
                    {
                        // this will be a loki label since due to the mapping done by grafana's open telemetry endpoint.
                        ["service.name"] = "my-service",
                        // this won't be a loki label
                        ["env"] = "prod",
                    });
                    rules.ExportProcessorType = OpenTelemetry.ExportProcessorType.Batch;
                    var endpoint = configuration["GRAFANA_CLOUD_OTLP_ENDPOINT"];
                    var instanceId = configuration["GRAFANA_CLOUD_INSTANCE_ID"];
                    var apiToken = configuration["GRAFANA_CLOUD_API_TOKEN"];
                    options.Endpoint = new Uri(endpoint + "/v1/logs");
                    options.Protocol = OpenTelemetry.Exporter.OtlpExportProtocol.HttpProtobuf;
                    options.Headers = $"Authorization=Basic {EncodeToBase64(instanceId!, apiToken!)}";
                });
            });
matt-hensley commented 4 weeks ago

Thank you for posting a solution! I've been OOO, apologies for the delayed response.