hypertrace / hypertrace-ingester

Streaming jobs for Hypertrace
Other
13 stars 16 forks source link

Adds resource attribute enricher #376

Open Harnoor7 opened 1 year ago

Harnoor7 commented 1 year ago

Description

This PR enriches the spans using their resource tags adding additional infra metadata context. Infra metadata context can be quite useful in scenarios like:

  1. Users want to debug an application behaving abnormally in a certain cluster. The application could be running in multiple clusters but Hypertrace ignores the resource tags and users cannot differentiate via resource tags.
  2. Users want to see metrics of applications running in different environments. (Example: P99 latency of service running in stage cluster or any k8s node etc.)

Resource attributes can be attached to spans using agents or collectors before the span reaches the Hypertrace ingestion pipeline. Few ways to attach infra metadata context to spans:

  1. Using the K8s attribute processor of otel-collector to append resource tags with infra tags.
  2. Using agents running on nodes like Otel-agent or Jaeger agent to attach tags like - (cluster.name, node.name) with the help of environment variables.
Screenshot 2023-03-11 at 12 45 10 AM

Sample Enricher config:

ResourceAttributeEnricher {
    class = "org.hypertrace.traceenricher.enrichment.enrichers.ResourceAttributeEnricher"
    attributes = ["pod.name","node.name","cluster.name","ip","service.version","node.selector"]
    attributesToMatch {
        pod.name = "host.name"
    }
  }

class : Java class to refer for this enricher. attributes : Resource attributes which would be added to the span attribute if they exist in the process tags. attributesToMatch: Could be used if a resource attribute needs to refer to some other attribute (For example the Jaeger client library would add k8s pod name to span process tags as 'host.name').

Testing

Added required test cases and the change is live on the production cluster for 3+ months now.

Checklist:

Documentation

N/A