OTel pipeline improvements

[petewall]

• Set service.instance.id to k8s.pod.uid if not set
• Drop unneeded traces using a Filter Processor: http.route= /live|/healthy|/ready
  • Make this default and configurable
• Explore including otelcol.processor.resourcedetection to gather cloud and resource information
  • Will need smart defaults and make it configurable

[cyrille-leclerc]

• Explore including otelcol.processor.resourcedetection to gather cloud and resource information

  • Will need smart defaults and make it configurable

A challenge with using the Resource Detection Processor is that this processor is intended to run on the host that produces the telemetry, typically as a DaemonSet when deployed on Kubernetes.

Otherwise, it is likely to produce erroneous attributes. Typically, host.name, host.id, or host.ip will be erroneous if the Alloy instance doesn't run on the same host as the instrumented workloads. Attributes like cloud.availability_zone, cloud.region, or k8s.cluster.name are less likely to be erroneous.