Tracing: Prom SD resolves wrong target/container from Pod

[grelland]

Hello!

I'm running Agent in k8s for forwarding traces to Tempo.

I've instrumented a test application which sends OTLP traces to Agent. The application is a Pod with two containers: an app container and a sidecar (SQL proxy).

The relevant part of the agent config looks like this:

traces:
     # snip
      scrape_configs:
        - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          job_name: kubernetes-pods
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            - action: replace
              source_labels:
                - __meta_kubernetes_namespace
              target_label: namespace
            - action: replace
              source_labels:
                - __meta_kubernetes_pod_name
              target_label: pod
            - action: replace
              source_labels:
                - __meta_kubernetes_pod_container_name
              target_label: container
            - action: replace
              source_labels:
                - __meta_kubernetes_pod_node_name
              target_label: node
          tls_config:
            ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            insecure_skip_verify: false

Now - the traces are flowing just fine through this and ends up in Tempo as expected.

The issue I'm facing, however, is that the spans end up with the container resource attribute from the aforementioned sidecar container, and not the app container - which actually emits the trace.

Note:

• The sidecar is not a scrape target for prometheus at all.
• Debug logs from the agent shows both containers being discovered as targets

I've tried my best to understand if this is a configuration issue, bug or instrumentation issue, but am not getting much wiser. If my understanding is correct the trace processing in Agent uses promsd to resolve the k8s labels for a Pod, but I cannot identify the logic (if any) for handling of multiple targets given a Pod, and how that might map to the source of a trace.

Thanks!

[rfratto]

Hi, sorry for the delay on replying.

The documentation for this functionality states that:

# Configures what methods to use to do association between spans and pods.
# PromSD processor matches the IP address of the metadata labels from the k8s API
# with the IP address obtained from the specified pod association method.
# If a match is found then the span is labeled.
#
# Options are `ip`, `net.host.ip`, `k8s.pod.ip`, `hostname` and `connection`.
#   - `ip`, `net.host.ip` and `k8s.pod.ip`, `hostname` match spans tags.
#   - `connection` inspects the context from the incoming requests (gRPC and HTTP).
#
# Tracing instrumentation is commonly the responsible for tagging spans
# with IP address to the labels mentioned above.
# If running on kubernetes, `k8s.pod.ip` can be automatically attached via the
# downward API. For example, if you're using OTel instrumentation libraries, set
# OTEL_RESOURCE_ATTRIBUTES=k8s.pod.ip=$(POD_IP) to inject spans with the sender
# pod's IP.
#
# By default, all methods are enabled, and evaluated in the order specified above.
# Order of evaluation is honored when multiple methods are enabled.
prom_sd_pod_associations:
  [ - <string> ... ]

Given that, it's always based on the network address of the sender, which isn't particularly useful in this case since both containers share an IP.

The way to try to resolve it would be to modify relabel_configs to drop any container which isn't relevant, although doing this in a generic way which always works for your use case may be unfortunately difficult.

[grelland]

Thanks for the feedback!

That's unfortunate, but I do get the limitation given traces are pushed, so it makes a lot of sense.

Following your suggestion I've implemented a relabelling rule which coalesces a pod annotation and the meta label, allowing us to manually annotate any pods with an override container-name. Sort of solves our very specific use case. Would be nice if there was a more generic solution, however.

Seems like opentelemetry-collector-contrib introduced the ability to resolve container attributes by resolving from the container ID - and then leaving it to SDK/instrumentation to provide said ID. Perhaps adopting this approach would work for Agent too?

[rfratto]

Seems like opentelemetry-collector-contrib https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/20340 to resolve container attributes by resolving from the container ID - and then leaving it to SDK/instrumentation to provide said ID. Perhaps adopting this approach would work for Agent too?

It would take a bit of refactoring of promsdprocessor to implement since it's primarily IP based, but yes, that same approach should also work for the agent too.

[iarlyy]

I have a similar case, however the pod/container name of the resource attributes showing up in some traces, is from another pods (not part of the same deployment)

I've added the ip otel resource attr:

  - name: POD_IP
    valueFrom:
      fieldRef:
        fieldPath: status.podIP
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: "k8s.pod.ip=$(POD_IP)" 

and changed prom_sd_pod_associations in the agent configuration:

prom_sd_pod_associations:
              - "ip"

But still getting wrong association.