Nested env variables not resolved, unable to use downward API

tiithansen commented 1 month ago

Component(s)

auto-instrumentation

What happened?

Description

We want traces to be exporter to OTEL collector running on the same host/node. For that purpose we use downward API to get status.hostIP and reference it in OTEL_EXPORTER_OTLP_ENDPOINT end, but OTEL operator does not resolve it and exporter gets invalid IP.

Steps to Reproduce

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: instrumentation
spec:
  propagators:
  - tracecontext
  - baggage
  sampler:
    type: parentbased_traceidratio
    argument: {{ .Values.instrumentation.sampler.parentbased_traceidratio | quote }}
  env:
  - name: OTEL_BSP_MAX_EXPORT_BATCH_SIZE
    value: "512"
  - name: OTEL_EXPORTER_HOST_IP
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP
  nodejs:
    env:
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: "http://$(OTEL_EXPORTER_HOST_IP):4317"

Expected Result

Env OTEL_EXPORTER_OTLP_ENDPOINT should contain correct address with host ip.

Actual Result

Env OTEL_EXPORTER_OTLP_ENDPOINT contains string as http://$(OTEL_EXPORTER_HOST_IP):4317 instead which is not resolved.

Kubernetes Version

1.28

Operator version

version: 0.47.0 appVersion: 0.93.0

Collector version

Custom

Environment information

No response

Log output

No response

Additional context

No response

pavolloffay commented 1 month ago

instrumentation: Instrumentation now automatically add node and pod ips as env vars OTEL_NODE_IP and OTEL_POD_IP to instrumented containers. (#2769)

The env vars with the IPs are already injected

pavolloffay commented 1 month ago

Could you please paste here the pod spec after the instrumentation is injected?

pavolloffay commented 1 month ago

This is the code for setting the exporter endpoint https://github.com/open-telemetry/opentelemetry-operator/blob/5bafbfdceb4f44328e157fd6c96ea256daf48bc4/pkg/instrumentation/sdk.go#L288

I am not sure why it should not work

fyuan1316 commented 1 month ago

Expected Result Env OTEL_EXPORTER_OTLP_ENDPOINT should contain correct address with host ip.

https://github.com/open-telemetry/opentelemetry-operator/blob/5bafbfdceb4f44328e157fd6c96ea256daf48bc4/pkg/instrumentation/sdk.go#L73-L76

After looking at the source code implementation, I think there might be some misunderstanding here.

By design both nodejs.env and Instrumentation spec.env end up injecting pod container[0].env. No string substitution occurs during the injection process. Therefore, the value of OTEL_EXPORTER_OTLP_ENDPOINT will always be "http://$(OTEL_EXPORTER_HOST_IP):4317".

So this could be a potential requirement rather than a bug? @pavolloffay @tiithansen

open-telemetry / opentelemetry-operator