Open Telemetry headers setter extension not working when any processor is added to the trace pipeline

[SimranCode]

Component(s)

No response

What happened?

When I use Headers Setter extension to set the Tenant ID and Sampling Processor or any processor for that matter for Traces in Otel Collector Config, the data doesnt get exported to the endpoint.

Both of these features works fine when added individually.

Could you provide any workaround for this if there is no fix available for now?

Collector version

0.91

Environment information

No response

OpenTelemetry Collector configuration

extensions:
  headers_setter:
    headers:
      - action: upsert
        key: X-Scope-OrgID
        from_context: tenantid
receivers:
  otlp:
    protocols:
      grpc:
        include_metadata: true
        endpoint: 0.0.0.0:5555
      http:
        include_metadata: true

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
    const_labels:
      tenant: tenant1

  otlp:
    endpoint: "tempo:4317"
    tls:
      insecure: true
    auth:
      authenticator: headers_setter

  loki:
    endpoint: "http://loki:3100/loki/api/v1/push"
    auth:
      authenticator: headers_setter 

processors:
  tail_sampling:
    decision_wait: 1s
    num_traces: 1000
    expected_new_traces_per_sec: 100
    policies:
      [
          {
            name: latency-policy,
            type: latency,
            latency: {threshold_ms: 1000}
          }        

      ]

  transform:
    metric_statements:
      - context: datapoint
        statements:
        - set(attributes["tenantname"], resource.attributes["tenantid"])
  batch:
    timeout: 1s
    send_batch_size: 1024

service:
  extensions: [headers_setter]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [transform]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: []
      exporters: [loki]

Log output

No response

Additional context

No response

[github-actions[bot]]

Pinging code owners for extension/headerssetter: @jpkrohling. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[jpkrohling]

Can you test replacing the tail-sampling with the transform processor in the traces pipeline and see if it works? Header setter will not work with processors that reassemble incoming batches, such as the batch, group by, or tail-sampling processors.

[SimranCode]

Thanks for a quick response !!

Is there any workaround by which we can use both Sampling Processor and headers setter extension ?

[SimranCode]

@jpkrohling - Is there any other way possible to Sample the data and implement multitenancy at the same time?

[jpkrohling]

Right now, I don't think it's possible to keep the tenant information from your original context after the telemetry data goes through a tail-sampling processor. We have discussed that in the past (cc @bogdandrutu and @tigrannajaryan), especially around adding a TContext (telemetry context) in pdata, but that never got past the PoC stage.

[github-actions[bot]]

Pinging code owners for processor/tailsampling: @jpkrohling. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• processor/tailsampling: @jpkrohling
• extension/headerssetter: @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[jpkrohling]

We could use a similar approach to the batching processor, which allows keys to be added to the resulting context.

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• processor/tailsampling: @jpkrohling
• extension/headerssetter: @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[stanislawcabalasamsung]

Hello, I've encountered this issue. I've been using batch processor succesfully with passing required context. After adding tailsampling processor, it does not pass context , as noted in this issue.

I think that implementing similar feature, as in batch processor is good idea. Is this something considered to be implemneted? If so, is there any estimate when it might be done?