search

open-telemetry / opentelemetry-collector

OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
4.36k stars 1.44k forks source link

[kafkaexporter] Unable to setup separate pipelines for metrics vs traces to different kafka topics #2334

Closed nshampur closed 3 years ago

nshampur commented 3 years ago

Describe the bug Is it possible to use the same collector instance to export traces and metrics to distinct kafka topics

Steps to reproduce From reading the documentation and testing, it appears the kafka exporter configuration is a singleton element in the configuration. Attempting to leverage the same exporter in pipelines for metrics and traces, results in an odd behavior where the telemetry is sent to the kafka topic that defines the exporter first.

What did you expect to see? The ability to define a pipeline where metrics and traces are send to distinct kafka topics. If this limitation is by design, what is the recommendation for a configuration/setup where metrics, telemetry and logs can be sent to different kafka topics. The intention is to use the kafka receiver as the consumer on the other end. It's also not clear how one would configure the topic per telemetry type in the configuration.

What did you see instead? it seemed indeterminate which topic the telemetry goes to. It appears that the first exporter in the pipeline that gets activated by exporters_builder seems to be target. In some cases, metrics were sent to the traces topic (oltp_traces which is the default topic name) and some times to metrics topic.

What version did you use? Version: 0.17.0

What config did you use? `receivers: hostmetrics:
scrapers: cpu:

processors: batch:

exporters: logging: logLevel: debug

exporters: logging: logLevel: debug kafka: protocol_version: 2.0.0 brokers:

extensions: health_check: pprof: zpages:

service: extensions: [health_check,pprof,zpages] pipelines: traces: receivers: [otlp] processors: [batch] exporters: [logging, kafka] metrics: receivers: [otlp, hostmetrics] processors: [batch] exporters: [logging, kafka] `

Environment OS: Windows

tigrannajaryan commented 3 years ago

From reading the documentation and testing, it appears the kafka exporter configuration is a singleton element in the configuration.

This is not correct. You can have as many kafka exporters as you want, each having a different configuration. All exporters allow this. See the Collector design: https://github.com/open-telemetry/opentelemetry-collector/blob/master/docs/design.md

It should look roughly like this:

exporters:
  kafka/traces:
    topic: spans
    ...

  kafka/metrics:
    topic: metrics
    ...
service:
  pipelines:
    traces:
      exporters: [kafka/traces]
      ...
    metrics:
      exporters: [kafka/metrics]
      ...
nshampur commented 3 years ago

thanks for the clarification

dharmaplay commented 2 years ago

Hi @tigrannajaryan , seems it is not working, I tried to setup separate pipeline but I am not receiving the Metrics and Traces to separate pipeline ( I mean to separate topics, not defaulted topic ). Please refer https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/7585