[kafkareceiver] failed to unmarshal message {"kind": "receiver", "name": "kafka/metrics", "data_type": "metrics", "error": "proto: MetricsData: wiretype end group for non-group"}

narayanbhawar10 commented 2 months ago

Component(s)

receiver/kafka

What happened?

Description

Getting error while using kafka reciever to get metrics

Steps to Reproduce

Expected Result

Actual Result

Collector version

Latest

Environment information

Environment

OS: (e.g., "Ubuntu 20.04") Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
kafka:
protocol_version: 2.0.0
topic: mykafka
brokers:
- "localhost:9092"
resolve_canonical_bootstrap_servers_only: true
metadata:
retry:
max: 10
backoff: 5s
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
grpc:
endpoint: 0.0.0.0:4317
processors:
batch:
timeout: 5s
send_batch_size: 100000
resource:
attributes:
- key: test.key
value: "test-value"
action: insert

exporters:
debug:
verbosity: normal
clickhouse:
endpoint: tcp://127.0.0.1:9000?dial_timeout=10s
database: otel
async_insert: true
ttl: 72h
compress: lz4
create_schema: true
logs_table_name: otel_logs
traces_table_name: otel_traces
metrics_table_name: otel_metrics
timeout: 5s
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 300s

extensions:
health_check: {}
zpages: {}

service:
extensions: [zpages, health_check]
pipelines:
logs:
receivers: [ kafka ]
processors: [ batch ]
exporters: [ clickhouse ]
traces:
receivers: [ kafka ]
processors: [ batch ]
exporters: [ clickhouse ]
metrics:
receivers: [ kafka ]
processors: [ batch ]
exporters: [ clickhouse ]

Log output

2024-08-25T14:38:37.456+0530  info  kafkareceiver/kafka_receiver.go:551  Starting consumer group  {"kind": "receiver", "name": "kafka/metrics", "data_type": "metrics", "partition": 0}
2024-08-25T14:38:38.946+0530  debug  kafkareceiver/kafka_receiver.go:561  Kafka message claimed  {"kind": "receiver", "name": "kafka/metrics", "data_type": "metrics", "value": "Hello 11", "timestamp": "2024-08-25T14:38:38.942+0530", "topic": "quickstart"}
2024-08-25T14:38:38.946+0530  error  kafkareceiver/kafka_receiver.go:580  failed to unmarshal message  {"kind": "receiver", "name": "kafka/metrics", "data_type": "metrics", "error": "proto: MetricsData: wiretype end group for non-group"}

Additional context

No response

github-actions[bot] commented 2 months ago

Pinging code owners:

receiver/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

iblancasa commented 2 months ago

Could you format the config properly? @narayanbhawar10

wildum commented 2 months ago

Hello, the problem that you have is that you specify a topic "topic: mykafka" but you try to use it for your metrics and logs and traces pipelines. A topic should only contain one signal: either metrics, logs or traces. You have two options to solve your problem:

either you have three different topics named exactly "otlp_metrics", "otlp_tracees" and "otlp_spans". Then you can remove the "topic: mykafka" to use the default version per pipeline.
or if you want different names for your topics, you will need to create 3 kafka receivers in your config

more details about it in this ticket: https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/32735

open-telemetry / opentelemetry-collector-contrib