Memory leak problem with Opentelemetry Collector Using KafkaReceiver

Dennis8274 commented 3 months ago

Component(s)

receiver/kafka

What happened?

Description

a memory-leak found in my process,which fetch msg from kafka using sarama,a kafka golang sdk

Steps to Reproduce

Expected Result

Actual Result

Collector version

v0.98.0

Environment information

Environment

go version go1.20.13 darwin/amd64

OpenTelemetry Collector configuration

extensions:
  zpages:
  pprof:

receivers:
  kafka:
    brokers: xxx
    topic: zzz
    encoding:         json
    client_id:        khaos-collector
    group_id:         khaos-collector
    protocol_version: 1.1.1
    initial_offset:   latest
    metadata:
      retry:
        max:     10
        backoff: 3s
    autocommit:
      enable:   true
      interval: 10

processors:
  memory_limiter:
    check_interval: 10s
    limit_percentage: 80
    spike_limit_percentage: 20

exporters:
  nop:

service:
  telemetry:
    logs:
      level: INFO
      encoding: console
      sampling:
        enabled: false
    metrics:
      address: localhost:8888
  extensions: [zpages,pprof]
  pipeline:
    logs:
      receivers: [ kafka ]
      processors: [ memory_limiter ]
      exporters: [ nop ]

Log output

No response

Additional context

No response

github-actions[bot] commented 3 months ago

Pinging code owners:

receiver/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

crobert-1 commented 3 months ago

It looks like the answer from AWS is that this is expected for v1 of the aws-sdk-go package, and there isn't any workaround other than upgrading the dependency. Source: https://github.com/aws/aws-sdk-go/issues/5310

This is essentially a frequency of https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/33208, requesting all components upgrade to v2 of the AWS golang SDK.

If Sarama needs to upgrade its dependency as well, please check there for any open issues, or open a new issue to request it.

Sorry about this, and thanks for filing!

open-telemetry / opentelemetry-collector-contrib