Unable to achieve high throughput using otel kafka exporter

[Praveen2099]

Component(s)

No response

What happened?

Description

I am using below properties :-

Processor:
  batch:
    send_batch_size: 800
    send_batch_max_size: 900
    timeout: 1s

Exporter:
  kafka:
    brokers:
      - 'vip:nodePort'
    producer:
      max_message_bytes: 9000000
      compression: snappy
      required_acks: 1
      #flush_max_messages: 1000
    topic: testing.otlp.topic.new1
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 1000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 120s
    #topic_from_attribute: k8s.namespace.name
    encoding: otlp_json  # raw, otlp_proto
    protocol_version: 2.0.0

Kafka topic

Partition:6
Replica:3

Expected Result

100000 record per minute

Actual Result

1500 record per minute

So anyone can help me to give a configuration file of otel to achieve a higher throughout rate.

Collector version

v0.106.1

Environment information

Environment

OS: (e.g., "Ubuntu 20.04") Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

Processor:
  batch:
    send_batch_size: 800
    send_batch_max_size: 900
    timeout: 1s

Exporter:
  kafka:
    brokers:
      - 'vip:nodePort'
    producer:
      max_message_bytes: 9000000
      compression: snappy
      required_acks: 1
      #flush_max_messages: 1000
    topic: testing.otlp.topic.new1
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 1000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 120s
    #topic_from_attribute: k8s.namespace.name
    encoding: otlp_json  # raw, otlp_proto
    protocol_version: 2.0.0

Log output

No response

Additional context

No response

[github-actions[bot]]

Pinging code owners for exporter/kafka: @pavolloffay @MovieStoreGuy. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[atoulme]

How much memory and CPU did you have allocated to the process? Please try our troubleshooting guide, and in particular perform captures with the pprofextension to find out what is slowing things down.