Invalid Argument Error on `store_offset` Method in rdkafka-ruby

[fake-soul]

Description: I am encountering an issue with the rdkafka gem (version 0.18.0) when attempting to call the @consumer.store_offset(message) method. The error message returned is:

Local: Invalid argument or configuration (invalid_arg)
Error: the evaluation of `e.instance_variable_get(:backtrace)` failed with the exception '`backtrace' is not allowed as an instance variable name'

Environment:

• Gemfile:

  gem 'rdkafka', "0.18.0"

• Gemlock:

  rdkafka (0.18.0)
      ffi (~> 1.15)
      mini_portile2 (~> 2.6)
      rake (> 12)

• Ruby Version: Ruby 3.1.0

Reproduction Steps:

1. Create a consumer using the rdkafka gem.
2. Attempt to store an offset for a message:

   message = <Message in 'topic1' with key 'test-key-10', payload 'Hello, Kafka! 10', partition 0, offset 530, timestamp 2024-11-04 10:30:48 +0530>
   @consumer.store_offset(message)

Error Trace: The error occurs at the following locations:

• Line 413 in ../.rvm/gems/ruby-3.1.0/gems/rdkafka-0.18.0/lib/rdkafka/consumer.rb

Additional Information: This issue appears to have started from version rdkafka 0.15.1, as the same code works correctly with version rdkafka 0.15.0.

I would appreciate any guidance or fixes regarding this issue.

Thank you!

[mensfeld]

Works:

require 'rdkafka'

config = {
  :"bootstrap.servers" => "localhost:9092",
  :"group.id" => "ruby-test3",
  :'auto.offset.reset' => 'earliest',
  :'enable.auto.offset.store' => false
}
producer = Rdkafka::Config.new(config).producer
delivery_handles = []

100.times do |i|
  puts "Producing message #{i}"
  delivery_handles << producer.produce(
      topic:   "ruby-test-topic",
      payload: "Payload #{i}",
      key:     "Key #{i}"
  )
end

delivery_handles.each(&:wait)

consumer = Rdkafka::Config.new(config).consumer
consumer.subscribe("ruby-test-topic")

consumer.each do |message|
  puts "Message received: #{message}"
  consumer.store_offset(message)
end

Based on provided details, things operate as expected. If you want me to revisit this, please follow those guidelines: https://karafka.io/docs/Support/#issue-reporting-guide

This issue may occur if you do not follow the docs that clearly state:

When using this enable.auto.offset.store should be set to false in the config.

[mensfeld]

Also keep in mind that this library is a low-level binding to librdkafka. You may be better using karafka that is built on top of it.