How is broker.id generated?

[spuder]

I've used this cookbook with kafka 8 and it works great. With kafka 10, the range for valid broker ID's changed and I get an error anytime kafka starts.

The autogenerated broker ID is greater than the maximum number

Contents of config file.

# Generated by Chef.
# Local modifications will be overwritten.

advertised.host.name=dev-kafka01.local
auto.create.topics.enable=true
broker.id=1664189637
default.replication.factor=3
delete.topic.enable=true
log.retention.hours=24
log.roll.hours=24
num.partitions=8
port=6667
reserved.broker.max.id=1000000000

broker.id=1664189637 # 10 digits reserved.broker.max.id=1000000000 # 10 digits

Does this cookbook generate broker.id? or does kafka generate broker.id?

[spuder]

My quirky work around is to set reserved_broker_max_id and then use a node attribute to generate a random number once.

default['kafka']['broker'] = {
  advertised_host_name: Chef::Config[:node_name], 
  reserved_broker_max_id:  1_000_000_000
}

range = 1_000_000..999_999_999
node.set['kafka']['broker']['broker_id'] = rand(range) unless node['kafka']['broker']['broker_id']

[mthssdrbrg]

@spuder: What version are you using? In the latest released one (2.2.2), If you set the node['kafka']['broker']['broker.id'] to nil this cookbook shouldn't generate any ID, and Kafka will itself generate an available ID. If node['kafka']['broker']['broker.id'] isn't set to anything this cookbook will generate an ID from the node's IP address (https://github.com/mthssdrbrg/kafka-cookbook/blob/v2.2.2/recipes/_defaults.rb#L8).

[spuder]

I'm using 2.2.1 which appears to be the latest version (I don't see a 2.2.2)

https://github.com/mthssdrbrg/kafka-cookbook/blob/master/metadata.rb

[spuder]

Could you release 2.2.2 with that change?

[mthssdrbrg]

master is the "development" branch, v2.x is maintained at the v2.x branch.

Looking at it though, this change is already present in v2.2.1, but you need to explicitly set broker_id to nil otherwise this cookbook will generate an ID. So something like this should work:

default['kafka']['broker'] = {
  advertised_host_name: Chef::Config[:node_name], 
  broker_id: nil
}

FWIW, v2.2.2 can be found here or at the Supermarket.

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.