Allow auto setting of broker id

[csghuser]

The latest version of kafka allows the auto configuration of broker id, by not providing one in the configuration file.

As per https://thisdataguy.com/2016/02/04/about-big-kafka-broker-id/

Currently the cookbook will generate a broker id if none is defined with the following:

 node.default.kafka.broker.broker_id = node.ipaddress.gsub('.', '').to_i % 2**31

However, with the latest version of kafka, this auto generated number is higher than the default for:

reserved.broker.max.id

Which means kafka fails to start.

So either the cookbook needs to allow this parameter to be configured, or if no broker id is provided, do not attempt to automatically generate one via the above method using the IP address.

[mthssdrbrg]

There's actually a fix for this in >= v0.9.0 but you'd have to set the attribute to nil. This was my compromise in order to maintain backward compatibility with earlier releases, see #95 and v0.9.0 release notes.

As far as I know it should work, but if you find it to out otherwise please let me know and I'll look into it.

[csghuser]

Hi there, thanks for responding so quickly.

I tested with the following (I've omitted other attributes)

default['kafka']['broker']['port'] = 9092
default['kafka']['broker']['broker']['id']=nil
default['kafka']['init_style'] = :systemd
default['kafka']['scala_version'] = '2.11'
default['kafka']['version'] = '0.9.0.1'

I then end up with:

/opt/kafka/config/server.properties

# Generated by Chef.
# Local modifications will be overwritten.

broker.id=
broker.id=17228947

So I'm not sure if it is something I'm doing wrong? With the above result kafka fails to start with:

[2016-05-07 11:26:25,054] FATAL  (kafka.Kafka$)
java.lang.IllegalArgumentException: requirement failed: broker.id must be equal or greater than -1 and not greater than reserved.broker.max.id
        at scala.Predef$.require(Predef.scala:219)

[mthssdrbrg]

Hm, that's definitely not the expected behaviour. Gonna see if I can reproduce this locally. Which version of the cookbook are you using?

[csghuser]

This is with version kafka-2.0.1, taken from chef supermarket.

Thanks!

[mthssdrbrg]

Alright, I've been able to reproduce it locally, gonna see if I can find a solution and push a new patch release.

[mthssdrbrg]

@csghuser: would it be possible for you to test a version from a branch (optional-broker-configuration-fixup)?

[csghuser]

@mthssdrbrg Yep, that seems to have sorted it!

broker.id is now omitted from the server.properties file with the following set:

default['kafka']['broker']['broker']['id']=nil

Thanks for taking a look at this! Will there be a new release to supermarket soon?

[mthssdrbrg]

@csghuser: That's great, thanks a lot for reporting this. There should be a release out towards the end of today.

[csghuser]

@mthssdrbrg That's great, thanks, very much appreciated.

[mthssdrbrg]

Just released v2.0.2, so closing this. @csghuser please let me know if you run into any further issues :)

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.