search

Yelp / kafka-utils

Apache License 2.0
313 stars 128 forks source link

Does Kafka-utils support SASL #209

Open cocotier opened 5 years ago

cocotier commented 5 years ago

Does Kafka-utils support SASL with Kafka Client properties?

djoul2706 commented 5 years ago

Hi, I'm also interested by that subject. Found any solution yet ?

iMajna commented 4 years ago

@cocotier @djoul2706 are you referring to Kafka protocol or? I managed to run kafka-utils processes under SASL_PLAINTEXT since if you dig into a code a bit kafka-utils is almost pure reading of Zookeepers znodes (which are already recommended to be in world:r mode) and using http and ssh

tmendenhall commented 4 years ago

@iMajna How did you change the configuration settings to allow for a SASL connection? I am interested in allowing for that type of connection.

iMajna commented 4 years ago

@tmendenhall Just to clarify, kafka_utils in my case didn't behave like client which authenticated to Kafka cluster I think that is not intention atm.

My cluster sample looks the same as for Kafka cluster which supports only PLAINTEXT protocol. So no changes are needed in sample_config.yml!

Important thing to know it that Kafka clusters which are Kerberized by default are putting host to null, thats a known issue. Like this: If we try to get the information for kafka broker znode after enabling kerberos, it shows as below:

get /brokers/ids/1002 {"jmx_port":-1,"timestamp":"1312312312","endpoints":["PLAINTEXTSASL://xxxx.domain.com:9092"],"host":null,"version":2,"port":-1}

Thats the reason why you can't read any hosts while trying to execute rolling_restart for example.kafka_utilsis reading host and since it is null you get null as feedback.

The only way to handle this is to manually change it. Be aware that after each restart of broker host var in zookeeper is changed to default null again

The default ‘host’ and ‘port’ here in Zookeeper are used to maintain backward compatibility for older Kafka clients, but only PLAINTEXT protocol is supported as default. If the broker does not listen on PLAINTEXT protocol, for example PLAINTEXTSASL after kerberizing, an empty endpoint (that is, "host":null,"port”:-1) will be registered and older clients will break.

Changing host property in Zookeeper: [zk: localhost:2181(CONNECTED) 2] set /brokers/ids/1001 {"jmx_port":-1,"timestamp":"1312312312","endpoints":["PLAINTEXTSASL://txxxx.domain.com:9092"],"host":"txxxx.domain.com","version":3,"port":9092}

Hope you find this helpful. :)

jeffwidman commented 4 years ago

The underlying kafka-python lib supports some SASL setups...