TOPIC_AUTHORIZATION_FAILED

[pmdcosta]

Hi, I have followed the steps in README, but I am getting the following error, in both the producer and consumer :

[2016-06-06 10:35:35,592] WARN Error while fetching metadata with correlation id 0 : {test=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
[2016-06-06 10:35:35,681] WARN Error while fetching metadata with correlation id 1 : {test=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
[2016-06-06 10:35:35,784] WARN Error while fetching metadata with correlation id 2 : {test=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
[2016-06-06 10:35:35,886] WARN Error while fetching metadata with correlation id 3 : {test=TOPIC_AUTHORIZATION_FAILED} (org.apache.kafka.clients.NetworkClient)
...

Kafka logs also show errors:

[2016-06-06 10:26:08,702] INFO Registered broker 0 at path /brokers/ids/0 with addresses: SSL -> EndPoint(c7001.symantec.dev.com,9093,SSL) (kafka.utils.ZkUtils)
[2016-06-06 10:26:08,705] INFO Kafka version : 0.9.0.1 (org.apache.kafka.common.utils.AppInfoParser)
[2016-06-06 10:26:08,705] INFO Kafka commitId : 23c69d62a0cabf06 (org.apache.kafka.common.utils.AppInfoParser)
[2016-06-06 10:26:08,706] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2016-06-06 10:26:08,932] ERROR [KafkaApi-0] error when handling request Name:UpdateMetadataRequest;Version:1;Controller:0;ControllerEpoch:1;CorrelationId:0;ClientId:0;AliveBrokers:0 : (EndPoint(c7001.symantec.dev.com,9093,SSL));PartitionState: (kafka.server.KafkaApis)
kafka.common.ClusterAuthorizationException: Request Request(0,192.168.70.101:9093-192.168.70.101:60583,Session(User:CN=c7001.symantec.dev.com,OU=test,O=test,L=test,ST=test,C=test,c7001.symantec.dev.com/192.168.70.101),null,1465172768903,SSL) is not authorized.
    at kafka.server.KafkaApis.authorizeClusterAction(KafkaApis.scala:910)
    at kafka.server.KafkaApis.handleUpdateMetadataRequest(KafkaApis.scala:158)
    at kafka.server.KafkaApis.handle(KafkaApis.scala:74)
    at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
    at java.lang.Thread.run(Thread.java:745)
[2016-06-06 10:26:38,614] ERROR [KafkaApi-0] error when handling request Name:LeaderAndIsrRequest;Version:0;Controller:0;ControllerEpoch:1;CorrelationId:1;ClientId:0;Leaders:BrokerEndPoint(0,c7001.symantec.dev.com,9093);PartitionState:(test,0) -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0) (kafka.server.KafkaApis)
kafka.common.ClusterAuthorizationException: Request Request(0,192.168.70.101:9093-192.168.70.101:60583,Session(User:CN=c7001.symantec.dev.com,OU=test,O=test,L=test,ST=test,C=test,c7001.symantec.dev.com/192.168.70.101),null,1465172798607,SSL) is not authorized.
    at kafka.server.KafkaApis.authorizeClusterAction(KafkaApis.scala:910)
    at kafka.server.KafkaApis.handleLeaderAndIsrRequest(KafkaApis.scala:113)
    at kafka.server.KafkaApis.handle(KafkaApis.scala:72)
    at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
    at java.lang.Thread.run(Thread.java:745)
[2016-06-06 10:26:38,617] ERROR [KafkaApi-0] error when handling request Name:UpdateMetadataRequest;Version:1;Controller:0;ControllerEpoch:1;CorrelationId:2;ClientId:0;AliveBrokers:0 : (EndPoint(c7001.symantec.dev.com,9093,SSL));PartitionState:[test,0] -> (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:1),ReplicationFactor:1),AllReplicas:0) (kafka.server.KafkaApis)
kafka.common.ClusterAuthorizationException: Request Request(0,192.168.70.101:9093-192.168.70.101:60583,Session(User:CN=c7001.symantec.dev.com,OU=test,O=test,L=test,ST=test,C=test,c7001.symantec.dev.com/192.168.70.101),null,1465172798616,SSL) is not authorized.
    at kafka.server.KafkaApis.authorizeClusterAction(KafkaApis.scala:910)
    at kafka.server.KafkaApis.handleUpdateMetadataRequest(KafkaApis.scala:158)
    at kafka.server.KafkaApis.handle(KafkaApis.scala:74)
    at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
    at java.lang.Thread.run(Thread.java:745)
[2016-06-06 10:26:46,552] INFO Processing notification(s) to /kafka-acl-changes (kafka.common.ZkNodeChangeNotificationListener)

Any idea what could be happening ?

[supermonk]

Hi, Can you run the below command and check what it is listing on server and client side, generally while running the act command you might have a copy paste error. If any error just destroy the vagrant box and run the commands again

sh kafka_2.11-0.9.0.1/bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --list

[reiabreu]

Getting the same error. Here are the permissions

Current ACLs for resourceTopic:test`: User:* has Allow permission for operations: Write from hosts: 192.168.70.101 User:* has Allow permission for operations: Read from hosts: 192.168.70.102

Current ACLs for resource Group:group102: User:* has Allow permission for operations: Read from hosts: 192.168.70.102

Current ACLs for resource Cluster:kafka-cluster: User:* has Allow permission for operations: All from hosts: 192.168.70.101`

[supermonk]

I will test that right now and update asap

[supermonk]

https://github.com/Symantec/kafka-security-0.9/commit/2e9dace9fff9ab60f3a14f91866ec09132f16fcf

There was a space issue in the shell Script Which I have addressed..

Can you try again... Below are the screenshots.

[supermonk]

@pmdcosta @reiabreu : Let me know if it worked or not?

[reiabreu]

I had notice the missing space and issued the correct command. I'll try it again and get back to you.

[reiabreu]

@supermonk @pmdcosta I started again from scratch using only the provided instructions and I can confirm it's working. Cheers man. @supermonk: Do ACLs only work with SSL?

[supermonk]

@reiabreu : Cool .. I did not verify that, but I think it might work without SSL , as every consumer has a consumer group.. it should. ( But Without encryption and just using ACL I guess no security as a whole)

@reiabreu : If you try different setup with kerberos or any other thing and have any additional changes which might help others... let me know I can merge them..

[reiabreu]

@supermonk For now, I'm interested in the ACLs only. If SSL is mandatory for that (which makes sense), I will set it up accordingly. Your project is a great resource for that.

[reiabreu]

@supermonk, ACL work on your vagrants without SSL. However, I'm unable to reproduce this behaviour outside vagrant. Is there any step outside the scripts that I need to do on the server side to enable ACLs? Cheers

[supermonk]

@reiabreu : Cool... btw.. which port did u use 9092? for ACL without SSL?

No nothing much.. Couple of check point..

right Kafka Version and below scripts as needed.. /vagrant/data/step1-all.sh => update software, install java, kafka, zoo) sh /vagrant/data/step2server.sh => Become CA root, generate public and private key) sh /vagrant/data/step3client.sh => generates ca request and puts in shared folder /vagrant/data)

https://github.com/Symantec/kafka-security-0.9/tree/master/Scripts

[reiabreu]

Yes, I enabled 9092 and also kept 9093. I was able to consume using both ports and the ACLs worked on both

[reiabreu]

@supermonk

[supermonk]

@reiabreu cool.. I will close the issue now..

[skanduri1987]

Hello, After Enabling the ACL i am getting below error:

Error while fetching metadata with correlation id 56 : {acl-test1=TOPIC_AUTHORIZATION_FAILED}

Below is the command we are using

sh kafka-acls.sh --authorizer-properties zookeeper.connect=

<IP>:2181,<IP>:2181,<IP>:2181

--add --allow-principal User:admin --operation Write --topic acl-test1 --group=*

Any help is really appreciated.

at the consumer level, while consuming we are getting below error?

18/11/17 22:22:14 ERROR tools.ConsoleConsumer$: Unknown error when running consumer:

org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized to access group: console-consumer-5647

to overcome this issue I used below command but it still it is not working.

sh kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=<IP>:2181,<IP>:2181,<IP>:2181 --add --allow-principal User:santosh --topic acl-test --consumer --group=*

can you please guide me with clear steps that need to follow. I am using CDH 5.15.1 and kafka

3.1.0-1.3.1.0.p0.35.

--