ERROR Failed to populate broker logDir state

[saritago]

@efeg We are seeing this error message only for cluster state call. All other calls are loading without any issues. We are using 2.0.55 release. If it was a true authorization issue then no other calls should have worked, but we are seeing this error only for a couple of kafka clusters. What could be the issue?

[2019-11-22 09:40:19,256] ERROR Error processing GET request '/kafka_cluster_state' due to: 'Failed to populate broker logDir state.'. (com.linkedin.kafka.cruisecontrol.servlet.KafkaCruiseControlServlet)
java.lang.RuntimeException: Failed to populate broker logDir state.
    at com.linkedin.kafka.cruisecontrol.servlet.response.KafkaClusterState.getJSONString(KafkaClusterState.java:94)
    at com.linkedin.kafka.cruisecontrol.servlet.response.KafkaClusterState.discardIrrelevantAndCacheRelevant(KafkaClusterState.java:469)
    at com.linkedin.kafka.cruisecontrol.servlet.response.AbstractCruiseControlResponse.discardIrrelevantResponse(AbstractCruiseControlResponse.java:49)
    at com.linkedin.kafka.cruisecontrol.servlet.response.AbstractCruiseControlResponse.writeSuccessResponse(AbstractCruiseControlResponse.java:40)
    at com.linkedin.kafka.cruisecontrol.servlet.handler.AbstractRequest.handle(AbstractRequest.java:35)
    at com.linkedin.kafka.cruisecontrol.servlet.KafkaCruiseControlServlet.handleGet(KafkaCruiseControlServlet.java:257)
    at com.linkedin.kafka.cruisecontrol.servlet.KafkaCruiseControlServlet.doGetOrPost(KafkaCruiseControlServlet.java:175)
    at com.linkedin.kafka.cruisecontrol.servlet.KafkaCruiseControlServlet.doGet(KafkaCruiseControlServlet.java:159)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
    at org.eclipse.jetty.server.Server.handle(Server.java:505)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.
    at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
    at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
    at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:104)
    at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:274)
    at com.linkedin.kafka.cruisecontrol.servlet.response.KafkaClusterState.populateKafkaBrokerLogDirState(KafkaClusterState.java:341)
    at com.linkedin.kafka.cruisecontrol.servlet.response.KafkaClusterState.getJsonStructure(KafkaClusterState.java:263)
    at com.linkedin.kafka.cruisecontrol.servlet.response.KafkaClusterState.getJSONString(KafkaClusterState.java:91)
    ... 36 more
Caused by: org.apache.kafka.common.errors.ClusterAuthorizationException: Cluster authorization failed.

[efeg]

@saritago The exception comes from here.

Note that response.logDirInfos() will be empty if and only if the user is not authorized to describe cluster resource. Do you have any specific authorization issues foe these clusters?

[saritago]

Correct this was a cluster level authorization issue. Was resolved by adding below

/apps/install/kafka/bin/kafka-acls.sh  --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=$(cat /apps/install/kafka/config/server.properties | grep 'zookeeper.connect=' | cut -c19-) --add --allow-principal User:* --operation Describe --cluster kafka-cluster
Adding ACLs for resource `Cluster:LITERAL:kafka-cluster`:
     User:* has Allow permission for operations: Describe from hosts: *

Current ACLs for resource `Cluster:LITERAL:kafka-cluster`:
     User:* has Allow permission for operations: Describe from hosts: *

[saritago]

thank you, am closing the issue.