search

linkedin / cruise-control

Cruise-control is the first of its kind to fully automate the dynamic workload rebalance and self-healing of a Kafka cluster. It provides great value to Kafka users by simplifying the operation of Kafka clusters.
https://github.com/linkedin/cruise-control/tags
BSD 2-Clause "Simplified" License
2.74k stars 587 forks source link

KafkaCruiseControlException: java.lang.NullPointerExceptio #1521

Closed klDen closed 3 years ago

klDen commented 3 years ago

Hello!

After trying to execute Rebalance Broker Disks, I get the below stack trace NPE.

Seems like there's a null object in this line https://github.com/linkedin/cruise-control/blob/master/cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/model/Broker.java#L81 according to the stack trace.

CC version used 2.5.46 Kafka version used 2.6

[2021-04-19 10:28:43,426] ERROR Error processing POST request '/rebalance' due to: 'com.linkedin.kafka.cruisecontrol.exception.KafkaCruiseControlException: java.lang.NullPointerException'. (com.linkedin.kafka.cruisecontrol.servlet.KafkaCruiseControlServlet)                                                                                                                            
java.util.concurrent.ExecutionException: com.linkedin.kafka.cruisecontrol.exception.KafkaCruiseControlException: java.lang.NullPointerException                                                                                                                                                                                                                                              
        at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395) ~[?:?]                                                                                                                                                                                                                                                                                               
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2022) ~[?:?]                                                                                                                                                                                                                                                                                                    
        at com.linkedin.kafka.cruisecontrol.servlet.handler.async.AbstractAsyncRequest.getResponse(AbstractAsyncRequest.java:57) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]                                                                                                                                                                                                                     
        at com.linkedin.kafka.cruisecontrol.servlet.handler.AbstractRequest.handle(AbstractRequest.java:41) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]                                                                                                                                                                                                                                          
        at com.linkedin.kafka.cruisecontrol.servlet.KafkaCruiseControlServlet.handlePost(KafkaCruiseControlServlet.java:217) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]                                                                                                                                                                                                                         
        at com.linkedin.kafka.cruisecontrol.servlet.KafkaCruiseControlServlet.doGetOrPost(KafkaCruiseControlServlet.java:125) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]                                                                                                                                                                                                                        
        at com.linkedin.kafka.cruisecontrol.servlet.KafkaCruiseControlServlet.doPost(KafkaCruiseControlServlet.java:104) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]                                                                                                                                                                                                                             
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) ~[javax.servlet-api-3.1.0.jar:3.1.0]                                                                                                                                                                                                                                                                                 
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]                                                                                                                                                                                                                                                                                 
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791) ~[jetty-servlet-9.4.38.v20210224.jar:9.4.38.v20210224]                                                                                                                                                                                                                                                     
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550) ~[jetty-servlet-9.4.38.v20210224.jar:9.4.38.v20210224]                                                                                                                                                                                                                                                 
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.38.v20210224.jar:9.4.38.v20210224]                                                                                                                                                                                                                                           
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.38.v20210224.jar:9.4.38.v20210224]                                                                                                                                                                                                                                          
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.38.v20210224.jar:9.4.38.v20210224]                                                                                                                                                                                                                                           
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435) ~[jetty-server-9.4.38.v20210224.jar:9.4.38.v20210224]                                                                                                                                                                                                                                          
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.38.v20210224.jar:9.4.38.v20210224]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[jetty-servlet-9.4.38.v20210224.jar:9.4.38.v20210224]
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.38.v20210224.jar:9.4.38.v20210224]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.38.v20210224.jar:9.4.38.v20210224]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350) ~[jetty-server-9.4.38.v20210224.jar:9.4.38.v20210224]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.38.v20210224.jar:9.4.38.v20210224]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.38.v20210224.jar:9.4.38.v20210224]
        at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.38.v20210224.jar:9.4.38.v20210224]
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) ~[jetty-server-9.4.38.v20210224.jar:9.4.38.v20210224]
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) [jetty-server-9.4.38.v20210224.jar:9.4.38.v20210224]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) [jetty-server-9.4.38.v20210224.jar:9.4.38.v20210224]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:279) [jetty-server-9.4.38.v20210224.jar:9.4.38.v20210224]
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [jetty-io-9.4.38.v20210224.jar:9.4.38.v20210224]
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [jetty-io-9.4.38.v20210224.jar:9.4.38.v20210224]
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [jetty-io-9.4.38.v20210224.jar:9.4.38.v20210224]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) [jetty-util-9.4.38.v20210224.jar:9.4.38.v20210224]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) [jetty-util-9.4.38.v20210224.jar:9.4.38.v20210224]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) [jetty-util-9.4.38.v20210224.jar:9.4.38.v20210224]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) [jetty-util-9.4.38.v20210224.jar:9.4.38.v20210224]
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383) [jetty-util-9.4.38.v20210224.jar:9.4.38.v20210224]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882) [jetty-util-9.4.38.v20210224.jar:9.4.38.v20210224]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036) [jetty-util-9.4.38.v20210224.jar:9.4.38.v20210224]
        at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: com.linkedin.kafka.cruisecontrol.exception.KafkaCruiseControlException: java.lang.NullPointerException
        at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.computeResult(GoalBasedOperationRunnable.java:162) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]
        at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.RebalanceRunnable.workWithoutClusterModel(RebalanceRunnable.java:114) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]
        at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.computeResult(GoalBasedOperationRunnable.java:168) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]
        at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.RebalanceRunnable.getResult(RebalanceRunnable.java:87) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]
        at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.RebalanceRunnable.getResult(RebalanceRunnable.java:30) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]
        at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.OperationRunnable.run(OperationRunnable.java:45) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]
        at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.run(GoalBasedOperationRunnable.java:35) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        ... 1 more
Caused by: java.lang.NullPointerException
        at com.linkedin.kafka.cruisecontrol.model.Broker.<init>(Broker.java:81) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]
        at com.linkedin.kafka.cruisecontrol.model.Host.createBroker(Host.java:164) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]
        at com.linkedin.kafka.cruisecontrol.model.Rack.createBroker(Rack.java:277) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]
        at com.linkedin.kafka.cruisecontrol.model.ClusterModel.createBroker(ClusterModel.java:935) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]
        at com.linkedin.kafka.cruisecontrol.monitor.LoadMonitor.populateClusterCapacity(LoadMonitor.java:518) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]
        at com.linkedin.kafka.cruisecontrol.monitor.LoadMonitor.clusterModel(LoadMonitor.java:563) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]
        at com.linkedin.kafka.cruisecontrol.KafkaCruiseControl.clusterModel(KafkaCruiseControl.java:350) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]
        at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.ProposalsRunnable.workWithClusterModel(ProposalsRunnable.java:87) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]
        at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.computeResult(GoalBasedOperationRunnable.java:156) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]
        at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.RebalanceRunnable.workWithoutClusterModel(RebalanceRunnable.java:114) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]
        at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.computeResult(GoalBasedOperationRunnable.java:168) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]
        at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.RebalanceRunnable.getResult(RebalanceRunnable.java:87) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]
        at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.RebalanceRunnable.getResult(RebalanceRunnable.java:30) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]
        at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.OperationRunnable.run(OperationRunnable.java:45) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]
        at com.linkedin.kafka.cruisecontrol.servlet.handler.async.runnable.GoalBasedOperationRunnable.run(GoalBasedOperationRunnable.java:35) ~[cruise-control-2.5.46-SNAPSHOT.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        ... 1 more
efeg commented 3 years ago

rebalance_disk is a parameter that tells Cruise Control whether to balance the disk load between logDirs of each broker; hence, it requires JBOD Kafka deployment, where brokers have multiple logDirs (i.e. having a single logDir per broker makes this parameter irrelevant).

Based on logs, your cluster seems to be configured as a non-JBOD cluster. Hence, attempts to rebalance cluster with rebalance_disk=true would fail. You may check this Wiki to learn more about configuring a cluster for JBOD deployments.

That being said, NullPointerException is an unacceptable response. Ideally, the response should (1) fail with an IllegalStateException and (2) provide clear explanation of why the request has failed.