search

pinterest / DoctorK

DoctorK is a service for Kafka cluster auto healing and workload balancing
Apache License 2.0
633 stars 92 forks source link

Error in starting DoctorKafkaMain 0.2.4.3 #89

Open venkatwilliams opened 5 years ago

venkatwilliams commented 5 years ago

Did anyone faced this issue.

I am using kafka_2.11-1.1.0 and doctorkafka 0.2.4.3 to run doctorkafka.

Running the process: java -server \ -cp libs/*:doctorkafka-0.2.4.3-jar-with-dependencies.jar \ com.pinterest.doctorkafka.DoctorKafkaMain \ server /Users/venkat/doctorkafka/drkafka/config/doctorkafka.dev.yaml

Console Log: ERROR StatusLogger No Log4j 2 configuration file found. Using default configuration (logging only errors to the console), or user programmatically provided configurations. Set system property 'log4j2.debug' to show Log4j 2 internal initialization logging. See https://logging.apache.org/log4j/2.x/manual/configuration.html for instructions on how to configure Log4j 2 INFO [2019-01-30 10:22:07,989] io.dropwizard.server.DefaultServerFactory: Registering jersey handler with root path prefix: / INFO [2019-01-30 10:22:07,991] io.dropwizard.server.DefaultServerFactory: Registering admin handler with root path prefix: / INFO [2019-01-30 10:22:07,991] io.dropwizard.assets.AssetsBundle: Registering AssetBundle with name: assets for path /* INFO [2019-01-30 10:22:08,113] com.pinterest.doctorkafka.util.ZookeeperClient: Initialize curator with zkurl:localhost:2181 INFO [2019-01-30 10:22:08,146] org.apache.curator.utils.Compatibility: Running in ZooKeeper 3.4.x compatibility mode INFO [2019-01-30 10:22:08,162] org.apache.curator.framework.imps.CuratorFrameworkImpl: Starting INFO [2019-01-30 10:22:08,226] org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT INFO [2019-01-30 10:22:08,226] org.apache.zookeeper.ZooKeeper: Client environment:host.name=10.50.1.202 INFO [2019-01-30 10:22:08,226] org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.8.0_191 INFO [2019-01-30 10:22:08,226] org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation INFO [2019-01-30 10:22:08,226] org.apache.zookeeper.ZooKeeper: Client environment:java.home=/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/jre INFO [2019-01-30 10:22:08,226] org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=libs/kafkastats-0.2.4.3-jar-with-dependencies.jar:libs/doctorkafka-0.2.4.3-jar-with-dependencies.jar:libs/doctorkafka-0.2.4.3.jar:libs/kafkastats-0.2.4.3.jar:doctorkafka-0.2.4.3-jar-with-dependencies.jar INFO [2019-01-30 10:22:08,226] org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/Users/venkat/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:. INFO [2019-01-30 10:22:08,226] org.apache.zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/var/folders/28/qrgydfnx1jv4cl703nzp3lhc92z1zs/T/ INFO [2019-01-30 10:22:08,226] org.apache.zookeeper.ZooKeeper: Client environment:java.compiler= INFO [2019-01-30 10:22:08,226] org.apache.zookeeper.ZooKeeper: Client environment:os.name=Mac OS X INFO [2019-01-30 10:22:08,226] org.apache.zookeeper.ZooKeeper: Client environment:os.arch=x86_64 INFO [2019-01-30 10:22:08,226] org.apache.zookeeper.ZooKeeper: Client environment:os.version=10.13.6 INFO [2019-01-30 10:22:08,226] org.apache.zookeeper.ZooKeeper: Client environment:user.name=venkat INFO [2019-01-30 10:22:08,226] org.apache.zookeeper.ZooKeeper: Client environment:user.home=/Users/venkat INFO [2019-01-30 10:22:08,226] org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/Users/venkat/softwares/doctorkafka INFO [2019-01-30 10:22:08,227] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@1352434e INFO [2019-01-30 10:22:08,244] org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) INFO [2019-01-30 10:22:08,245] org.apache.curator.framework.imps.CuratorFrameworkImpl: Default schema INFO [2019-01-30 10:22:08,272] org.apache.zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session INFO [2019-01-30 10:22:08,281] org.apache.zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1689a86df6e004d, negotiated timeout = 60000 INFO [2019-01-30 10:22:08,288] org.apache.curator.framework.state.ConnectionStateManager: State change: CONNECTED INFO [2019-01-30 10:22:08,515] org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181/cluster1 sessionTimeout=30000 watcher=org.I0Itec.zkclient.ZkClient@69d787b5 INFO [2019-01-30 10:22:08,515] org.I0Itec.zkclient.ZkEventThread: Starting ZkClient event thread. INFO [2019-01-30 10:22:08,516] org.I0Itec.zkclient.ZkClient: Waiting for keeper state SyncConnected INFO [2019-01-30 10:22:08,517] org.apache.zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) INFO [2019-01-30 10:22:08,517] org.apache.zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181, initiating session INFO [2019-01-30 10:22:08,519] org.apache.zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1689a86df6e004e, negotiated timeout = 30000 INFO [2019-01-30 10:22:08,519] org.I0Itec.zkclient.ZkClient: zookeeper state changed (SyncConnected) INFO [2019-01-30 10:22:08,530] kafka.utils.Log4jControllerRegistration$: Registered kafka:type=kafka.Log4jController MBean 15:52:08.598 [pool-6-thread-1] ERROR com.pinterest.doctorkafka.DoctorKafkaMain - DoctorKafka start failed java.lang.NullPointerException: null at java.util.Hashtable.put(Hashtable.java:460) ~[?:1.8.0_191] at com.pinterest.doctorkafka.util.KafkaUtils.getKafkaConsumer(KafkaUtils.java:101) ~[kafkastats-0.2.4.3-jar-with-dependencies.jar:?] at com.pinterest.doctorkafka.replicastats.ReplicaStatsManager.readPastReplicaStats(ReplicaStatsManager.java:208) ~[doctorkafka-0.2.4.3-jar-with-dependencies.jar:?] at com.pinterest.doctorkafka.DoctorKafka.start(DoctorKafka.java:51) ~[doctorkafka-0.2.4.3-jar-with-dependencies.jar:?] at com.pinterest.doctorkafka.DoctorKafkaMain.lambda$run$0(DoctorKafkaMain.java:68) ~[doctorkafka-0.2.4.3-jar-with-dependencies.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_191] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_191] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191] INFO [2019-01-30 10:22:08,650] com.twitter.ostrich.stats.LatchedStatsListener$$anon$1: Starting LatchedStatsListener INFO [2019-01-30 10:22:08,831] com.twitter.ostrich.admin.AdminHttpService: Admin HTTP interface started on port 2052. INFO [2019-01-30 10:22:08,888] io.dropwizard.server.ServerFactory: Starting DoctorKafkaMain INFO [2019-01-30 10:22:08,937] org.eclipse.jetty.setuid.SetUIDListener: Opened application@2bfaba70{HTTP/1.1,[http/1.1]}{0.0.0.0:8080} INFO [2019-01-30 10:22:08,938] org.eclipse.jetty.server.Server: jetty-9.4.z-SNAPSHOT; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12 INFO [2019-01-30 10:22:09,309] io.dropwizard.jersey.DropwizardResourceConfig: The following paths were found for the configured resources:

GET     /api/cluster (com.pinterest.doctorkafka.api.ClusterApi)
DELETE  /api/cluster/{clusterName}/admin/maintenance (com.pinterest.doctorkafka.api.MaintenanceApi)
GET     /api/cluster/{clusterName}/admin/maintenance (com.pinterest.doctorkafka.api.MaintenanceApi)
PUT     /api/cluster/{clusterName}/admin/maintenance (com.pinterest.doctorkafka.api.MaintenanceApi)
GET     /api/cluster/{clusterName}/broker (com.pinterest.doctorkafka.api.BrokerApi)

INFO [2019-01-30 10:22:09,311] org.eclipse.jetty.server.handler.ContextHandler: Started i.d.j.MutableServletContextHandler@2faa55bb{/,null,AVAILABLE} INFO [2019-01-30 10:22:09,320] org.eclipse.jetty.server.AbstractConnector: Started application@2bfaba70{HTTP/1.1,[http/1.1]}{0.0.0.0:8080} INFO [2019-01-30 10:22:09,320] org.eclipse.jetty.server.Server: Started @2696ms WARN [2019-01-30 10:22:18,923] com.pinterest.doctorkafka.util.MetricsPusher: Failed to send stats to OpenTSDB, will retry up to next interval ! java.net.ConnectException: Connection refused (Connection refused) ! at java.net.PlainSocketImpl.socketConnect(Native Method) ! at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ! at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ! at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ! at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ! at java.net.Socket.connect(Socket.java:589) ! at com.pinterest.doctorkafka.util.OpenTsdbClient.sendMetrics(OpenTsdbClient.java:107) ! ... 2 common frames omitted ! Causing: com.pinterest.doctorkafka.util.OpenTsdbClient$ConnectionFailedException: java.net.ConnectException: Connection refused (Connection refused) ! at com.pinterest.doctorkafka.util.OpenTsdbClient.sendMetrics(OpenTsdbClient.java:109) ! at com.pinterest.doctorkafka.util.MetricsPusher.sendMetrics(MetricsPusher.java:101) ! at com.pinterest.doctorkafka.util.MetricsPusher.run(MetricsPusher.java:129) ^C15:52:20.336 [Thread-2] ERROR com.pinterest.doctorkafka.DoctorKafkaMain - Failure in stopping operator java.lang.NullPointerException: null at com.pinterest.doctorkafka.DoctorKafka.stop(DoctorKafka.java:78) ~[doctorkafka-0.2.4.3-jar-with-dependencies.jar:?] at com.pinterest.doctorkafka.DoctorKafkaMain$OperatorCleanupThread.run(DoctorKafkaMain.java:153) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?] INFO [2019-01-30 10:22:20,342] org.eclipse.jetty.server.AbstractConnector: Stopped application@2bfaba70{HTTP/1.1,[http/1.1]}{0.0.0.0:8080} INFO [2019-01-30 10:22:20,345] org.eclipse.jetty.server.handler.ContextHandler: Stopped i.d.j.MutableServletContextHandler@2faa55bb{/,null,UNAVAILABLE}

yuyang08 commented 5 years ago

@venkatwilliams we run doctorkafka 0.2.4.3 in production, and haven't seen errors like this earlier. Could you tell more about your kafka cluster setting and doctorkafka configuration?

venkatwilliams commented 5 years ago

I am running single node kafka(1.1.0) cluster in my laptop. Doctorkafka configuration details are attached.

doctorkafka.properties.txt