Hanging Using Storm Storage

[pkloves]

We're using Kafka 0.8 and Storm KafkaSpout, so I am running kafka-offset-monitor like:

java -cp ~/bin/KafkaOffsetMonitor-assembly-0.4.1-SNAPSHOT.jar com.quantifind.kafka.offsetapp.OffsetGetterWeb --offsetStorage storm --stormZKOffsetBase "/kafka" --zk <zk1>,<zk2>,<zk3> --port 9091 --refresh 15.seconds --retain 7.days --dbName ~/offsetapp >> $HOME/logs/KafkaOffsetMonitor.log

This worked perfectly until recently, when the second topic stops collecting:

serving resources from: jar:file:/home/svc_rts_admin/bin/KafkaOffsetMonitor-assembly-0.4.1-SNAPSHOT.jar!/offsetapp 2017-07-20 21:00:30 INFO Server:272 - jetty-8.y.z-SNAPSHOT 2017-07-20 21:00:30 INFO ZkEventThread:64 - Starting ZkClient event thread. 2017-07-20 21:00:30 INFO ZooKeeper:100 - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2017-07-20 21:00:30 INFO ZooKeeper:100 - Client environment:host.name=np1rtou101v.corp.halliburton.com 2017-07-20 21:00:30 INFO ZooKeeper:100 - Client environment:java.version=1.8.0_121 2017-07-20 21:00:30 INFO ZooKeeper:100 - Client environment:java.vendor=Oracle Corporation 2017-07-20 21:00:30 INFO ZooKeeper:100 - Client environment:java.home=/usr/java/jdk1.8.0_121/jre 2017-07-20 21:00:30 INFO ZooKeeper:100 - Client environment:java.class.path=/home/svc_rts_admin/bin/KafkaOffsetMonitor-assembly-0.4.1-SNAPSHOT.jar 2017-07-20 21:00:30 INFO ZooKeeper:100 - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2017-07-20 21:00:30 INFO ZooKeeper:100 - Client environment:java.io.tmpdir=/tmp 2017-07-20 21:00:30 INFO ZooKeeper:100 - Client environment:java.compiler=<NA> 2017-07-20 21:00:30 INFO ZooKeeper:100 - Client environment:os.name=Linux 2017-07-20 21:00:30 INFO ZooKeeper:100 - Client environment:os.arch=amd64 2017-07-20 21:00:30 INFO ZooKeeper:100 - Client environment:os.version=3.8.13-118.10.2.el6uek.x86_64 2017-07-20 21:00:30 INFO ZooKeeper:100 - Client environment:user.name=svc_rts_admin 2017-07-20 21:00:30 INFO ZooKeeper:100 - Client environment:user.home=/home/svc_rts_admin 2017-07-20 21:00:30 INFO ZooKeeper:100 - Client environment:user.dir=/home/svc_rts_admin 2017-07-20 21:00:30 INFO ZooKeeper:438 - Initiating client connection, connectString=<zk1>,<zk2>,<zk3> sessionTimeout=30000 watcher=org.I0Itec.zkclient.ZkClient@6146eb4c 2017-07-20 21:00:30 INFO AbstractConnector:338 - Started SocketConnector@0.0.0.0:9091 2017-07-20 21:00:30 INFO ZkClient:934 - Waiting for keeper state SyncConnected 2017-07-20 21:00:30 INFO ClientCnxn:975 - Opening socket connection to server <zk3>/<zk3-ip>:2181. Will not attempt to authenticate using SASL (unknown error) 2017-07-20 21:00:30 INFO ClientCnxn:852 - Socket connection established to <zk3>/<zk3-ip>:2181, initiating session 2017-07-20 21:00:30 INFO ClientCnxn:1235 - Session establishment complete on server <zk3>/<zk3-ip>:2181, sessionid = 0x35ca2bfc22f0303, negotiated timeout = 30000 2017-07-20 21:00:30 INFO ZkClient:711 - zookeeper state changed (SyncConnected) {"topology":{"id":"e85c767b-1f13-487f-81dd-13f5cd8e3285","name":"rts"},"offset":203401318,"partition":0,"broker":{"host":"<kafka_node4>","port":6667},"topic":"rts"} {"topology":{"id":"f8ea6724-1210-4912-981c-1f6bcbb3c13f","name":"rts"},"offset":2518449807,"partition":0,"broker":{"host":"<kafka-_node4>","port":6667},"topic":"rts_rt"} 2017-07-20 21:05:21 INFO ContextHandler:843 - stopped o.e.j.s.ServletContextHandler{/,jar:file:/home/svc_rts_admin/bin/KafkaOffsetMonitor-assembly-0.4.1-SNAPSHOT.jar!/offsetapp}

If I look at the sqlite DB, there's data for all 10 partitions for the first Kafka topic (rts) but nothing for the second one (rts_rt). Just a hunch, but this problem started when the offset for rts_rt rose above 2,000,000,000. Could this be a data type issue?

[bigcash]

run it error,org.apache.kafka.common.KafkaException: File conf/server-client-jaas.confcannot be read. so, where is the file "server-client-jaas.conf", i can't find it, thanks

[rcasey212]

@bigcash Your question truly has nothing to do with the above issue. Please open a new issue in the future. Per your issue, please see this already closed issue here: https://github.com/Morningstar/kafka-offset-monitor/issues/16

[bigcash]

@rcasey212 thank you