search

reiseburo / verspaetung

Verspätung is a small utility which aims to help identify delay of Kafka consumers
http://reiseburo.github.io/verspaetung/
MIT License
11 stars 2 forks source link

Verspaetung is leaking memory #18

Closed rtyler closed 9 years ago

rtyler commented 9 years ago

After a few days of operation:

17:07:01.394 [TreeCache-1] ERROR o.a.c.f.listen.ListenerContainer - Listener (com.github.lookout.verspaetung.zk.StandardTreeWatcher@58943431) threw an exception
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.Arrays.copyOf(Arrays.java:2219) ~[na:1.7.0_67]
        at java.util.concurrent.CopyOnWriteArrayList.add(CopyOnWriteArrayList.java:420) ~[na:1.7.0_67]
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.leftShift(DefaultGroovyMethods.java:11775) ~[verspaetung-0.1.5-all.jar:na]
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.leftShift(DefaultGroovyMethods.java:11792) ~[verspaetung-0.1.5-all.jar:na]
        at org.codehaus.groovy.runtime.dgm$377.invoke(Unknown Source) ~[verspaetung-0.1.5-all.jar:na]
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271) ~[verspaetung-0.1.5-all.jar:na]
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53) ~[verspaetung-0.1.5-all.jar:na]
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120) ~[verspaetung-0.1.5-all.jar:na]
        at com.github.lookout.verspaetung.zk.AbstractConsumerTreeWatcher.trackConsumerOffset(AbstractConsumerTreeWatcher.groovy:71) ~[verspaetung-0.1.5-all.jar:na]
        at com.github.lookout.verspaetung.zk.AbstractConsumerTreeWatcher$trackConsumerOffset$0.callCurrent(Unknown Source) ~[na:na]
        at com.github.lookout.verspaetung.zk.AbstractConsumerTreeWatcher.childEvent(AbstractConsumerTreeWatcher.groovy:55) ~[verspaetung-0.1.5-all.jar:na]
        at org.apache.curator.framework.recipes.cache.TreeCache$2.apply(TreeCache.java:685) ~[verspaetung-0.1.5-all.jar:na]
        at org.apache.curator.framework.recipes.cache.TreeCache$2.apply(TreeCache.java:679) ~[verspaetung-0.1.5-all.jar:na]
        at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) ~[verspaetung-0.1.5-all.jar:na]
        at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [verspaetung-0.1.5-all.jar:na]
        at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) [verspaetung-0.1.5-all.jar:na]
        at org.apache.curator.framework.recipes.cache.TreeCache.callListeners(TreeCache.java:678) [verspaetung-0.1.5-all.jar:na]
        at org.apache.curator.framework.recipes.cache.TreeCache.access$1400(TreeCache.java:69) [verspaetung-0.1.5-all.jar:na]
        at org.apache.curator.framework.recipes.cache.TreeCache$4.run(TreeCache.java:790) [verspaetung-0.1.5-all.jar:na]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_67]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_67]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_67]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_67]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_67]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_67]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
rtyler commented 9 years ago

This may get fixed by the work I'm doing in #17 since the structure of tracking latest and consumer offsets is pretty different.

Currently evaluating the build with #20 incorporated to see if it's leaking on my local machine

polynomial commented 9 years ago

omg, this almost made my entire day the worst day ever.

rtyler commented 9 years ago

I believe I've fixed this now with the dropwizard changes, will reopen if I see again