search

salesforce / storm-dynamic-spout

A framework for building spouts for Apache Storm and a Kafka based spout for dynamically skipping messages to be processed later.
BSD 3-Clause "New" or "Revised" License
40 stars 13 forks source link

Null Pointer Exceptions in SidelineSpoutHandler and Consumer #83

Closed daniel-dara closed 6 years ago

daniel-dara commented 6 years ago

My team recently started and stopped some sidelines and then found these Null Pointer Exceptions in our logs.

2017-11-29 16:42:31.793 [Curator-PathChildrenCache-0] o.a.c.f.r.c.PathChildrenCache [ERROR]  
java.lang.NullPointerException: null
    at com.salesforce.storm.spout.sideline.handler.SidelineSpoutHandler.stopSidelining(SidelineSpoutHandler.java:305) ~[stormjar.jar:?]
    at com.salesforce.storm.spout.sideline.SpoutTriggerProxy.stopSidelining(SpoutTriggerProxy.java:65) ~[stormjar.jar:?]
[redacted]
[redacted]
[redacted]
    at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:522) [stormjar.jar:?]
    at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:516) [stormjar.jar:?]
    at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93) [stormjar.jar:?]
    at org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) [stormjar.jar:?]
    at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85) [stormjar.jar:?]
    at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:514) [stormjar.jar:?]
    at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) [stormjar.jar:?]
    at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:773) [stormjar.jar:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_102]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_102]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_102]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_102]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_102]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_102]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102],

and

2017-11-29 15:12:04.880 [Thread-498-EventStreamSpout-executor[98 98]] o.a.s.d.executor [ERROR]  
java.lang.NullPointerException: null
    at com.salesforce.storm.spout.dynamic.kafka.Consumer.getKafkaConsumer(Consumer.java:155) ~[stormjar.jar:?]
    at com.salesforce.storm.spout.dynamic.kafka.Consumer.metrics(Consumer.java:656) ~[stormjar.jar:?]
    at com.salesforce.storm.spout.dynamic.kafka.Consumer.getMaxLag(Consumer.java:663) ~[stormjar.jar:?]
    at com.salesforce.storm.spout.dynamic.VirtualSpout.getMaxLag(VirtualSpout.java:572) ~[stormjar.jar:?]
    at com.salesforce.storm.spout.dynamic.coordinator.SpoutPartitionProgressMonitor.reportStatus(SpoutPartitionProgressMonitor.java:75) ~[stormjar.jar:?]
    at com.salesforce.storm.spout.dynamic.coordinator.SpoutMonitor.reportStatus(SpoutMonitor.java:386) ~[stormjar.jar:?]
    at com.salesforce.storm.spout.dynamic.coordinator.SpoutMonitor.run(SpoutMonitor.java:239) ~[stormjar.jar:?]
    at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626) ~[?:1.8.0_102]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102],

I don't know anything about them but I imagine they could either be safely handled or a more specific/friendly exception thrown in their stead.

stanlemon commented 6 years ago

@danielsd Just FYI. I redacted some class path entries there that were specific to your deployment.

I think for some reason the sideline was missing a starting offset, as that seems to be the first exceptions error: https://github.com/salesforce/storm-dynamic-spout/blob/0.9.0/src/main/java/com/salesforce/storm/spout/sideline/handler/SidelineSpoutHandler.java#L305

This is fixed in 0.10 currently, but that release isn't cut yet.

@Crim we'll have to figure out if we want to patch this in 0.9 or not and how exactly to orchestrate a release for it if we do.

Crim commented 6 years ago

If we want to patch 0.9.x we can branch from the release tag a branch called "0.9.x" or some variation, back port the fix, bump the version to 0.9.1 and cut a release

stanlemon commented 6 years ago

In #86 I added checks and log statements. This is most likely related to a serialization issue though.