search

verisign / storm-graphite

An Apache Storm IMetricsConsumer that forwards Storm's built-in metrics to a Graphite server for real-time graphing, visualization, and operational dashboards.
Other
75 stars 38 forks source link

"Cannot get field 'bolt' because union is currently set to spout" exception in Storm UI #12

Closed ixalon closed 9 years ago

ixalon commented 9 years ago

Hi, we're running Storm 0.9.4 and we're trying to get storm-graphite running.

Both when using the shadowJar built from the 0.1.4 tag or when built from master (0.1.5-SNAPSHOT) we're seeing the following error in the Storm UI when visiting the page for any specific topology. We see the topology page successfully for a few moments after startup, but as soon as the "Prepared bolt __metricscom.verisign.storm.metrics.GraphiteMetricsConsumer:(3)" is logged by a worker, we start seeing the following in the UI:

java.lang.RuntimeException: Cannot get field 'bolt' because union is currently set to spout
    at backtype.storm.generated.ExecutorSpecificStats.get_bolt(ExecutorSpecificStats.java:245)
    at backtype.storm.ui.core$aggregate_bolt_stats$fn__7730.invoke(core.clj:173)
    at clojure.core$map$fn__4207.invoke(core.clj:2485)
    at clojure.lang.LazySeq.sval(LazySeq.java:42)
    at clojure.lang.LazySeq.seq(LazySeq.java:60)
    at clojure.lang.RT.seq(RT.java:484)
    at clojure.core$seq.invoke(core.clj:133)
    at clojure.core$map$fn__4207.invoke(core.clj:2479)
    at clojure.lang.LazySeq.sval(LazySeq.java:42)
    at clojure.lang.LazySeq.seq(LazySeq.java:60)
    at clojure.lang.Cons.next(Cons.java:39)
    at clojure.lang.RT.boundedLength(RT.java:1654)
    at clojure.lang.RestFn.applyTo(RestFn.java:130)
    at clojure.core$apply.invoke(core.clj:619)
    at backtype.storm.ui.core$aggregate_counts.invoke(core.clj:119)
    at backtype.storm.ui.core$aggregate_bolt_stats.invoke(core.clj:173)
    at backtype.storm.ui.core$compute_executor_capacity.invoke(core.clj:274)
    at clojure.core$map$fn__4207.invoke(core.clj:2485)
    at clojure.lang.LazySeq.sval(LazySeq.java:42)
    at clojure.lang.LazySeq.seq(LazySeq.java:60)
    at clojure.lang.RT.seq(RT.java:484)
    at clojure.core$seq.invoke(core.clj:133)
    at clojure.core$map$fn__4207.invoke(core.clj:2479)
    at clojure.lang.LazySeq.sval(LazySeq.java:42)
    at clojure.lang.LazySeq.seq(LazySeq.java:60)
    at clojure.lang.RT.seq(RT.java:484)
    at clojure.core$seq.invoke(core.clj:133)
    at clojure.core$apply.invoke(core.clj:617)
    at backtype.storm.ui.core$compute_bolt_capacity.invoke(core.clj:288)
    at backtype.storm.ui.core$visualization_data$iter__7843__7847$fn__7848.invoke(core.clj:399)
    at clojure.lang.LazySeq.sval(LazySeq.java:42)
    at clojure.lang.LazySeq.seq(LazySeq.java:60)
    at clojure.lang.Cons.next(Cons.java:39)
    at clojure.lang.RT.next(RT.java:598)
    at clojure.core$next.invoke(core.clj:64)
    at clojure.core$dorun.invoke(core.clj:2781)
    at clojure.core$doall.invoke(core.clj:2796)
    at backtype.storm.ui.core$visualization_data.invoke(core.clj:434)
    at backtype.storm.ui.core$topology_page.invoke(core.clj:642)
    at backtype.storm.ui.core$fn__8268.invoke(core.clj:853)
    at compojure.core$make_route$fn__6447.invoke(core.clj:93)
    at compojure.core$if_route$fn__6435.invoke(core.clj:39)
    at compojure.core$if_method$fn__6428.invoke(core.clj:24)
    at compojure.core$routing$fn__6453.invoke(core.clj:106)
    at clojure.core$some.invoke(core.clj:2443)
    at compojure.core$routing.doInvoke(core.clj:106)
    at clojure.lang.RestFn.applyTo(RestFn.java:139)
    at clojure.core$apply.invoke(core.clj:619)
    at compojure.core$routes$fn__6457.invoke(core.clj:111)
    at ring.middleware.reload$wrap_reload$fn__6482.invoke(reload.clj:14)
    at backtype.storm.ui.core$catch_errors$fn__8307.invoke(core.clj:909)
    at ring.middleware.keyword_params$wrap_keyword_params$fn__7124.invoke(keyword_params.clj:27)
    at ring.middleware.nested_params$wrap_nested_params$fn__7163.invoke(nested_params.clj:65)
    at ring.middleware.params$wrap_params$fn__7096.invoke(params.clj:55)
    at ring.middleware.multipart_params$wrap_multipart_params$fn__7191.invoke(multipart_params.clj:103)
    at ring.middleware.flash$wrap_flash$fn__7372.invoke(flash.clj:14)
    at ring.middleware.session$wrap_session$fn__7361.invoke(session.clj:43)
    at ring.middleware.cookies$wrap_cookies$fn__7292.invoke(cookies.clj:160)
    at ring.adapter.jetty$proxy_handler$fn__7572.invoke(jetty.clj:16)
    at ring.adapter.jetty.proxy$org.mortbay.jetty.handler.AbstractHandler$0.handle(Unknown Source)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    at org.mortbay.jetty.Server.handle(Server.java:326)
    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
    at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

As soon as we remove the metrics config from storm.yaml and restart, everything returns to normal.

There's no obvious error in the worker logs and metrics are being sent to graphite, so we're a bit flummoxed!

Many thanks, Chris

KevinJMao commented 9 years ago

Hi Chris,

Can you see if you observe the same behavior after registering only Storm's built-in LoggingMetricsConsumer?

Kevin

miguno commented 9 years ago

Closed because of inactivity.

imcom commented 9 years ago

I propose to re-open this issue since my problem is quite related.

I am using Storm 0.9.3 on production and after I added graphite-metrics to topology I got error:

Cannot get field 'spout' because union is currently set to bolt

ixalon commented 9 years ago

We never got to the bottom of this and ended up writing our own subclass of storm-metrics-reporter's StormMetricProcessor to send data to graphite in the form we require. Whilst not as feature-rich as storm-graphite, it allowed us to collect the metrics we needed.

qianlongzju commented 8 years ago

Hi, guys, my problem is exactly the same with @ixalon. There is no problem registering only Storm's built-in LoggingMetricsConsumer. And i had tried https://github.com/Aloomaio/storm-graphite-metrics before, which does not trigger this exception either.

qianlongzju commented 8 years ago

PS. I am running strom 0.9.4, and I build the storm-graphite jar using 0.2.3

lakshmanantokbox commented 8 years ago

I am getting same exception. I am running it on Storm 0.10.x