search

verisign / storm-graphite

An Apache Storm IMetricsConsumer that forwards Storm's built-in metrics to a Graphite server for real-time graphing, visualization, and operational dashboards.
Other
75 stars 38 forks source link

storm-nimbus failing after Graphite metics consumer configuration in storm.yml #27

Open ghost opened 8 years ago

ghost commented 8 years ago

Hi, I am having problems in starting nimbus when I configured the Graphite metrics consumer in storm.yml.

I configured storm.yml as follows.

topology.metrics.consumer.register:
  - class: "com.verisign.storm.metrics.GraphiteMetricsConsumer"
    parallelism.hint: 1
    argument:
      metrics.reporter.name: "com.verisign.storm.metrics.reporters.graphite.GraphiteReporter"
      metrics.graphite.host: "graphite.localhost"
      metrics.graphite.port: "2003"
      metrics.graphite.prefix: "storm.test.metrics"
      metrics.graphite.min-connect-attempt-interval-secs: "5"
      # Optional arguments can also be supplied to enable UDP
      metrics.graphite.protocol: "udp"

I have a storm cluster on a machine with CentOS 6 and a graphite server running locally via docker.

Can you point me out if I missed the configurations or else?

The log of failed nimbus is as follows. 

2016-03-26T10:52:15.836+0900 org.apache.storm.zookeeper.ClientCnxn [INFO] Session establishment complete on server 0.0.0.0/0.0.0.0:2181, sessioo
nid = 0x152eda1a4e8d176, negotiated timeout = 20000
2016-03-26T10:52:15.838+0900 o.a.s.c.framework.state.ConnectionStateManager [INFO] State change: CONNECTED
2016-03-26T10:52:15.839+0900 backtype.storm.zookeeper [INFO] Zookeeper state update: :connected:none
2016-03-26T10:52:16.859+0900 org.apache.storm.zookeeper.ZooKeeper [INFO] Session: 0x152eda1a4e8d176 closed
2016-03-26T10:52:16.859+0900 org.apache.storm.zookeeper.ClientCnxn [INFO] EventThread shut down
2016-03-26T10:52:16.861+0900 b.storm.utils.StormBoundedExponentialBackoffRetry [INFO] The baseSleepTimeMs [1000] the maxSleepTimeMs [30000] thee
 maxRetries [5]
2016-03-26T10:52:16.862+0900 o.a.s.curator.framework.imps.CuratorFrameworkImpl [INFO] Starting
2016-03-26T10:52:16.862+0900 org.apache.storm.zookeeper.ZooKeeper [INFO] Initiating client connection, connectString=0.0.0.0:2181/storm sessionn
Timeout=20000 watcher=org.apache.storm.curator.ConnectionState@a0e33db
2016-03-26T10:52:16.864+0900 org.apache.storm.zookeeper.ClientCnxn [INFO] Opening socket connection to server 0.0.0.0/0.0.0.0:2181. Will not att
tempt to authenticate using SASL (unknown error)
2016-03-26T10:52:16.864+0900 org.apache.storm.zookeeper.ClientCnxn [INFO] Socket connection established to 0.0.0.0/0.0.0.0:2181, initiating sess
sion
2016-03-26T10:52:16.868+0900 org.apache.storm.zookeeper.ClientCnxn [INFO] Session establishment complete on server 0.0.0.0/0.0.0.0:2181, sessioo
nid = 0x152eda1a4e8d177, negotiated timeout = 20000
2016-03-26T10:52:16.868+0900 o.a.s.c.framework.state.ConnectionStateManager [INFO] State change: CONNECTED
2016-03-26T10:52:16.923+0900 backtype.storm.daemon.nimbus [INFO] Starting Nimbus server...
2016-03-26T10:52:17.752+0900 backtype.storm.daemon.nimbus [ERROR] Error when processing event
java.lang.NullPointerException: null
        at clojure.lang.Numbers.ops(Numbers.java:942) ~[clojure-1.5.1.jar:na]
        at clojure.lang.Numbers.isZero(Numbers.java:90) ~[clojure-1.5.1.jar:na]

2016-03-26T10:52:15.838+0900 o.a.s.c.framework.state.ConnectionStateManager [INFO] State change: CONNECTED
2016-03-26T10:52:15.839+0900 backtype.storm.zookeeper [INFO] Zookeeper state update: :connected:none
2016-03-26T10:52:16.859+0900 org.apache.storm.zookeeper.ZooKeeper [INFO] Session: 0x152eda1a4e8d176 closed
2016-03-26T10:52:16.859+0900 org.apache.storm.zookeeper.ClientCnxn [INFO] EventThread shut down
2016-03-26T10:52:16.862+0900 o.a.s.curator.framework.imps.CuratorFrameworkImpl [INFO] Starting
2016-03-26T10:52:16.868+0900 o.a.s.c.framework.state.ConnectionStateManager [INFO] State change: CONNECTED
2016-03-26T10:52:16.923+0900 backtype.storm.daemon.nimbus [INFO] Starting Nimbus server...
2016-03-26T10:52:17.752+0900 backtype.storm.daemon.nimbus [ERROR] Error when processing event
java.lang.NullPointerException: null
        at clojure.lang.Numbers.ops(Numbers.java:942) ~[clojure-1.5.1.jar:na]
        at clojure.lang.Numbers.isZero(Numbers.java:90) ~[clojure-1.5.1.jar:na]
        at backtype.storm.util$partition_fixed.invoke(util.clj:830) ~[storm-core-0.9.4.jar:0.9.4]
        at clojure.lang.AFn.applyToHelper(AFn.java:163) [clojure-1.5.1.jar:na]
        at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na]
        at clojure.core$apply.invoke(core.clj:617) ~[clojure-1.5.1.jar:na]
        at clojure.lang.AFn.applyToHelper(AFn.java:163) [clojure-1.5.1.jar:na]
        at clojure.lang.RestFn.applyTo(RestFn.java:132) ~[clojure-1.5.1.jar:na]
        at clojure.core$apply.invoke(core.clj:619) ~[clojure-1.5.1.jar:na]
        at clojure.core$partial$fn__445.doInvoke(core.clj:2396) ~[na:na]
        at clojure.lang.RestFn.invoke(RestFn.java:408) ~[clojure-1.5.1.jar:na]
        at backtype.storm.util$map_val$iter__274__278$fn__279.invoke(util.clj:294) ~[storm-core-0.9.4.jar:0.9.4]
        at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.5.1.jar:na]
        at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.5.1.jar:na]
        at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30) ~[clojure-1.5.1.jar:na]
        at clojure.core.protocols$eval2802$fn__2803.invoke(protocols.clj:54) ~[na:na]
        at clojure.core.protocols$eval2735$fn__2736$G__2726__2749.invoke(protocols.clj:13) ~[na:na]
        at clojure.core$reduce.invoke(core.clj:6177) ~[clojure-1.5.1.jar:na]
        at clojure.core$into.invoke(core.clj:6229) ~[clojure-1.5.1.jar:na]
        at backtype.storm.util$map_val.invoke(util.clj:293) ~[storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.daemon.nimbus$compute_executors.invoke(nimbus.clj:422) ~[storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.daemon.nimbus$compute_executor__GT_component.invoke(nimbus.clj:433) ~[storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.daemon.nimbus$read_topology_details.invoke(nimbus.clj:326) ~[storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.daemon.nimbus$mk_assignments$iter__4109__4113$fn__4114.invoke(nimbus.clj:651) ~[storm-core-0.9.4.jar:0.9.4]
        at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.5.1.jar:na]
        at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.5.1.jar:na]
        at clojure.lang.RT.seq(RT.java:484) ~[clojure-1.5.1.jar:na]
        at clojure.core$seq.invoke(core.clj:133) ~[clojure-1.5.1.jar:na]
        at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30) ~[clojure-1.5.1.jar:na]
        at clojure.core.protocols$eval2802$fn__2803.invoke(protocols.clj:54) ~[na:na]
        at clojure.core.protocols$eval2735$fn__2736$G__2726__2749.invoke(protocols.clj:13) ~[na:na]
        at clojure.core$reduce.invoke(core.clj:6177) ~[clojure-1.5.1.jar:na]
        at clojure.core$into.invoke(core.clj:6229) ~[clojure-1.5.1.jar:na]
        at backtype.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:650) ~[storm-core-0.9.4.jar:0.9.4]
        at clojure.lang.RestFn.invoke(RestFn.java:410) ~[clojure-1.5.1.jar:na]
        at backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto____4291$fn__4296$fn__4297.invoke(nimbus.clj:909) ~[storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.daemon.nimbus$fn__4290$exec_fn__1754__auto____4291$fn__4296.invoke(nimbus.clj:908) ~[storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.timer$schedule_recurring$this__1538.invoke(timer.clj:99) ~[storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.timer$mk_timer$fn__1521$fn__1522.invoke(timer.clj:50) ~[storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.timer$mk_timer$fn__1521.invoke(timer.clj:42) [storm-core-0.9.4.jar:0.9.4]
        at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
2016-03-26T10:52:17.754+0900 backtype.storm.util [ERROR] Halting process: ("Error when processing an event")
java.lang.RuntimeException: ("Error when processing an event")
        at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) [storm-core-0.9.4.jar:0.9.4]
        at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.5.1.jar:na]
        at backtype.storm.daemon.nimbus$nimbus_data$fn__3716.invoke(nimbus.clj:76) [storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.timer$mk_timer$fn__1521$fn__1522.invoke(timer.clj:68) [storm-core-0.9.4.jar:0.9.4]
        at backtype.storm.timer$mk_timer$fn__1521.invoke(timer.clj:42) [storm-core-0.9.4.jar:0.9.4]
        at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
2016-03-26T10:52:17.755+0900 backtype.storm.daemon.nimbus [INFO] Shutting down master
ghost commented 8 years ago

Anyone?

Crim commented 8 years ago

Also randomly started seeing this. You have any luck sorting it out?

Crim commented 8 years ago

I ended up removing the nimbus subdirectory under the configured "storm.local.dir" and nimbus fired right up. 👍