search

riemann / riemann-dash

An HTTP dashboard for Riemann.
MIT License
248 stars 102 forks source link

"expired" events appearing from nowhere #104

Closed qnick closed 8 years ago

qnick commented 8 years ago

Hello. Here is the config for the riemann-server:

(streams
  (where (and
          (host #"^nkutselev05")
          (service #"^network.eth0.rx_packets"))
  #(info %)
  (index)
  ) ; where
  (expired #(info %))
)

Here is the riemann.log:

INFO [2015-10-19 21:42:59,293] defaultEventExecutorGroup-2-1 - riemann.config - #riemann.codec.Event{:host nkutselev05.staging.dw.sc.gwallet.com, :service network.eth0.rx_packets, :state ok, :description Network eth0 RX packets/sec, :metric 8.000018437745943, :tags [tensor linux], :time 1445316178, :ttl 120.0}
INFO [2015-10-19 21:43:29,295] defaultEventExecutorGroup-2-1 - riemann.config - #riemann.codec.Event{:host nkutselev05.staging.dw.sc.gwallet.com, :service network.eth0.rx_packets, :state ok, :description Network eth0 RX packets/sec, :metric 9.199983402917923, :tags [tensor linux], :time 1445316208, :ttl 120.0}
INFO [2015-10-19 21:43:59,294] defaultEventExecutorGroup-2-1 - riemann.config - #riemann.codec.Event{:host nkutselev05.staging.dw.sc.gwallet.com, :service network.eth0.rx_packets, :state ok, :description Network eth0 RX packets/sec, :metric 7.933317760392135, :tags [tensor linux], :time 1445316238, :ttl 120.0}
INFO [2015-10-19 21:44:29,294] defaultEventExecutorGroup-2-1 - riemann.config - #riemann.codec.Event{:host nkutselev05.staging.dw.sc.gwallet.com, :service network.eth0.rx_packets, :state ok, :description Network eth0 RX packets/sec, :metric 9.300014264605467, :tags [tensor linux], :time 1445316268, :ttl 120.0}
INFO [2015-10-19 21:44:59,293] defaultEventExecutorGroup-2-1 - riemann.config - #riemann.codec.Event{:host nkutselev05.staging.dw.sc.gwallet.com, :service network.eth0.rx_packets, :state ok, :description Network eth0 RX packets/sec, :metric 7.633263327392897, :tags [tensor linux], :time 1445316298, :ttl 120.0}
INFO [2015-10-19 21:45:29,294] defaultEventExecutorGroup-2-1 - riemann.config - #riemann.codec.Event{:host nkutselev05.staging.dw.sc.gwallet.com, :service network.eth0.rx_packets, :state ok, :description Network eth0 RX packets/sec, :metric 9.766674195666289, :tags [tensor linux], :time 1445316328, :ttl 120.0}

So everything looks fine. But look at the riemann-dash: image

WTF? These "expired" events are not supposed to be here.

aphyr commented 8 years ago

The dashboard does local expiry as well based on its best guess about what your Riemann clock is; if you've got events from nodes with varying clocks it might be seeing skew and prematurely expiring them.

qnick commented 8 years ago

Thank you for the answer. Fix this with syncing clocks.