search

scobal / seyren

An alerting dashboard for Graphite
Other
859 stars 220 forks source link

Exception in seyren logs for null data points. #430

Open gargnisha1605 opened 7 years ago

gargnisha1605 commented 7 years ago

Following are the logs formed: Can anyone suggest how to resolve this?

I have used Meter.mark(); to push to graphite

[seyren.check-scheduler-2] WARN c.s.c.service.schedule.CheckRunner - No value present for Fail 09/02/2017 16:44:53.516 [seyren.check-scheduler-0] WARN c.s.c.s.c.GraphiteTargetChecker - {"target":"REQUEST_FAILED","datapoints":[[null,1486638240],[null,1486638300],[null,1486638360],[null,1486638420],[null,1486638480],[null,1486638540],[null,1486638600],[null,1486638660],[null,1486638720],[null,1486638780]]} [seyren.check-scheduler-0] WARN c.s.c.s.c.GraphiteTargetChecker - REQUEST FAILURE failed to read from Graphite com.seyren.core.exception.InvalidGraphiteValueException: Could not find a valid datapoint for target: "REQUEST_FAILED" at com.seyren.core.service.checker.GraphiteTargetChecker.getLatestValue(GraphiteTargetChecker.java:83) ~[seyren-core-1.0.1-SNAPSHOT.jar:na] at com.seyren.core.service.checker.GraphiteTargetChecker.check(GraphiteTargetChecker.java:54) ~[seyren-core-1.0.1-SNAPSHOT.jar:na] at com.seyren.core.service.schedule.CheckRunner.run(CheckRunner.java:65) ~[seyren-core-1.0.1-SNAPSHOT.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) ~[na:1.6.0_30] at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) ~[na:1.6.0_30] at java.util.concurrent.FutureTask.run(FutureTask.java:166) ~[na:1.6.0_30] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) ~[na:1.6.0_30] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) ~[na:1.6.0_30] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) ~[na:1.6.0_30] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.6.0_30] at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_30] [seyren.check-scheduler-0] WARN c.s.c.service.schedule.CheckRunner - No value present for REQUEST_FAILED

TyBrown commented 7 years ago

It looks like Seyren is unable to find the metric in your Graphite tree named "REQUEST_FAILED". Double check that you are entering the complete metric name in the "Target" field on the Check in the Seyren GUI.

Perhaps share with us a screenshot of your Seyren Check and an example Graph from the Graphite web application so we can see what you're working with?

Cheers!

gargnisha1605 commented 7 years ago

image Pls find the graphite screenshot, there is no datapoint against it in graphite.

And target for seyren : alias(sumSeries(derivative(prod...*.REQUEST_FAILED.count)),'REQUEST_FAILED')

TyBrown commented 7 years ago

If your'e not seeing any data points in the Graph in Graphite, it could mean a few things.

  1. Check that your smallest storage schema matches how frequently you are sending data to Graphite.
  2. Check that the metrics are actually arriving in Graphite.

You can try applying the function transformNull(metric, 0) to the metric to test and see if perhaps your data is arriving with gaps.

Basically though, you need to make sure that you have a graph that displays properly in Graphite web so that you can convert it to a Seyren check, otherwise exceptions like you posted will be what you get.

gargnisha1605 commented 7 years ago

My storage schemas : [carbon] pattern = ^carbon. retentions = 60:90d

[metric_1min_for_3m] pattern = .* retentions = 1m:45d,60m:90d

How can I check that metric is actually arriving in graphite?

Also I am getting some other exceptions as well, after I deleted the checks which were in unknown state in seyren alert dashboard. Please find the stack trace: 10/02/2017 12:48:22.137 [http-bio-8080-exec-13] WARN o.j.resteasy.core.ExceptionHandler - failed to execute javax.ws.rs.NotFoundException: Could not find resource for full path: http://30.0.9.122:8080/api/metrics/total at org.jboss.resteasy.core.registry.SegmentNode.match(SegmentNode.java:112) ~[resteasy-jaxrs-3.0.6.Final.jar:na] at org.jboss.resteasy.core.registry.RootNode.match(RootNode.java:43) ~[resteasy-jaxrs-3.0.6.Final.jar:na] at org.jboss.resteasy.core.registry.RootClassNode.match(RootClassNode.java:48) ~[resteasy-jaxrs-3.0.6.Final.jar:na] at org.jboss.resteasy.core.ResourceMethodRegistry.getResourceInvoker(ResourceMethodRegistry.java:444) ~[resteasy-jaxrs-3.0.6.Final.jar:na] at org.jboss.resteasy.core.SynchronousDispatcher.getInvoker(SynchronousDispatcher.java:234) ~[resteasy-jaxrs-3.0.6.Final.jar:na] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:171) ~[resteasy-jaxrs-3.0.6.Final.jar:na] at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220) ~[resteasy-jaxrs-3.0.6.Final.jar:na] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56) ~[resteasy-jaxrs-3.0.6.Final.jar:na] at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51) ~[resteasy-jaxrs-3.0.6.Final.jar:na] at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) ~[seyren-1.1.0.jar:na] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) ~[seyren-1.1.0.jar:na] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) ~[seyren-1.1.0.jar:na] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) ~[seyren-1.1.0.jar:na] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) ~[seyren-1.1.0.jar:na]

TyBrown commented 7 years ago

I'm not sure about that 2nd exception.

As for checking if the metrics are arriving... you can look at /opt/graphite/storage/log/carbon-cache/*/creates.log to see if the metrics got created. You should also be able to see the metric whisper files at /opt/graphite/storage/whisper, unless you specified a different path in your carbon.conf file.

gargnisha1605 commented 7 years ago

I have checked in the location and only m1_rate.wsp is present. There is no count.wsp file. But graphs are visible in Grafana. So how can it be possible.

Also after deleting those alerts, this problem must have resolved, right? But I am not getting alerts for other metrics as well even after WARN level has reached. Also if one metric is throwing exception, so alerts will not be generated for other metrics as well?

Also, until count.wsp of the metric gets created, I should not create an alert to avoid this Exception? Or is there any way, to stop these alerts from pushing to seyren, instead of deleting those checks and once again creating them.