remote graphite host

[spycom]

Hi! I'm trying to connect my seyren instance with remote graphite host, which is far away and I don't have any access to it, except https://myhost.com. Right now mvn reports some fails with URL and hosts checks. Is there any way available to do so? Thank you.

[obazoud]

Hi,

Coud you give us more information like mvn reports, log, stacktrace ... ?

Thanks.

[spycom]

Sorry, here is my report file with failures:

---

Test set: com.seyren.core.util.config.SeyrenConfigTest

Tests run: 24, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 0.227 sec <<< FAILURE! defaultGraphiteUrlIsCorrect(com.seyren.core.util.config.SeyrenConfigTest) Time elapsed: 0.02 sec <<< FAILURE! java.lang.AssertionError: Expected: is "http://localhost:80" but: was "https://dash.myhost.com" at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8) at com.seyren.core.util.config.SeyrenConfigTest.defaultGraphiteUrlIsCorrect(SeyrenConfigTest.java:45) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74) defaultGraphiteSchemeIsCorrect(com.seyren.core.util.config.SeyrenConfigTest) Time elapsed: 0.002 sec <<< FAILURE! java.lang.AssertionError: Expected: is "http" but: was "https" at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8) at com.seyren.core.util.config.SeyrenConfigTest.defaultGraphiteSchemeIsCorrect(SeyrenConfigTest.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74) defaultGraphiteHostIsCorrect(com.seyren.core.util.config.SeyrenConfigTest) Time elapsed: 0.006 sec <<< FAILURE! java.lang.AssertionError: Expected: is "localhost:80" but: was "dash.myhost.com" at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20) at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8) at com.seyren.core.util.config.SeyrenConfigTest.defaultGraphiteHostIsCorrect(SeyrenConfigTest.java:65) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)

[scobal]

You can run your mvn command with -DskipTests

eg: mvn clean verify -DskipTests

The Seyren tests rely on the default set of environment variables unfortunately :)

[spycom]

Thank you so much! That really helped!) it might be great to mention it in installation guide. But now I have another question (sorry): I've got alerts when my values are greater than "warn" and "error" values, but what about reverse situation? I would like to be notified if the number of connections lower than my critical values.

Thank you so much for a great tool!

added a little bit later: I've figured out! no problems)

[scobal]

Cool :)

If you want the reverse situation you can enter an error value that is lower than the warn value

[scobal]

I'm going to close this. Feel free to re-open if you have any questions

[spycom]

hello again! after a week of good work I faced an issue: I've got some alerts, which didn't show at "check" page, so I restarted seyren and after that it reports that my graphite instance is not available! here is a piece of seyren.log:

28/12/2013 14:11:46.427 [seyren.check-scheduler-3] WARN c.s.c.s.c.GraphiteTargetChecker - Cloudsave CSV Cpu failed to read from Graphite com.seyren.core.util.graphite.GraphiteReadException: Failed to read from Graphite at com.seyren.core.util.graphite.GraphiteHttpClient.getTargetJson(GraphiteHttpClient.java:120) ~[seyren-core-1.0.0-SNAPSHOT.jar:na] at com.seyren.core.service.checker.GraphiteTargetChecker.check(GraphiteTargetChecker.java:50) ~[seyren-core-1.0.0-SNAPSHOT.jar:na] at com.seyren.core.service.schedule.CheckRunner.run(CheckRunner.java:65) [seyren-core-1.0.0-SNAPSHOT.jar:na] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_45] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_45] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_45] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) [na:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] Caused by: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null') at [Source: org.apache.http.conn.EofSensorInputStream@c7efee; line: 3, column: 2] at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1369) ~[jackson-core-2.2.3.jar:2.2.3] at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:532) ~[jackson-core-2.2.3.jar:2.2.3] at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:453) ~[jackson-core-2.2.3.jar:2.2.3] at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2278) ~[jackson-core-2.2.3.jar:2.2.3] at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:779) ~[jackson-core-2.2.3.jar:2.2.3] at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:665) ~[jackson-core-2.2.3.jar:2.2.3] at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:2926) ~[jackson-databind-2.2.3.jar:na] at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:2873) ~[jackson-databind-2.2.3.jar:na] at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:1719) ~[jackson-databind-2.2.3.jar:na] at com.seyren.core.service.checker.JsonNodeResponseHandler.handleResponse(JsonNodeResponseHandler.java:37) ~[seyren-core-1.0.0-SNAPSHOT.jar:na] at com.seyren.core.service.checker.JsonNodeResponseHandler.handleResponse(JsonNodeResponseHandler.java:28) ~[seyren-core-1.0.0-SNAPSHOT.jar:na] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:218) ~[httpclient-4.3.1.jar:4.3.1] at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:160) ~[httpclient-4.3.1.jar:4.3.1] at com.seyren.core.util.graphite.GraphiteHttpClient.getTargetJson(GraphiteHttpClient.java:118) ~[seyren-core-1.0.0-SNAPSHOT.jar:na] ... 9 common frames omitted

what should I check?

[spycom]

issue resolved after changing SEYREN_URL several times, not sure what was the root cause! but I still have alerts in "Recent alerts" which I can't delete. little confusing. and the last thing: is there any way to alert not only if my values are not in a range, but if it changes too much? F.e. my value can be +/- 10% of my average value for the last 15 minutes. It will be very useful to monitoring spikes and drops.

[scobal]

You may find the holtWinters functions in Graphite useful:

http://graphite.readthedocs.org/en/1.0/functions.html#graphite.render.functions.holtWintersAberration

John Allspaw talks about using them for anomaly detection from page 34 here:

http://qconlondon.com/dl/qcon-london-2012/slides/JohnAllspaw_FaultToleranceAnomalyDetectionAndAnticipationPatterns.pdf

[spycom]

thats cool, thank you, but I have no access to the production graphite server, except http://myhost:80 Seyren is good in grabbing all info through http, is there any way I can store it in my local graphite host?

[scobal]

That's a bit out of the scope of Seyren I'm afraid.

You could try using holtWinters in your Seyren target, which would allow Seyren to alert when there is a spike or drop. Something like:

holtWintersAberration(memory.free,10)

I'm no expert with holtWinters functions though so you might have to do some searching.

[spycom]

That's exactly what I need! Thank you so much! Very useful staff.

[spycom]

Seyren is really awesome) but here is another issue: I added some checks like holtWintersAberration(users.counter,10) or holtWintersAberration(users.value*,10) and usually it's ok, but fore some checks graphs for 1 day and 1 week are missed! link to them looks like: http://192.168.57.124:8080/api/checks/52c5528fb7607d1114fb6d6d/image/?&width=1200&height=350&from=-1440Minutes I receive response 200 OK and there is nothing. How can I investigate that issue?

[spycom]

there are a lot of messages in logs about checks that looks good, but there is nothing about checks which are bad.