search

addthis / metrics-reporter-config

Apache License 2.0
74 stars 52 forks source link

config3 support for Riemann #27

Closed snazy closed 8 years ago

snazy commented 8 years ago

Note: I've got no riemann setup in place - just ported the code and did a brief check.

yuesong commented 8 years ago

I don't have riemann either. It'd be great if someone who has can actually test this pull request...

KowalczykBartek commented 8 years ago

Hi, first of all thanks for this pull request @snazy !

I tested it, and maybe I miss-configured something - but I had to add protobuf-java 2.5.0

without it, java.lang.NoClassDefFoundError is thrown

Exception (java.lang.NoClassDefFoundError) encountered during startup: com/google/protobuf/MessageLite java.lang.NoClassDefFoundError: com/google/protobuf/MessageLite at com.aphyr.riemann.client.RiemannClient.tcp(RiemannClient.java:36) at com.codahale.metrics.riemann.Riemann.getClient(Riemann.java:43) at com.codahale.metrics.riemann.Riemann.<init>(Riemann.java:39) at com.codahale.metrics.riemann.Riemann.<init>(Riemann.java:35) at com.addthis.metrics3.reporter.config.RiemannReporterConfig.enable(RiemannReporterConfig.java:78) at com.addthis.metrics3.reporter.config.ReporterConfig.enableRiemann(ReporterConfig.java:193) at com.addthis.metrics3.reporter.config.ReporterConfig.enableAll(ReporterConfig.java:238) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:360) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:562) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:690) Caused by: java.lang.ClassNotFoundException: com.google.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 10 more ERROR 22:03:05 Exception encountered during startup java.lang.NoClassDefFoundError: com/google/protobuf/MessageLite at com.aphyr.riemann.client.RiemannClient.tcp(RiemannClient.java:36) ~[riemann-java-client-0.2.10.jar:na] at com.codahale.metrics.riemann.Riemann.getClient(Riemann.java:43) ~[riemann-java-client-0.2.10.jar:na] at com.codahale.metrics.riemann.Riemann.<init>(Riemann.java:39) ~[riemann-java-client-0.2.10.jar:na] at com.codahale.metrics.riemann.Riemann.<init>(Riemann.java:35) ~[riemann-java-client-0.2.10.jar:na] at com.addthis.metrics3.reporter.config.RiemannReporterConfig.enable(RiemannReporterConfig.java:78) ~[reporter-config3-3.0.2-SNAPSHOT.jar:3.0.2-SNAPSHOT] at com.addthis.metrics3.reporter.config.ReporterConfig.enableRiemann(ReporterConfig.java:193) ~[reporter-config3-3.0.2-SNAPSHOT.jar:3.0.2-SNAPSHOT] at com.addthis.metrics3.reporter.config.ReporterConfig.enableAll(ReporterConfig.java:238) ~[reporter-config3-3.0.2-SNAPSHOT.jar:3.0.2-SNAPSHOT] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:360) [main/:na] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:562) [main/:na] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:690) [main/:na] Caused by: java.lang.ClassNotFoundException: com.google.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_60] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_60] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) ~[na:1.8.0_60] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_60] ... 10 common frames omitted

With added this required 3 jars (riemann-java-client-0.2.10.jar , netty-3.6.1.Final.jar , protobuf-java-2.5.0.jar) I am able to push and receive metrics in my local riemann .

@yuesong is it necessary to test it on real Riemann ? or local one is sufficient ?

yuesong commented 8 years ago

I honestly don't know how different a "real Riemann" is from a "local one". @KowalczykBartek, do you use Riemann? Do you want to submit your change?

stevenmccord commented 8 years ago

@yuesong do you have what you did to get this to work with Riemann? I have a hosted Riemann instance, but I don't think that is the issue, I am getting similar errors to above, I am running a Cassandra 3.7 instance and it is complaining that it can't send to Riemann b/c the reporter isn't supported in 3, so trying to see if I can get this going with a manually compiled jar. Thanks!

yuesong commented 8 years ago

@stevenmccord I didn't do any work related to Riemann. The last update was that @KowalczykBartek tested some local change to make it work. If @KowalczykBartek submits his change, I'm happy to merge it such that people can test it more...

KowalczykBartek commented 8 years ago

Hi,

sorry for delay ;)

@stevenmccord it will work if you build metrics-reporter-config from this branch and add protobuf-java 2.5.0 - it is required, but information about that is missing in pull-request.

I will try to test it again tomorrow and push required changes - I was sure that someone did it already :)

P.S if you will have any problem with running this Riemann integration just let me know I will try help :)

stevenmccord commented 8 years ago

@KowalczykBartek awesome, I really appreciate it, I have been playing with the BlueMountain library that Riemann mentions as well, but it seems a little weird, as it requires me to remove things out of my cassandra.yaml file, which I am a little worried about.

https://github.com/BlueMountainCapital/riemann-cassandra/pull/2

I assume if I was able to get this to work then I think the BlueMountain library is essentially just tying into the same metric registry.... Have you had any experience with that?

I will work to include the protobu in the pom as well and give that a test and let you know.

KowalczykBartek commented 8 years ago

@stevenmccord you will only need to add protobuf-java 2.5.0 into lib/ directory in cassandra itself. And of course replace already included metrics-reporter-config jar with built by yourself :)

but it seems a little weird, as it requires me to remove things out of my cassandra.yaml file, which I am a little worried about.

what you had to remove ?

stevenmccord commented 8 years ago

@KowalczykBartek for that library, this is the error that I get:

org.apache.cassandra.exceptions.ConfigurationException: Invalid yaml. Please remove properties [tracetype_query_ttl, credentials_validity_in_ms, windows_timer_interval, enable_scripted_user_defined_functions, tracetype_repair_ttl, roles_validity_in_ms, disk_optimization_strategy, enable_user_defined_functions, concurrent_materialized_view_writes, transparent_data_encryption_options, hints_flush_period_in_ms, role_manager, max_hints_file_size_in_mb, batch_size_fail_threshold_in_kb] from your cassandra.yaml
    at org.apache.cassandra.config.YamlConfigurationLoader$MissingPropertiesChecker.check(YamlConfigurationLoader.java:162) ~[riemann-cassandra-0.0.3.jar:na]
    at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:115) ~[riemann-cassandra-0.0.3.jar:na]
    at org.apache.cassandra.config.YamlConfigurationLoader.loadConfig(YamlConfigurationLoader.java:84) ~[riemann-cassandra-0.0.3.jar:na]
    at org.apache.cassandra.config.DatabaseDescriptor.loadConfig(DatabaseDescriptor.java:161) ~[riemann-cassandra-0.0.3.jar:na]
    at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:136) ~[riemann-cassandra-0.0.3.jar:na]
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:168) [riemann-cassandra-0.0.3.jar:na]
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:564) [riemann-cassandra-0.0.3.jar:na]
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:653) [riemann-cassandra-0.0.3.jar:na]
Invalid yaml. Please remove properties [tracetype_query_ttl, credentials_validity_in_ms, windows_timer_interval, enable_scripted_user_defined_functions, tracetype_repair_ttl, roles_validity_in_ms, disk_optimization_strategy, enable_user_defined_functions, concurrent_materialized_view_writes, transparent_data_encryption_options, hints_flush_period_in_ms, role_manager, max_hints_file_size_in_mb, batch_size_fail_threshold_in_kb] from your cassandra.yaml
Fatal configuration error; unable to start. See log for stacktrace.

Once I remove those from the conifguration it does actually work and sends the events. Let me get your PR a shot as well and I will let you know though....

stevenmccord commented 8 years ago

@KowalczykBartek this is the specific error that I receive when I try to start my cassandra master node. 

INFO  22:33:48 Trying to load metrics-reporter-config from file: metrics_reporter_riemann.yaml
Exception (java.lang.NoClassDefFoundError) encountered during startup: com/addthis/metrics/reporter/config/AbstractReporterConfig
java.lang.NoClassDefFoundError: com/addthis/metrics/reporter/config/AbstractReporterConfig
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:357)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:585)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:714)
Caused by: java.lang.ClassNotFoundException: com.addthis.metrics.reporter.config.AbstractReporterConfig
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 15 more
ERROR 22:33:48 Exception encountered during startup
java.lang.NoClassDefFoundError: com/addthis/metrics/reporter/config/AbstractReporterConfig
    at java.lang.ClassLoader.defineClass1(Native Method) ~[na:1.8.0_102]
    at java.lang.ClassLoader.defineClass(ClassLoader.java:763) ~[na:1.8.0_102]
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[na:1.8.0_102]
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) ~[na:1.8.0_102]
    at java.net.URLClassLoader.access$100(URLClassLoader.java:73) ~[na:1.8.0_102]
    at java.net.URLClassLoader$1.run(URLClassLoader.java:368) ~[na:1.8.0_102]
    at java.net.URLClassLoader$1.run(URLClassLoader.java:362) ~[na:1.8.0_102]
    at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_102]
    at java.net.URLClassLoader.findClass(URLClassLoader.java:361) ~[na:1.8.0_102]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_102]
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) ~[na:1.8.0_102]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_102]
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:357) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:585) [apache-cassandra-3.7.jar:3.7]
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:714) [apache-cassandra-3.7.jar:3.7]
Caused by: java.lang.ClassNotFoundException: com.addthis.metrics.reporter.config.AbstractReporterConfig
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[na:1.8.0_102]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_102]
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) ~[na:1.8.0_102]
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[na:1.8.0_102]
    ... 15 common frames omitted

I have replaced the reporter-config jars with the SNAPSHOT that gets built with mvn package

Here is a list of the jars in the /usr/share/cassandra/lib directory

ST4-4.0.8.jar
airline-0.6.jar
antlr-runtime-3.5.2.jar
asm-5.0.4.jar
caffeine-2.2.6.jar
cassandra-driver-core-3.0.1-shaded.jar
cassandra-driver-internal-only-3.0.0-6af642d.zip
commons-cli-1.1.jar
commons-codec-1.2.jar
commons-lang3-3.1.jar
commons-math3-3.2.jar
compress-lzf-0.8.4.jar
concurrent-trees-2.4.0.jar
concurrentlinkedhashmap-lru-1.4.jar
disruptor-3.0.1.jar
ecj-4.4.2.jar
futures-2.1.6-py2.py3-none-any.zip
guava-18.0.jar
high-scale-lib-1.0.6.jar
hppc-0.5.4.jar
jackson-core-asl-1.9.2.jar
jackson-mapper-asl-1.9.2.jar
jamm-0.3.0.jar
javax.inject.jar
jbcrypt-0.3m.jar
jcl-over-slf4j-1.7.7.jar
jflex-1.6.0.jar
jna-4.0.0.jar
joda-time-2.4.jar
json-simple-1.1.jar
libthrift-0.9.2.jar
log4j-over-slf4j-1.7.7.jar
logback-classic-1.1.3.jar
logback-core-1.1.3.jar
lz4-1.3.0.jar
metrics-core-3.1.0.jar
metrics-logback-3.1.0.jar
netty-all-4.0.36.Final.jar
ohc-core-0.4.3.jar
ohc-core-j8-0.4.3.jar
primitive-1.0.jar
protobuf-java-2.5.0.jar
reporter-config-base-3.0.0.jar
reporter-config3-3.0.0.jar
sigar-1.6.4.jar
sigar-bin
six-1.7.3-py2.py3-none-any.zip
slf4j-api-1.7.7.jar
snakeyaml-1.11.jar
snappy-java-1.1.1.7.jar
snowball-stemmer-1.3.0.581.1.jar
stream-2.5.2.jar
thrift-server-0.3.7.jar
KowalczykBartek commented 8 years ago

@stevenmccord sorry for almost one day of delay but I guess that we have different time zones :D

Did you built metrics-reporter-config from @snazy 's riemann-config3-support branch ? I did it, replaced old report-config-base and report-config3 jars , and everything was ok.

I had same issue when I run cassandra with @snazy 's reporter-config3 jar but without replacing report-config-base .

stevenmccord commented 8 years ago

@KowalczykBartek no worries at all. Yeah I checkout this branch and then I did a mvn package in the directory and then pulled that out from:

reporter-config3/target/reporter-config3-3.0.2-SNAPSHOT.jar

Then I just move these into the lib directory and override the jar:

ADD files/reporter-config3-3.0.2-SNAPSHOT.jar /usr/share/cassandra/lib/reporter-config3-3.0.0.jar
ADD files/reporter-config3-3.0.2-SNAPSHOT.jar /usr/share/cassandra/lib/reporter-config-base-3.0.0.jar

Anything look odd there?

KowalczykBartek commented 8 years ago

Can you paste classpath that Cassandra loads ? It will be logger during Cassandra startup :)

P.S I guess that there is a typo, in 

ADD files/reporter-config3-3.0.2-SNAPSHOT.jar /usr/share/cassandra/lib/reporter-config3-3.0.0.jar
ADD files/reporter-config3-3.0.2-SNAPSHOT.jar /usr/share/cassandra/lib/reporter-config-base-3.0.0.jar

It should be 

ADD files/reporter-config3-3.0.2-SNAPSHOT.jar /usr/share/cassandra/lib/reporter-config3-3.0.0.jar
ADD files/reporter-config-base-3.0.2-SNAPSHOT-sources.jar    /usr/share/cassandra/lib/reporter-config-base-3.0.0.jar

You are interested in two jars, one from : reporter-config-base directory second : reporter-config2 directory

stevenmccord commented 8 years ago

Ah, I believe I missed the base jar.

I am sure this is documented somewhere, but do you know how to add a tag, or how the event looks when it gets sent to riemann?

It doesn't error so it starts up fine, just trying to see the metrics in the dashboard. Thanks again for all your help!

KowalczykBartek commented 8 years ago

Great :)

It is fragment from Riemann logs : 

#riemann.codec.Event{:host "XXXXX", :service "org.apache.cassandra.metrics.Table.CoordinatorReadLatency.akka.config mean_rate", :state nil, :description nil, :metric 0.0, :tags nil, :time 1475869962, :ttl nil, :metric-type "Timer"}
#riemann.codec.Event{:host "XXXXX", :service "org.apache.cassandra.metrics.Table.CoordinatorReadLatency.akka.eventsbytag1 max", :state nil, :description nil, :metric 0.0, :tags nil, :time 1475869962, :ttl nil, :metric-type "Timer"}
#riemann.codec.Event{:host "XXXXX", :service "org.apache.cassandra.metrics.Table.CoordinatorReadLatency.akka.eventsbytag1 mean", :state nil, :description nil, :metric NaN, :tags nil, :time 1475869962, :ttl nil, :metric-type "Timer"}
#riemann.codec.Event{:host "XXXXX", :service "org.apache.cassandra.metrics.Table.CoordinatorReadLatency.akka.eventsbytag1 min", :state nil, :description nil, :metric 0.0, :tags nil, :time 1475869962, :ttl nil, :metric-type "Timer"}
#riemann.codec.Event{:host "XXXXX", :service "org.apache.cassandra.metrics.Table.CoordinatorReadLatency.akka.eventsbytag1 stddev", :state nil, :description nil, :metric NaN, :tags nil, :time 1475869962, :ttl nil, :metric-type "Timer"}
#riemann.codec.Event{:host "XXXXX", :service "org.apache.cassandra.metrics.Table.CoordinatorReadLatency.akka.eventsbytag1 p50", :state nil, :description nil, :metric 0.0, :tags nil, :time 1475869962, :ttl nil, :metric-type "Timer"}
#riemann.codec.Event{:host "XXXXX", :service "org.apache.cassandra.metrics.Table.CoordinatorReadLatency.akka.eventsbytag1 p75", :state nil, :description nil, :metric 0.0, :tags nil, :time 1475869962, :ttl nil, :metric-type "Timer"}
#riemann.codec.Event{:host "XXXXX", :service "org.apache.cassandra.metrics.Table.CoordinatorReadLatency.akka.eventsbytag1 p95", :state nil, :description nil, :metric 0.0, :tags nil, :time 1475869962, :ttl nil, :metric-type "Timer"}

Are you using local Riemann for testing purposes ? If not, it is really convenient : http://riemann.io/quickstart.html

stevenmccord commented 8 years ago

Yeah I actually have it in a docker container, I don't think they are getting there, I just realized I see this in the log of the cassandra startup:

INFO  20:56:58 Trying to load metrics-reporter-config from file: metrics_reporter_riemann.yaml
ERROR 20:56:58 Tried to enable RiemannReporter, but class com.codahale.metrics.riemann.RiemannReporter was not found

So something still seems to be mis-configured on my side. I do have it running in a docker container, and I have other things connected to the riemann server so I have the Riemann server up. 

ADD files/reporter-config3-3.0.2-SNAPSHOT.jar \
    /usr/share/cassandra/lib/reporter-config3-3.0.0.jar
ADD files/reporter-config-base-3.0.2-SNAPSHOT.jar \
    /usr/share/cassandra/lib/reporter-config-base-3.0.0.jar
ADD files/protobuf-java-2.5.0.jar /usr/share/cassandra/lib/
KowalczykBartek commented 8 years ago

In your jars list : 

riemann-java-client-0.2.10.jar
netty-3.6.1.Final.jar

are missing :)

stevenmccord commented 8 years ago

And we are successful!! Thanks @KowalczykBartek really appreciate your time!

KowalczykBartek commented 8 years ago

Hi @yuesong - I created another pull-request https://github.com/addthis/metrics-reporter-config/pull/35 ( unfortunately I am not able to modify @snazy 's pull request ) - I resolved conflicts and modified README about missing com.google.protobuf dependency :)

hope this help. Cheers !