search

inthefabric / Fabric

The collective mind awaits your input.
www.inthefabric.com
Other
5 stars 0 forks source link

Build Titan with Rexster Metrics #13

Closed zachkinstner closed 11 years ago

zachkinstner commented 11 years ago

As of the latest "metrics" branch commit of my Titan fork, I am able to build Titan 0.3.1 against the Tinkerpop 2.4.0-SNAPSHOT repositories.

First attempt resulted in an immediate "class not found" runtime error for com.yammer.metrics.Metrics. I figured out that I needed metrics-core 3.0.0-BETA. Added that to the titan-all POM.

Now, a new runtime error occurs:

13/05/16 17:59:24 ERROR service.CassandraDaemon: Exception encountered during startup
java.lang.NoClassDefFoundError: com/yammer/metrics/core/MetricName
        at org.apache.cassandra.metrics.DroppedMessageMetrics.<init>(DroppedMessageMetrics.java:43)
        at org.apache.cassandra.net.MessagingService.<init>(MessagingService.java:303)
        at org.apache.cassandra.net.MessagingService.<init>(MessagingService.java:67)
        at org.apache.cassandra.net.MessagingService$MSHandle.<clinit>(MessagingService.java:292)
        at org.apache.cassandra.net.MessagingService.instance(MessagingService.java:296)
        at org.apache.cassandra.service.StorageService.<init>(StorageService.java:232)
        at org.apache.cassandra.service.StorageService.<clinit>(StorageService.java:134)
        at org.apache.cassandra.locator.DynamicEndpointSnitch.<init>(DynamicEndpointSnitch.java:83)
        at org.apache.cassandra.locator.DynamicEndpointSnitch.<init>(DynamicEndpointSnitch.java:59)
        at org.apache.cassandra.config.DatabaseDescriptor.createEndpointSnitch(DatabaseDescriptor.java:522)
        at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescriptor.java:336)
        at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(DatabaseDescriptor.java:122)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:150)
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:366)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:409)
        at com.thinkaurelius.titan.diskstorage.cassandra.embedded.CassandraDaemonWrapper$CassandraStarter.run(CassandraDaemonWrapper.java:64)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.ClassNotFoundException: com.yammer.metrics.core.MetricName
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
        ... 22 more
Exception encountered during startup: com/yammer/metrics/core/MetricName

I suspect this is because of a dependency mismatch between Rexster's metrics-core (3.0.0-BETA) and cassandra-all's (2.0.3). I don't know how/if it's possible to resolve this.

zachkinstner commented 11 years ago

Titan metrics (not this specific issue, though) are being discussed at https://github.com/thinkaurelius/titan/issues/268.

@spmallette, is what I'm trying to do here (Titan 0.3.1 + Rexster 2.4.0-SNAPSHOT metrics) even feasible? From the referenced issue, it seems like Titan may use its own, non-Rexster metrics system in the future.

zachkinstner commented 11 years ago

Brief attempt at reducing Rexster's Metrics version to 2.0.3: https://github.com/zachkinstner/rexster/commit/39053dc791a4c9881efe7d9b421f5b7218d871dd. Things started to get ugly in the Graphite section, so I bailed.

I don't need metrics right now, but I'd like to have something in place. I can continue using the DataStax OpsCenter to watch metrics at the Cassandra level.

spmallette commented 11 years ago

hmm...didn't realize org.apache.cassandra:cassandra-all:jar:1.2.2 already had a dependency on metrics. i don't usually use betas, but in this case, i found it generally stable and there were some things i really wanted to have. what happens if you exclude: com.yammer.metrics:metrics-core:jar:2.0.3 in your pom for cassandra-all? will things run?

zachkinstner commented 11 years ago

Also, HBase 0.94.1 depends upon Metrics 2.1.2.

will things run?

In the runtime error above, Cassandra is looking for com.yammer.metrics.core.MetricName -- this exists in 2.0.3, but not in 3.0.0-BETA (doesn't have this file or namespace).

As far as I can tell, Eclipse automatically omits/excludes the older version of a dependency in case of collision. I think my builds are excluding Metrics 2.0.3 already (it's not present in the lib folder).

spmallette commented 11 years ago

uh...not sure. don't kow what eclipse does with dependency management. the best way to be sure is to rely on pom.xml exclusion. if cassandra is directly referencing that class, we maybe have a problem here. wish i'd known that when i started my work on rexster. :heart: :gun:

don't think i can go backward given the features i pushed out there....will have to sort this out with dan in https://github.com/thinkaurelius/titan/issues/268

zachkinstner commented 11 years ago

A bit of (very) good news for Fabric: when using local/Berkeley mode, builds from my Titan "metrics" branch run without the error above. This means I can run my integration tests against a TitanServer capable of performing load/save GraphSON tasks with Byte properties (Blueprints 2.4.0-SNAPSHOT feature).

Metrics aren't working yet (HTTP mode produces a blank result). Will investigate later, it's not vital at the moment...

zachkinstner commented 11 years ago

Metrics seem to be working sufficiently well now -- see notes in thinkaurelius/titan#268.

spmallette commented 11 years ago

out of curiosity...which reporters are you using?

zachkinstner commented 11 years ago

Graphite on my cluster database, and HTTP on my local database. Console seemed to be working fine, as well.

I'd note that the "excludes" reporter property seemed to be ignored. In HTTP, for example, those excluded namespaces would still be present in the JSON output.

Off topic: could Titan collect Cassandra metrics? I'm thinking Titan could somehow append those metrics to the ones Rexster provides. I'm not sure what Cassandra provides, but I could see it being useful when trying to determine which functionaliy layer is causing a particular slowdown/bottleneck. I can make a Titan feature issue for it, if this is interesting to you.

spmallette commented 11 years ago

that's odd about "excludes"...worked with me when testing graphite. i used a free graphite hosting service and they only allowed 100 metrics and rexster produces a lot more than that so i used excludes to cut it down. lemme check the http reporter with it

spmallette commented 11 years ago

I don't think the HTTP reporter allows for those settings (that was a limitation of codahale metrics itself). i will look into it more closely though and change the wiki docs accordingly if that is the case. excludes does work for the console. does it not work for graphite?

zachkinstner commented 11 years ago

My excludes use:

<excludes>http.rest.*</excludes>

I just deleted all my Graphite data (nothing valuable yet, no worries) to check this out. All of those rest key paths still show up in Graphite. I'm also using a prefix:

<prefix>fabric</prefix>

Do I need to append that prefix in my excludes, like fabric.http.rest.*?

zachkinstner commented 11 years ago

Note for self: to clear out Graphite, delete everything in the /opt/graphite/storage/whisper/ dir.

spmallette commented 11 years ago

i don't think you include the prefix from what i can remember. can you please confirm that the console reporter works for you with exclusions?

spmallette commented 11 years ago

btw, there is no way to pass includes/excludes to the metrics servlet as best i can tell:

https://github.com/codahale/metrics/blob/master/metrics-servlets/src/main/java/com/codahale/metrics/servlets/MetricsServlet.java

I've updated the wiki docs accordingly.