search

junegunn / hbase-jruby

A JRuby binding for HBase
MIT License
38 stars 14 forks source link

AggregationClient not available after 0.95 #33

Closed graphex closed 10 years ago

graphex commented 10 years ago

I can't seem to do any aggregation operations against hbase 0.95 or above. resolve_dependency! gives this output (similar for 0.96 0.95 as well):

jruby-1.7.12 :003 > HBase.resolve_dependency! 'cdh5.0.1', :verbose => true
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building hbase-jruby 0.4.7
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:2.5.1:resolve (default-cli) @ hbase-project ---
[INFO]
[INFO] --- maven-dependency-plugin:2.5.1:build-classpath (default-cli) @ hbase-project ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.304s
[INFO] Finished at: Mon Jun 02 15:38:10 MDT 2014
[INFO] Final Memory: 10M/81M
[INFO] ------------------------------------------------------------------------
14/06/02 15:38:14 INFO Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
Java classes not found: org.apache.hadoop.hbase.io.hfile.Compression, org.apache.hadoop.hbase.client.coprocessor.AggregationClient

Attempting to call aggregation methods results in this error:

jruby-1.7.12 :002 > table.project("n:mynum").aggregate(:sum)
NameError: uninitialized constant HBase::Scoped::Aggregation::AggregationClient
    ...hbase-jruby-92b1e269549e/lib/hbase-jruby/scoped/aggregation.rb:50:in `aggregation_impl'
    ...hbase-jruby-92b1e269549e/lib/hbase-jruby/scoped/aggregation.rb:43:in `aggregate'
...

It looks like the protocol for communicating with coprocessors changed at 0.95, but I haven't yet found any documentation on how it was restructured.

junegunn commented 10 years ago

Because of this issue and the comments there

I'd rather have us compiling on bigtop ci than have this class. Will remove it and turn it into an example in a day or so unless we come up w/ a resolution.

I committed the patch to 0.95, the one that removes all to do w/ AggregationClient.

I had the impression that AggregationClient was permanently removed from HBase code base. And it really looks like that CDH5 (which is based on HBase 0.96.1) does not include it. However, I realized the class reappeared in the recent versions.

jruby-1.7.12 :002 > HBase.resolve_dependency! '0.98'
jruby-1.7.12 :003 > org.apache.hadoop.hbase.client.coprocessor.AggregationClient
 => Java::OrgApacheHadoopHbaseClientCoprocessor::AggregationClient

I haven't had time to test AggregationClient on 0.98 or above.

graphex commented 10 years ago

I think after 0.95 the method takes a string instead of an HTable instance. Using the 0.98 dependency I get:

jruby-1.7.12 :002 > table.project("n:mynum").aggregate(:sum)
NameError: no method 'sum' for arguments (org.jruby.java.proxies.ArrayJavaProxy,org.apache.hadoop.hbase.client.coprocessor.LongColumnInterpreter,org.apache.hadoop.hbase.client.Scan) on Java::OrgApacheHadoopHbaseClientCoprocessor::AggregationClient
  available overloads:
    (org.apache.hadoop.hbase.TableName,org.apache.hadoop.hbase.coprocessor.ColumnInterpreter,org.apache.hadoop.hbase.client.Scan)
    (org.apache.hadoop.hbase.client.HTable,org.apache.hadoop.hbase.coprocessor.ColumnInterpreter,org.apache.hadoop.hbase.client.Scan)

Maybe related to the switch to protobuf coprocessors? https://issues.apache.org/jira/browse/HBASE-6785

junegunn commented 10 years ago

Hmm, obviously the method signature has changed.

The first argument used to be a byte array denoting the name of the table, but now the method expects HTable instance instead. The tricky thing is that hbase-jruby should support both versions of the API. I'll see what I can do.

junegunn commented 10 years ago

Just released 0.5.1 with the above commit. Let me know if you run into any problem. Thanks.