search

googleapis / java-bigtable-hbase

Java libraries and HBase client extensions for accessing Google Cloud Bigtable
https://cloud.google.com/bigtable/
Apache License 2.0
174 stars 179 forks source link

Using bigtable-client with sqoop #1812

Closed szewi closed 6 years ago

szewi commented 6 years ago

I am working on using sqoop with BigTable. On configured a dataproc cluster with sqoop 1.4.7 and installed hbase 1.3.2, I added bigtable-hbase-1.x-hadoop-1.3.0.jar to hbase/lib and modified hbase-site.xml as described in cloud-bigtable-examples.

As a result:

Does anyone make it working with sqoop or there is no way to make it working? I tried several version of bigtable-client with no luck. Import to locally installed HBase works so it's not a problem with sqoop and rather bigtable-client issue.

Full error message:

18/05/07 14:31:04 ERROR tool.ImportTool: Import failed: java.io.IOException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240) at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:432) at org.apache.hadoop.hbase.client.ConnectionManager.createConnection(ConnectionManager.java:425) at org.apache.hadoop.hbase.client.ConnectionManager.getConnectionInternal(ConnectionManager.java:303) at org.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.java:238) at org.apache.sqoop.mapreduce.HBaseImportJob.jobSetup(HBaseImportJob.java:163) at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:268) at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:692) at org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:127) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:520) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238) ... 16 more Caused by: java.lang.IllegalArgumentException: Bigtable does not support managed connections. at org.apache.hadoop.hbase.client.AbstractBigtableConnection.(AbstractBigtableConnection.java:130) at com.google.cloud.bigtable.hbase1_x.BigtableConnection.(BigtableConnection.java:54) ... 21 more

sduskis commented 6 years ago

At first glance, it looks like sqoop's code needs to change. The stacktrace shows the use of the HBaseAdmin class, which is only usable with HBase. HBase 1.0 introduced new interfaces that allowed Cloud Bigtable to extend the HBase interfaces. I'm guessing that sqoop will have a few of changes.

Here's the list of differences.

sduskis commented 6 years ago

I'm going to close this bug, since it sounds like sqoop is not yet up-to-date with the idioms introduced in HBase 1.0, which is about 3 years ago. It seems like the Cloud Bigtable client is working as intended. I'm going to close this issue. If you would like, feel free to open a bug on https://github.com/GoogleCloudPlatform/cloud-bigtable-examples which is more oriented towards samples of client use and samples of integrations.