search

alibaba / DataX

DataX是阿里云DataWorks数据集成的开源版本。
Other
15.73k stars 5.39k forks source link

不安装hadoop下使用hbase在进行写任务时报错，是必须安装hadoop吗？ #2096

Open voidlop opened 4 months ago

voidlop commented 4 months ago

我使用的hbase是2.5.8版本，用的1.1的write。 下面是报错：

WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/home/admi/Desktop/datax/datax/plugin/writer/hbase11xwriter/libs/hadoop-auth-2.5.1.jar) to method sun.security.krb5.Config.getInstance() WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 4月 11, 2024 2:04:42 下午 org.apache.hadoop.util.NativeCodeLoader 警告: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 4月 11, 2024 2:04:42 下午 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper 信息: Process identifier=hconnection-0x1ddc09dd connecting to ZooKeeper ensemble=localhost:2181 4月 11, 2024 2:04:42 下午 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper 信息: Process identifier=hconnection-0x5850041 connecting to ZooKeeper ensemble=localhost:2181 4月 11, 2024 2:04:42 下午 org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper 信息: Process identifier=hconnection-0x6b378ff4 connecting to ZooKeeper ensemble=localhost:2181 2024-04-10 23:04:42.968 [0-0-10-writer] INFO ZooKeeper - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2024-04-10 23:04:42.971 [0-0-10-writer] INFO ZooKeeper - Client environment:host.name=admi-virtual-machine 2024-04-10 23:04:42.973 [0-0-10-writer] INFO ZooKeeper - Client environment:java.version=11.0.22 2024-04-10 23:04:42.973 [0-0-10-writer] INFO ZooKeeper - Client environment:java.vendor=Ubuntu 2024-04-10 23:04:42.973 [0-0-10-writer] INFO ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-11-openjdk-amd64 2024-04-10 23:04:42.973 [0-0-10-writer] INFO ZooKeeper - Client environment:java.class.path=/home/admi/Desktop/datax/datax/lib/httpcore-4.4.13.jar:/home/admi/Desktop/datax/datax/lib/groovy-all-2.1.9.jar:/home/admi/Desktop/datax/datax/lib/httpclient-4.5.13.jar:/home/admi/Desktop/datax/datax/lib/commons-configuration-1.10.jar:/home/admi/Desktop/datax/datax/lib/commons-cli-1.2.jar:/home/admi/Desktop/datax/datax/lib/slf4j-api-1.7.10.jar:/home/admi/Desktop/datax/datax/lib/hamcrest-core-1.3.jar:/home/admi/Desktop/datax/datax/lib/commons-io-2.4.jar:/home/admi/Desktop/datax/datax/lib/commons-collections-3.2.1.jar:/home/admi/Desktop/datax/datax/lib/commons-lang-2.6.jar:/home/admi/Desktop/datax/datax/lib/datax-common-0.0.1-SNAPSHOT.jar:/home/admi/Desktop/datax/datax/lib/fastjson2-2.0.23.jar:/home/admi/Desktop/datax/datax/lib/logback-classic-1.0.13.jar:/home/admi/Desktop/datax/datax/lib/commons-logging-1.1.1.jar:/home/admi/Desktop/datax/datax/lib/datax-core-0.0.1-SNAPSHOT.jar:/home/admi/Desktop/datax/datax/lib/commons-math3-3.1.1.jar:/home/admi/Desktop/datax/datax/lib/commons-lang3-3.3.2.jar:/home/admi/Desktop/datax/datax/lib/datax-transformer-0.0.1-SNAPSHOT.jar:/home/admi/Desktop/datax/datax/lib/commons-codec-1.11.jar:/home/admi/Desktop/datax/datax/lib/commons-beanutils-1.9.2.jar:/home/admi/Desktop/datax/datax/lib/fluent-hc-4.5.jar:/home/admi/Desktop/datax/datax/lib/janino-2.5.16.jar:/home/admi/Desktop/datax/datax/lib/logback-core-1.0.13.jar:. 2024-04-10 23:04:42.976 [0-0-10-writer] INFO ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib 2024-04-10 23:04:42.977 [0-0-10-writer] INFO ZooKeeper - Client environment:java.io.tmpdir=/tmp 2024-04-10 23:04:42.977 [0-0-10-writer] INFO ZooKeeper - Client environment:java.compiler= 2024-04-10 23:04:42.977 [0-0-10-writer] INFO ZooKeeper - Client environment:os.name=Linux 2024-04-10 23:04:42.977 [0-0-10-writer] INFO ZooKeeper - Client environment:os.arch=amd64 2024-04-10 23:04:42.977 [0-0-10-writer] INFO ZooKeeper - Client environment:os.version=5.15.0-102-generic 2024-04-10 23:04:42.977 [0-0-10-writer] INFO ZooKeeper - Client environment:user.name=admi 2024-04-10 23:04:42.977 [0-0-10-writer] INFO ZooKeeper - Client environment:user.home=/home/admi 2024-04-10 23:04:42.977 [0-0-10-writer] INFO ZooKeeper - Client environment:user.dir=/home/admi/Desktop/datax/datax/bin 2024-04-10 23:04:42.978 [0-0-10-writer] INFO ZooKeeper - Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x6b378ff40x0, quorum=localhost:2181, baseZNode=/hbase 2024-04-10 23:04:42.978 [0-0-3-writer] INFO ZooKeeper - Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x58500410x0, quorum=localhost:2181, baseZNode=/hbase 2024-04-10 23:04:42.980 [0-0-13-writer] INFO ZooKeeper - Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=hconnection-0x1ddc09dd0x0, quorum=localhost:2181, baseZNode=/hbase 2024-04-10 23:04:43.015 [0-0-10-writer-SendThread(localhost:2181)] INFO ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2024-04-10 23:04:43.016 [0-0-3-writer-SendThread(localhost:2181)] INFO ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2024-04-10 23:04:43.016 [0-0-13-writer-SendThread(localhost:2181)] INFO ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) 2024-04-10 23:04:43.033 [0-0-3-writer-SendThread(localhost:2181)] INFO ClientCnxn - Socket connection established to localhost/127.0.0.1:2181, initiating session 2024-04-10 23:04:43.037 [0-0-10-writer-SendThread(localhost:2181)] INFO ClientCnxn - Socket connection established to localhost/127.0.0.1:2181, initiating session 2024-04-10 23:04:43.040 [0-0-13-writer-SendThread(localhost:2181)] INFO ClientCnxn - Socket connection established to localhost/127.0.0.1:2181, initiating session 2024-04-10 23:04:43.046 [0-0-3-writer-SendThread(localhost:2181)] INFO ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x10000094e90000f, negotiated timeout = 40000 2024-04-10 23:04:43.048 [0-0-10-writer-SendThread(localhost:2181)] INFO ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x10000094e900010, negotiated timeout = 40000 2024-04-10 23:04:43.052 [0-0-13-writer-SendThread(localhost:2181)] INFO ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x10000094e900011, negotiated timeout = 40000 2024-04-10 23:04:43.268 [0-0-3-writer] ERROR WriterRunner - Writer Runner Received Exceptions: com.alibaba.datax.common.exception.DataXException: Code:[Hbasewriter-02], Description:[获取Hbase连接时出错.]. - java.io.IOException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119) at com.alibaba.datax.plugin.writer.hbase11xwriter.Hbase11xHelper.getHbaseConnection(Hbase11xHelper.java:48) at com.alibaba.datax.plugin.writer.hbase11xwriter.Hbase11xHelper.getBufferedMutator(Hbase11xHelper.java:86) at com.alibaba.datax.plugin.writer.hbase11xwriter.HbaseAbstractTask.(HbaseAbstractTask.java:38) at com.alibaba.datax.plugin.writer.hbase11xwriter.NormalTask.(NormalTask.java:27) at com.alibaba.datax.plugin.writer.hbase11xwriter.Hbase11xWriter$Task.init(Hbase11xWriter.java:58) at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:44) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238) ... 9 more Caused by: java.lang.UnsupportedOperationException: Constructor threw an exception for org.apache.hadoop.hbase.ipc.RpcClientImpl at org.apache.hadoop.hbase.util.ReflectionUtils.instantiate(ReflectionUtils.java:54) at org.apache.hadoop.hbase.util.ReflectionUtils.instantiateWithCustomCtor(ReflectionUtils.java:34) at org.apache.hadoop.hbase.ipc.RpcClientFactory.createClient(RpcClientFactory.java:64) at org.apache.hadoop.hbase.ipc.RpcClientFactory.createClient(RpcClientFactory.java:48) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.(ConnectionManager.java:638) ... 14 more Caused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at org.apache.hadoop.hbase.util.ReflectionUtils.instantiate(ReflectionUtils.java:46) ... 18 more Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.hbase.util.ClassSize at org.apache.hadoop.hbase.ipc.IPCUtil.(IPCUtil.java:74) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.RpcClientImpl.(RpcClientImpl.java:1092) at org.apache.hadoop.hbase.ipc.RpcClientImpl.(RpcClientImpl.java:1118) ... 23 more

voidlop commented 4 months ago

已解决，我太蠢了乐

HuDaYe123 commented 3 months ago

11对2.0.X的版本兼容怎么做的

voidlop commented 3 months ago

我已经解决这个问题了，我没有看到2.0版本的hbase插件，是我试图用1.0版本的插件去读写2.0版本的数据库的问题。

Message ID: @.***>