search

gitava / Big-Data-Study

For studying big data
2 stars 0 forks source link

hbase zookeeper - ERROR [main] zookeeper.ZooKeeperWatcher: hconnection-0x568258ac0x0, quorum=hdp-node-01:2181,hdp-node-02:2181,hdp-node-03:2181, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid #26

Closed gitava closed 4 years ago

gitava commented 4 years ago 
2020-07-02 06:35:08,101 ERROR [main] zookeeper.ZooKeeperWatcher: hconnection-0x568258ac0x0, quorum=hdp-node-01:2181,hdp-node-02:2181,hdp-node-03:2181, baseZNode=/hbase Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1102)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:284)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:468)
    at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
    at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:105)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:944)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:720)
    at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:649)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
    at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
    at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:450)
    at org.jruby.javasupport.JavaMethod.invokeStaticDirect(JavaMethod.java:362)
    at org.jruby.java.invokers.StaticMethodInvoker.call(StaticMethodInvoker.java:58)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:312)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:169)
    at org.jruby.ast.CallOneArgNode.interpret(CallOneArgNode.java:57)
    at org.jruby.ast.InstAsgnNode.interpret(InstAsgnNode.java:95)
    at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104)
    at org.jruby.ast.BlockNode.interpret(BlockNode.java:71)
    at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74)
    at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:169)
    at org.jruby.internal.runtime.methods.DefaultMethod.call(DefaultMethod.java:191)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:302)
    at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:144)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:148)
    at org.jruby.RubyClass.newInstance(RubyClass.java:822)
    at org.jruby.RubyClass$i$newInstance.call(RubyClass$i$newInstance.gen:65535)
    at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroOrNBlock.call(JavaMethod.java:249)
    at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:292)
    at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:135)
    at home.vagrant.hbase.bin.hirb.__file__(/home/vagrant/hbase/bin/hirb.rb:142)
    at home.vagrant.hbase.bin.hirb.load(/home/vagrant/hbase/bin/hirb.rb)
    at org.jruby.Ruby.runScript(Ruby.java:697)
    at org.jruby.Ruby.runScript(Ruby.java:690)
    at org.jruby.Ruby.runNormally(Ruby.java:597)
    at org.jruby.Ruby.runFromMain(Ruby.java:446)
    at org.jruby.Main.doRunFromMain(Main.java:369)
    at org.jruby.Main.internalRun(Main.java:258)
    at org.jruby.Main.run(Main.java:224)
    at org.jruby.Main.run(Main.java:208)
    at org.jruby.Main.main(Main.java:188)
gitava commented 4 years ago

ref: https://bbs.csdn.net/topics/392008034?list=25656731

hbase1.1.3不需要覆盖hadoop的jar包，恢复即可

gitava commented 4 years ago 
[vagrant@hdp-node-01 ~/hbase/logs]$tail -300f hbase-vagrant-master-hdp-node-01.log

2020-07-02 09:52:37,425 INFO  [main-SendThread(hdp-node-02:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2020-07-02 09:52:39,325 INFO  [main-SendThread(hdp-node-03:2181)] zookeeper.ClientCnxn: Opening socket connection to server hdp-node-03/192.168.33.103:2181. Will not attempt to authenticate using SASL (unknown error)
2020-07-02 09:52:39,325 INFO  [main-SendThread(hdp-node-03:2181)] zookeeper.ClientCnxn: Socket connection established to hdp-node-03/192.168.33.103:2181, initiating session
2020-07-02 09:52:39,327 INFO  [main-SendThread(hdp-node-03:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2020-07-02 09:52:39,427 ERROR [main] zookeeper.RecoverableZooKeeper: ZooKeeper create failed after 4 attempts
2020-07-02 09:52:39,749 INFO  [main] zookeeper.ZooKeeper: Session: 0x0 closed
2020-07-02 09:52:39,750 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster.
    at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2901)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:234)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:138)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127)
    at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2911)
Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException: master:160000x0, quorum=hdp-node-01:2181,hdp-node-02:2181,hdp-node-03:2181, baseZNode=/hbase Unexpected KeeperException creating base node
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:217)
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:188)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:634)
    at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:462)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2894)
    ... 5 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:665)
    at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:644)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1261)
    at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1239)
    at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:204)
    ... 13 more
2020-07-02 09:52:39,754 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x0
gitava commented 4 years ago 
[vagrant@hdp-node-02 ~]$sudo jps
3058 Jps
2086 DataNode
2199 NodeManager
2751 HQuorumPeer
gitava commented 4 years ago 
[vagrant@hdp-node-01 ~/hbase/logs]$sudo jps
2195 NameNode
2788 NodeManager
2663 ResourceManager
4552 HQuorumPeer
2505 SecondaryNameNode
5018 Jps
2335 DataNode
2959 JobHistoryServe
gitava commented 4 years ago 
[vagrant@hdp-node-03 ~]$sudo jps
2750 -- process information unavailable
2085 DataNode
3072 Jps
2198 NodeManager
gitava commented 4 years ago

https://stackoverflow.com/questions/27263302/how-to-reset-hbase-and-zookeeper

As you said hard reset of Zookeeper is the only solution.
gitava commented 4 years ago 
    <property>
    <!-- 设置zookeeper：数据目录 -->
        <name>hbase.zookeeper.property.dataDir</name>
        <value>/home/vagrant/hbase/zookeeper</value>
    </property>
gitava commented 4 years ago

hbase-vagrant-master-hdp-node-01.out.4

2020-07-02 09:33:58,571 WARN  [main-SendThread(hdp-node-02:2181)] zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
    at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
gitava commented 4 years ago 
[vagrant@hdp-node-01 ~/hbase/logs]$hbase-daemon.sh stop zookeeper
running zookeeper, logging to /home/vagrant/hbase/logs/hbase-vagrant-zookeeper-hdp-node-01.out
stopping zookeeper.
[vagrant@hdp-node-01 ~/hbase/logs]$hbase-daemon.sh start zookeeper
running zookeeper, logging to /home/vagrant/hbase/logs/hbase-vagrant-zookeeper-hdp-node-01.out
[vagrant@hdp-node-01 ~/hbase/logs]$
[vagrant@hdp-node-01 ~/hbase/logs]$hbase-daemon.sh stop zookeeper
gitava commented 4 years ago

check ports...

[vagrant@hdp-node-01 ~]$netstat -lnp|grep 2181
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6       0      0 :::2181                 :::*                    LISTEN      6239/java
[vagrant@hdp-node-01 ~]$ps -ef|grep 6239
vagrant   6239  6225  0 10:22 pts/0    00:00:04 /home/vagrant/jdk/bin/java -Dproc_zookeeper -XX:OnOutOfMemoryError=kill -9 %p -XX:+UseConcMarkSweepGC -Dhbase.log.dir=/home/vagrant/hbase/logs -Dhbase.log.file=hbase-vagrant-zookeeper-hdp-node-01.log -Dhbase.home.dir=/home/vagrant/hbase -Dhbase.id.str=vagrant -Dhbase.root.logger=INFO,RFA -Djava.library.path=/home/vagrant/hadoop/lib/native -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.zookeeper.HQuorumPeer start
vagrant   6446  3981  0 10:39 pts/0    00:00:00 grep --color=auto 6239
gitava commented 4 years ago 
[vagrant@hdp-node-03 ~/hbase/zookeeper]$netstat -lnp|grep 2181
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6       0      0 :::2181                 :::*                    LISTEN      4126/java
[vagrant@hdp-node-03 ~/hbase/zookeeper]$ps -ef|grep 4126
vagrant   4126  4112  0 10:22 pts/0    00:00:03 /home/vagrant/jdk/bin/java -Dproc_zookeeper -XX:OnOutOfMemoryError=kill -9 %p -XX:+UseConcMarkSweepGC -Dhbase.log.dir=/home/vagrant/hbase/logs -Dhbase.log.file=hbase-vagrant-zookeeper-hdp-node-03.log -Dhbase.home.dir=/home/vagrant/hbase -Dhbase.id.str=vagrant -Dhbase.root.logger=INFO,RFA -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.zookeeper.HQuorumPeer start
vagrant   4303  3032  0 10:40 pts/0    00:00:00 grep --color=auto 4126
[vagrant@hdp-node-03 ~/hbase/zookeeper]$
gitava commented 4 years ago 
[vagrant@hdp-node-02 ~/hbase]$netstat -lnp|grep 2181
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6       0      0 :::2181                 :::*                    LISTEN      3601/java
[vagrant@hdp-node-02 ~/hbase]$ps -ef|grep 3601
vagrant   3601  3587  0 10:22 pts/0    00:00:03 /home/vagrant/jdk/bin/java -Dproc_zookeeper -XX:OnOutOfMemoryError=kill -9 %p -XX:+UseConcMarkSweepGC -Dhbase.log.dir=/home/vagrant/hbase/logs -Dhbase.log.file=hbase-vagrant-zookeeper-hdp-node-02.log -Dhbase.home.dir=/home/vagrant/hbase -Dhbase.id.str=vagrant -Dhbase.root.logger=INFO,RFA -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.zookeeper.HQuorumPeer start
vagrant   3744  3033  0 10:40 pts/0    00:00:00 grep --color=auto 3601
[vagrant@hdp-node-02 ~/hbase]$
gitava commented 4 years ago

using sudo to start the services

[vagrant@hdp-node-01 ~]$sudo hbase/bin/start-hbase.sh

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
hdp-node-03: running zookeeper, logging to /home/vagrant/hbase/bin/../logs/hbase-root-zookeeper-hdp-node-03.out
hdp-node-02: running zookeeper, logging to /home/vagrant/hbase/bin/../logs/hbase-root-zookeeper-hdp-node-02.out
hdp-node-01: running zookeeper, logging to /home/vagrant/hbase/bin/../logs/hbase-root-zookeeper-hdp-node-01.out
running master, logging to /home/vagrant/hbase/bin/../logs/hbase-root-master-hdp-node-01.out
hdp-node-03: running regionserver, logging to /home/vagrant/hbase/bin/../logs/hbase-root-regionserver-hdp-node-03.out
hdp-node-02: running regionserver, logging to /home/vagrant/hbase/bin/../logs/hbase-root-regionserver-hdp-node-02.out
hdp-node-01: running regionserver, logging to /home/vagrant/hbase/bin/../logs/hbase-root-regionserver-hdp-node-01.out
gitava commented 4 years ago

Tried but not working

https://ask.csdn.net/questions/250643 修改/etc/hosts，取消下边的注释

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
gitava commented 4 years ago

after solution of ticket #29, this one is also solved.

check #29 for more solution details.