Closed gitava closed 4 years ago
ref: https://bbs.csdn.net/topics/392008034?list=25656731
hbase1.1.3不需要覆盖hadoop的jar包,恢复即可
[vagrant@hdp-node-01 ~/hbase/logs]$tail -300f hbase-vagrant-master-hdp-node-01.log
2020-07-02 09:52:37,425 INFO [main-SendThread(hdp-node-02:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2020-07-02 09:52:39,325 INFO [main-SendThread(hdp-node-03:2181)] zookeeper.ClientCnxn: Opening socket connection to server hdp-node-03/192.168.33.103:2181. Will not attempt to authenticate using SASL (unknown error)
2020-07-02 09:52:39,325 INFO [main-SendThread(hdp-node-03:2181)] zookeeper.ClientCnxn: Socket connection established to hdp-node-03/192.168.33.103:2181, initiating session
2020-07-02 09:52:39,327 INFO [main-SendThread(hdp-node-03:2181)] zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2020-07-02 09:52:39,427 ERROR [main] zookeeper.RecoverableZooKeeper: ZooKeeper create failed after 4 attempts
2020-07-02 09:52:39,749 INFO [main] zookeeper.ZooKeeper: Session: 0x0 closed
2020-07-02 09:52:39,750 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster.
at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2901)
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:234)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:138)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2911)
Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException: master:160000x0, quorum=hdp-node-01:2181,hdp-node-02:2181,hdp-node-03:2181, baseZNode=/hbase Unexpected KeeperException creating base node
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:217)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:188)
at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:634)
at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:462)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2894)
... 5 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:665)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:644)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1261)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1239)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:204)
... 13 more
2020-07-02 09:52:39,754 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x0
[vagrant@hdp-node-02 ~]$sudo jps
3058 Jps
2086 DataNode
2199 NodeManager
2751 HQuorumPeer
[vagrant@hdp-node-01 ~/hbase/logs]$sudo jps
2195 NameNode
2788 NodeManager
2663 ResourceManager
4552 HQuorumPeer
2505 SecondaryNameNode
5018 Jps
2335 DataNode
2959 JobHistoryServe
[vagrant@hdp-node-03 ~]$sudo jps
2750 -- process information unavailable
2085 DataNode
3072 Jps
2198 NodeManager
https://stackoverflow.com/questions/27263302/how-to-reset-hbase-and-zookeeper
As you said hard reset of Zookeeper is the only solution.
<property>
<!-- 设置zookeeper:数据目录 -->
<name>hbase.zookeeper.property.dataDir</name>
<value>/home/vagrant/hbase/zookeeper</value>
</property>
hbase-vagrant-master-hdp-node-01.out.4
2020-07-02 09:33:58,571 WARN [main-SendThread(hdp-node-02:2181)] zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[vagrant@hdp-node-01 ~/hbase/logs]$hbase-daemon.sh stop zookeeper
running zookeeper, logging to /home/vagrant/hbase/logs/hbase-vagrant-zookeeper-hdp-node-01.out
stopping zookeeper.
[vagrant@hdp-node-01 ~/hbase/logs]$hbase-daemon.sh start zookeeper
running zookeeper, logging to /home/vagrant/hbase/logs/hbase-vagrant-zookeeper-hdp-node-01.out
[vagrant@hdp-node-01 ~/hbase/logs]$
[vagrant@hdp-node-01 ~/hbase/logs]$hbase-daemon.sh stop zookeeper
check ports...
[vagrant@hdp-node-01 ~]$netstat -lnp|grep 2181
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6 0 0 :::2181 :::* LISTEN 6239/java
[vagrant@hdp-node-01 ~]$ps -ef|grep 6239
vagrant 6239 6225 0 10:22 pts/0 00:00:04 /home/vagrant/jdk/bin/java -Dproc_zookeeper -XX:OnOutOfMemoryError=kill -9 %p -XX:+UseConcMarkSweepGC -Dhbase.log.dir=/home/vagrant/hbase/logs -Dhbase.log.file=hbase-vagrant-zookeeper-hdp-node-01.log -Dhbase.home.dir=/home/vagrant/hbase -Dhbase.id.str=vagrant -Dhbase.root.logger=INFO,RFA -Djava.library.path=/home/vagrant/hadoop/lib/native -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.zookeeper.HQuorumPeer start
vagrant 6446 3981 0 10:39 pts/0 00:00:00 grep --color=auto 6239
[vagrant@hdp-node-03 ~/hbase/zookeeper]$netstat -lnp|grep 2181
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6 0 0 :::2181 :::* LISTEN 4126/java
[vagrant@hdp-node-03 ~/hbase/zookeeper]$ps -ef|grep 4126
vagrant 4126 4112 0 10:22 pts/0 00:00:03 /home/vagrant/jdk/bin/java -Dproc_zookeeper -XX:OnOutOfMemoryError=kill -9 %p -XX:+UseConcMarkSweepGC -Dhbase.log.dir=/home/vagrant/hbase/logs -Dhbase.log.file=hbase-vagrant-zookeeper-hdp-node-03.log -Dhbase.home.dir=/home/vagrant/hbase -Dhbase.id.str=vagrant -Dhbase.root.logger=INFO,RFA -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.zookeeper.HQuorumPeer start
vagrant 4303 3032 0 10:40 pts/0 00:00:00 grep --color=auto 4126
[vagrant@hdp-node-03 ~/hbase/zookeeper]$
[vagrant@hdp-node-02 ~/hbase]$netstat -lnp|grep 2181
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6 0 0 :::2181 :::* LISTEN 3601/java
[vagrant@hdp-node-02 ~/hbase]$ps -ef|grep 3601
vagrant 3601 3587 0 10:22 pts/0 00:00:03 /home/vagrant/jdk/bin/java -Dproc_zookeeper -XX:OnOutOfMemoryError=kill -9 %p -XX:+UseConcMarkSweepGC -Dhbase.log.dir=/home/vagrant/hbase/logs -Dhbase.log.file=hbase-vagrant-zookeeper-hdp-node-02.log -Dhbase.home.dir=/home/vagrant/hbase -Dhbase.id.str=vagrant -Dhbase.root.logger=INFO,RFA -Dhbase.security.logger=INFO,RFAS org.apache.hadoop.hbase.zookeeper.HQuorumPeer start
vagrant 3744 3033 0 10:40 pts/0 00:00:00 grep --color=auto 3601
[vagrant@hdp-node-02 ~/hbase]$
using sudo to start the services
[vagrant@hdp-node-01 ~]$sudo hbase/bin/start-hbase.sh
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
hdp-node-03: running zookeeper, logging to /home/vagrant/hbase/bin/../logs/hbase-root-zookeeper-hdp-node-03.out
hdp-node-02: running zookeeper, logging to /home/vagrant/hbase/bin/../logs/hbase-root-zookeeper-hdp-node-02.out
hdp-node-01: running zookeeper, logging to /home/vagrant/hbase/bin/../logs/hbase-root-zookeeper-hdp-node-01.out
running master, logging to /home/vagrant/hbase/bin/../logs/hbase-root-master-hdp-node-01.out
hdp-node-03: running regionserver, logging to /home/vagrant/hbase/bin/../logs/hbase-root-regionserver-hdp-node-03.out
hdp-node-02: running regionserver, logging to /home/vagrant/hbase/bin/../logs/hbase-root-regionserver-hdp-node-02.out
hdp-node-01: running regionserver, logging to /home/vagrant/hbase/bin/../logs/hbase-root-regionserver-hdp-node-01.out
https://ask.csdn.net/questions/250643 修改/etc/hosts,取消下边的注释
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
after solution of ticket #29, this one is also solved.
check #29 for more solution details.