hbase - cannot get log writer

gitava commented 4 years ago

hbase-vagrant-regionserver-hdp-node-02.log

2020-07-03 02:46:01,334 ERROR [RS_OPEN_META-hdp-node-02:16020-0] handler.OpenRegionHandler: Failed open of region=hbase:meta,,1.1588230740, starting to roll back the global memstore size.
java.io.IOException: cannot get log writer
    at org.apache.hadoop.hbase.wal.DefaultWALProvider.createWriter(DefaultWALProvider.java:395)
    at org.apache.hadoop.hbase.regionserver.wal.FSHLog.createWriterInstance(FSHLog.java:776)
    at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:736)
    at org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:643)
    at org.apache.hadoop.hbase.regionserver.wal.FSHLog.<init>(FSHLog.java:572)
    at org.apache.hadoop.hbase.wal.DefaultWALProvider.getWAL(DefaultWALProvider.java:138)
    at org.apache.hadoop.hbase.wal.WALFactory.getMetaWAL(WALFactory.java:261)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:2031)
    at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:364)
    at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:131)
    at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException: hflush
    at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.init(ProtobufLogWriter.java:95)
    at org.apache.hadoop.hbase.wal.DefaultWALProvider.createWriter(DefaultWALProvider.java:384)
    ... 13 more

gitava commented 4 years ago

https://blog.csdn.net/Sword1996/article/details/93526882

hbase.unsafe.stream.capability.enforce：使用本地文件系统设置为false，使用hdfs设置为true。但根据HBase 官方手册的说明：HBase 从2.0.0 开始默认使用的是asyncfs。解决办法：（每个节点）编辑配置文档：vim hbase-site.xml,并添加：
hbase.unsafe.stream.capability.enforce false
在master上再次启动start-hbase.sh，后查看： hbase(main):027:0> status 1 active master, 0 backup masters, 3 servers, 29 dead, 0.3333 average load Took 0.0158 seconds

这里有很多dead节点，hbase在节点重启后不会清理（达到最大限制），可以手动清理。

gitava commented 4 years ago

[vagrant@hdp-node-01 ~/hbase/conf]$hbase hbck -repair
2020-07-03 03:47:15,698 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hbase Fsck connecting to ZooKeeper ensemble=hdp-node-01:2181,hdp-node-02:2181,hdp-node-03:2181
2020-07-03 03:47:15,707 INFO  [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2020-07-03 03:47:15,708 INFO  [main] zookeeper.ZooKeeper: Client environment:host.name=hdp-node-01
2020-07-03 03:47:15,708 INFO  [main] zookeeper.ZooKeeper: Client environment:java.version=1.7.0_80
2020-07-03 03:47:15,708 INFO  [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2020-07-03 03:47:15,708 INFO  [main] zookeeper.ZooKeeper: Client environment:java.home=/home/vagrant/jdk/jre
2020-07-03 03:47:15,708 INFO  [main] zookeeper.ZooKeeper: educe/lib/leveldbjni-all-1.8.jar:/home/vagrant/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/home/vagrant/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/vagrant/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.9.2.jar:/home/vagrant/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.9.2.jar:/home/vagrant/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.9.2.jar:/home/vagrant/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.2.jar:/home/vagrant/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.9.2.jar:/home/vagrant/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.9.2.jar:/home/vagrant/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.9.2.jar:/home/vagrant/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.9.2-tests.jar:/home/vagrant/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.9.2.jar::/home/vagrant/myclass:/home/vagrant/hadoop/etc/hadoop
2020-07-03 03:47:15,708 INFO  [main] zookeeper.ZooKeeper: Client environment:java.library.path=/home/vagrant/hadoop/lib/native
2020-07-03 03:47:15,708 INFO  [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2020-07-03 03:47:15,708 INFO  [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2020-07-03 03:47:15,708 INFO  [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2020-07-03 03:47:15,708 INFO  [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2020-07-03 03:47:15,708 INFO  [main] zookeeper.ZooKeeper: Client environment:os.version=3.10.0-1127.el7.x86_64
2020-07-03 03:47:15,708 INFO  [main] zookeeper.ZooKeeper: Client environment:user.name=vagrant
2020-07-03 03:47:15,708 INFO  [main] zookeeper.ZooKeeper: Client environment:user.home=/home/vagrant
2020-07-03 03:47:15,708 INFO  [main] zookeeper.ZooKeeper: Client environment:user.dir=/home/vagrant/hbase-1.6.0/conf
2020-07-03 03:47:15,709 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=hdp-node-01:2181,hdp-node-02:2181,hdp-node-03:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@36f233f
2020-07-03 03:47:15,729 INFO  [main-SendThread(hdp-node-02:2181)] zookeeper.ClientCnxn: Opening socket connection to server hdp-node-02/192.168.33.102:2181. Will not attempt to authenticate using SASL (unknown error)
2020-07-03 03:47:15,733 INFO  [main-SendThread(hdp-node-02:2181)] zookeeper.ClientCnxn: Socket connection established to hdp-node-02/192.168.33.102:2181, initiating session
HBaseFsck command line options: -repair
2020-07-03 03:47:15,795 INFO  [main-SendThread(hdp-node-02:2181)] zookeeper.ClientCnxn: Session establishment complete on server hdp-node-02/192.168.33.102:2181, sessionid = 0x17312c876a60003, negotiated timeout = 90000
2020-07-03 03:47:15,901 INFO  [main] util.HBaseFsck: Launching hbck
2020-07-03 03:47:16,102 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x7725aa0c connecting to ZooKeeper ensemble=hdp-node-01:2181,hdp-node-02:2181,hdp-node-03:2181
2020-07-03 03:47:16,103 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=hdp-node-01:2181,hdp-node-02:2181,hdp-node-03:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@49bf36f0
2020-07-03 03:47:16,127 INFO  [main-SendThread(hdp-node-02:2181)] zookeeper.ClientCnxn: Opening socket connection to server hdp-node-02/192.168.33.102:2181. Will not attempt to authenticate using SASL (unknown error)
2020-07-03 03:47:16,147 INFO  [main-SendThread(hdp-node-02:2181)] zookeeper.ClientCnxn: Socket connection established to hdp-node-02/192.168.33.102:2181, initiating session
2020-07-03 03:47:16,158 INFO  [main-SendThread(hdp-node-02:2181)] zookeeper.ClientCnxn: Session establishment complete on server hdp-node-02/192.168.33.102:2181, sessionid = 0x17312c876a60004, negotiated timeout = 90000
Version: 1.6.0
2020-07-03 03:47:16,684 INFO  [main] util.HBaseFsck: Loading regioninfos HDFS
2020-07-03 03:47:16,684 INFO  [main] util.HBaseFsck: Loading HBase regioninfo from HDFS...

2020-07-03 03:47:17,141 INFO  [main] util.HBaseFsck: Checking HBase region split map from HDFS data...
2020-07-03 03:47:17,141 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
2020-07-03 03:47:17,151 INFO  [main] util.HBaseFsck: No integrity errors.  We are done with this phase. Glorious.
2020-07-03 03:47:17,151 INFO  [main] util.HBaseFsck: Computing mapping of all store files

2020-07-03 03:47:17,172 INFO  [main] util.HBaseFsck: Validating mapping using HDFS state
2020-07-03 03:47:17,172 INFO  [main] util.HBaseFsck: Computing mapping of all link files

2020-07-03 03:47:17,194 INFO  [main] util.HBaseFsck: Validating mapping using HDFS state
Number of live region servers: 3
Number of dead region servers: 0
Master: hdp-node-01,16000,1593747995503
Number of backup masters: 0
Average load: 0.6666666666666666
Number of requests: 0
Number of regions: 2
Number of regions in transition: 0
2020-07-03 03:47:17,310 INFO  [main] util.HBaseFsck: Loading regionsinfo from the hbase:meta table

Number of empty REGIONINFO_QUALIFIER rows in hbase:meta: 0
2020-07-03 03:47:17,369 INFO  [main] util.HBaseFsck: getHTableDescriptors == tableNames => []
2020-07-03 03:47:17,371 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x18972cc3 connecting to ZooKeeper ensemble=hdp-node-01:2181,hdp-node-02:2181,hdp-node-03:2181
2020-07-03 03:47:17,371 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=hdp-node-01:2181,hdp-node-02:2181,hdp-node-03:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@6cfbc2d1
2020-07-03 03:47:17,373 INFO  [main-SendThread(hdp-node-02:2181)] zookeeper.ClientCnxn: Opening socket connection to server hdp-node-02/192.168.33.102:2181. Will not attempt to authenticate using SASL (unknown error)
2020-07-03 03:47:17,373 INFO  [main-SendThread(hdp-node-02:2181)] zookeeper.ClientCnxn: Socket connection established to hdp-node-02/192.168.33.102:2181, initiating session
2020-07-03 03:47:17,377 INFO  [main-SendThread(hdp-node-02:2181)] zookeeper.ClientCnxn: Session establishment complete on server hdp-node-02/192.168.33.102:2181, sessionid = 0x17312c876a60005, negotiated timeout = 90000
2020-07-03 03:47:17,387 INFO  [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
2020-07-03 03:47:17,387 INFO  [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x17312c876a60005
2020-07-03 03:47:17,391 INFO  [main] zookeeper.ZooKeeper: Session: 0x17312c876a60005 closed
2020-07-03 03:47:17,392 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x17312c876a60005
Number of Tables: 0
2020-07-03 03:47:17,393 INFO  [main] util.HBaseFsck: Loading region directories from HDFS

2020-07-03 03:47:17,418 INFO  [main] util.HBaseFsck: Loading region information from HDFS

2020-07-03 03:47:17,470 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x7c9ae69d connecting to ZooKeeper ensemble=hdp-node-01:2181,hdp-node-02:2181,hdp-node-03:2181
2020-07-03 03:47:17,470 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=hdp-node-01:2181,hdp-node-02:2181,hdp-node-03:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@1d9a8d54
2020-07-03 03:47:17,472 INFO  [main-SendThread(hdp-node-03:2181)] zookeeper.ClientCnxn: Opening socket connection to server hdp-node-03/192.168.33.103:2181. Will not attempt to authenticate using SASL (unknown error)
2020-07-03 03:47:17,473 INFO  [main-SendThread(hdp-node-03:2181)] zookeeper.ClientCnxn: Socket connection established to hdp-node-03/192.168.33.103:2181, initiating session
2020-07-03 03:47:17,478 INFO  [main-SendThread(hdp-node-03:2181)] zookeeper.ClientCnxn: Session establishment complete on server hdp-node-03/192.168.33.103:2181, sessionid = 0x27312c8727d0004, negotiated timeout = 90000
2020-07-03 03:47:17,489 INFO  [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x27312c8727d0004
2020-07-03 03:47:17,492 INFO  [main] zookeeper.ZooKeeper: Session: 0x27312c8727d0004 closed
2020-07-03 03:47:17,492 INFO  [main] util.HBaseFsck: Checking and fixing region consistency
2020-07-03 03:47:17,493 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x27312c8727d0004
2020-07-03 03:47:17,508 INFO  [main] util.HBaseFsck: Handling overlap merges in parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
2020-07-03 03:47:17,611 INFO  [main] util.HBaseFsck: Finishing hbck
Summary:
Table hbase:meta is okay.
    Number of regions: 1
    Deployed on:  hdp-node-02,16020,1593747996975
Table hbase:namespace is okay.
    Number of regions: 1
    Deployed on:  hdp-node-03,16020,1593747996709
0 inconsistencies detected.
Status: OK
2020-07-03 03:47:17,619 INFO  [main] zookeeper.ZooKeeper: Session: 0x17312c876a60003 closed
2020-07-03 03:47:17,620 INFO  [main] client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
2020-07-03 03:47:17,620 INFO  [main] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x17312c876a60004
2020-07-03 03:47:17,620 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x17312c876a60003
2020-07-03 03:47:17,624 INFO  [main] zookeeper.ZooKeeper: Session: 0x17312c876a60004 closed
2020-07-03 03:47:17,624 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x17312c876a60004
[vagrant@hdp-node-01 ~/hbase/conf]$

gitava commented 4 years ago

Error solved although hdp3 still like this ..it's not a hbase issue (ref: #37)

[vagrant@hdp-node-03 ~/hbase/logs]$jps
2113 DataNode
3476 -- process information unavailable
3783 Jps
2226 NodeManager
3587 -- process information unavailable
[vagrant@hdp-node-03 ~/hbase/logs]$

gitava commented 4 years ago

review the log hbase-vagrant-regionserver-hdp-node-02.log, error disappeared.

[vagrant@hdp-node-01 ~]$jps
2833 NodeManager
6514 HRegionServer
6821 Jps
2550 SecondaryNameNode
3034 JobHistoryServer
6283 HQuorumPeer
2380 DataNode
6380 HMaster
2239 NameNode
[vagrant@hdp-node-01 ~]$hbase shell
HBase Shell
Use "help" to get list of supported commands.
Use "exit" to quit this interactive shell.
Version 1.6.0, r5ec5a5b115ee36fb28903667c008218abd21b3f5, Fri Feb 14 12:00:03 PST 2020

hbase(main):001:0> list
TABLE
0 row(s) in 0.4230 seconds

=> []
hbase(main):002:0>

gitava / Big-Data-Study

hbase - cannot get log writer #35