search

Huawei-Hadoop / hindex

Secondary Index for HBase
Apache License 2.0
591 stars 286 forks source link

hindex write issues #42

Closed hy2014 closed 10 years ago

hy2014 commented 10 years ago

These days i do write performance test. There is two table, on include index(called idx_table), other not(called no_idx_table). Write no_idx_table is ok, but after write some data to idx_table, region server always throw some exception: org.apache.hadoop.hbase.NotServingRegionException: xxxx. is closing

In the http://hmaster:60010, in the "Regions in Transition", there are two region(one is idx_table, the other is idx_table_idx) display state=PENDING_CLOSE, ts=xxxxx, server=null

who can tell me what is wrong, it always happen this issues.

anoopsjohn commented 10 years ago

Can u test once by removing the "idx" from ur user table names? Call tables as table1 and table2 and test. Let us know there also u are seeing similar issues or not.

hy2014 commented 10 years ago

actually, my table names not include any "idx" string. "no_idx_table" is just a example, not the real table name.

anoopsjohn commented 10 years ago

What is the cluster size? Can u give the master and RS logs?

chrajeshbabu commented 10 years ago

can u run hbck to fix regions in transition? First we can find the reason why regions in transition. If possible can u give region server thread dump also?

hy2014 commented 10 years ago

After I run the hbck, it display all the region is ok. I can give the region server log:

2014-03-27 12:32:04,239 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received close region: event_log_pref_1,b800094060ab267603d3e50f688fb17a_gszg4,1395894708084.863d53f7f4e88d667131151c4266acfd.. Version of ZK closing node:0 2014-03-27 12:32:04,239 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of event_log_pref_1,b800094060ab267603d3e50f688fb17a_gszg4,1395894708084.863d53f7f4e88d667131151c4266acfd. 2014-03-27 12:32:04,239 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing event_log_pref_1,b800094060ab267603d3e50f688fb17a_gszg4,1395894708084.863d53f7f4e88d667131151c4266acfd.: disabling compactions & flushes 2014-03-27 12:32:04,240 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: waiting for 1 compactions to complete for region event_log_pref_1,b800094060ab267603d3e50f688fb17a_gszg4,1395894708084.863d53f7f4e88d667131151c4266acfd. 2014-03-27 12:32:04,258 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received close region: event_log_pref_1_idx,b800094060ab267603d3e50f688fb17a_gszg4,1395894712319.e4eabfe71644616730beca129a7b7138.. Version of ZK closing node:0 2014-03-27 12:32:04,258 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of event_log_pref_1_idx,b800094060ab267603d3e50f688fb17a_gszg4,1395894712319.e4eabfe71644616730beca129a7b7138. 2014-03-27 12:32:04,259 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing event_log_pref_1_idx,b800094060ab267603d3e50f688fb17a_gszg4,1395894712319.e4eabfe71644616730beca129a7b7138.: disabling compactions & flushes 2014-03-27 12:32:04,278 WARN org.apache.hadoop.hbase.regionserver.HRegion: Failed getting lock in batch put, row=b800094060ab267603d3e50f688fb17a_gszg4\x00idx1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00launch\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00b15f67000069c692195e8ff34181e666_7t85s org.apache.hadoop.hbase.NotServingRegionException: event_log_pref_1_idx,b800094060ab267603d3e50f688fb17a_gszg4,1395894712319.e4eabfe71644616730beca129a7b7138. is closing at org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:5819) at org.apache.hadoop.hbase.regionserver.HRegion.internalObtainRowLock(HRegion.java:3424) at org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:3519) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2289) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutateForIndex(HRegion.java:2158) at org.apache.hadoop.hbase.index.coprocessor.regionserver.IndexRegionObserver.postBatchMutate(IndexRegionObserver.java:398) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postBatchMutate(RegionCoprocessorHost.java:1612) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2485) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2117) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3809) at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)

hy2014 commented 10 years ago

Below is the master log:

2014-03-27 12:31:44,767 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=event_log_pref_1,,1395894708084.d1e2c0e6ee7ca3c8c84f3b5fe83264f1., src=ubuntu-cache,60020,1395883954547, dest=ubuntu-hnn,60020,1395883934550 2014-03-27 12:31:44,767 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of region event_log_pref_1,,1395894708084.d1e2c0e6ee7ca3c8c84f3b5fe83264f1. (offlining) 2014-03-27 12:31:44,767 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x144f193670500fe Creating unassigned node for d1e2c0e6ee7ca3c8c84f3b5fe83264f1 in a CLOSING state 2014-03-27 12:31:44,794 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to ubuntu-cache,60020,1395883954547 for region event_log_pref_1,,1395894708084.d1e2c0e6ee7ca3c8c84f3b5fe83264f1. 2014-03-27 12:31:44,794 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=event_log_pref_1,b800094060ab267603d3e50f688fb17a_gszg4,1395894708084.863d53f7f4e88d667131151c4266acfd., src=ubuntu-cache,60020,1395883954547, dest=ubuntu-hnn,60020,1395883934550 2014-03-27 12:31:44,794 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of region event_log_pref_1,b800094060ab267603d3e50f688fb17a_gszg4,1395894708084.863d53f7f4e88d667131151c4266acfd. (offlining) 2014-03-27 12:31:44,794 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x144f193670500fe Creating unassigned node for 863d53f7f4e88d667131151c4266acfd in a CLOSING state 2014-03-27 12:31:44,810 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to ubuntu-cache,60020,1395883954547 for region event_log_pref_1,b800094060ab267603d3e50f688fb17a_gszg4,1395894708084.863d53f7f4e88d667131151c4266acfd. 2014-03-27 12:31:44,810 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=event_log_pref_1_idx,b800094060ab267603d3e50f688fb17a_gszg4,1395894712319.e4eabfe71644616730beca129a7b7138., src=ubuntu-cache,60020,1395883954547, dest=ubuntu-hnn,60020,1395883934550 2014-03-27 12:31:44,810 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of region event_log_pref_1_idx,b800094060ab267603d3e50f688fb17a_gszg4,1395894712319.e4eabfe71644616730beca129a7b7138. (offlining) 2014-03-27 12:31:44,810 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x144f193670500fe Creating unassigned node for e4eabfe71644616730beca129a7b7138 in a CLOSING state 2014-03-27 12:31:44,832 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to ubuntu-cache,60020,1395883954547 for region event_log_pref_1_idx,b800094060ab267603d3e50f688fb17a_gszg4,1395894712319.e4eabfe71644616730beca129a7b7138. 2014-03-27 12:31:44,832 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=event_log_pref_1_idx,,1395894712319.8f3224e7893d218b7215caf57d897b22., src=ubuntu-cache,60020,1395883954547, dest=ubuntu-hnn,60020,1395883934550 2014-03-27 12:31:44,832 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of region event_log_pref_1_idx,,1395894712319.8f3224e7893d218b7215caf57d897b22. (offlining) 2014-03-27 12:31:44,832 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x144f193670500fe Creating unassigned node for 8f3224e7893d218b7215caf57d897b22 in a CLOSING state 2014-03-27 12:31:44,851 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to ubuntu-cache,60020,1395883954547 for region event_log_pref_1_idx,,1395894712319.8f3224e7893d218b7215caf57d897b22. 2014-03-27 12:31:44,900 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_CLOSED, server=ubuntu-cache,60020,1395883954547, region=d1e2c0e6ee7ca3c8c84f3b5fe83264f1 2014-03-27 12:31:44,900 DEBUG org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED event for d1e2c0e6ee7ca3c8c84f3b5fe83264f1 2014-03-27 12:31:44,900 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=event_log_pref_1,,1395894708084.d1e2c0e6ee7ca3c8c84f3b5fe83264f1. state=CLOSED, ts=1395894724323, server=ubuntu-cache,60020,1395883954547 2014-03-27 12:31:44,900 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x144f193670500fe Creating (or updating) unassigned node for d1e2c0e6ee7ca3c8c84f3b5fe83264f1 with OFFLINE state 2014-03-27 12:31:44,924 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=M_ZK_REGION_OFFLINE, server=ubuntu-hmaster,60000,1395884494516, region=d1e2c0e6ee7ca3c8c84f3b5fe83264f1 2014-03-27 12:31:44,924 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for event_log_pref_1,,1395894708084.d1e2c0e6ee7ca3c8c84f3b5fe83264f1. destination server is ubuntu-hnn,60020,1395883934550 2014-03-27 12:31:44,925 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan for region event_log_pref_1,,1395894708084.d1e2c0e6ee7ca3c8c84f3b5fe83264f1.; plan=hri=event_log_pref_1,,1395894708084.d1e2c0e6ee7ca3c8c84f3b5fe83264f1., src=ubuntu-cache,60020,1395883954547, dest=ubuntu-hnn,60020,1395883934550 2014-03-27 12:31:44,925 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Assigning region event_log_pref_1,,1395894708084.d1e2c0e6ee7ca3c8c84f3b5fe83264f1. to ubuntu-hnn,60020,1395883934550 2014-03-27 12:31:44,940 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_CLOSED, server=ubuntu-cache,60020,1395883954547, region=8f3224e7893d218b7215caf57d897b22 2014-03-27 12:31:44,940 DEBUG org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED event for 8f3224e7893d218b7215caf57d897b22 2014-03-27 12:31:44,940 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=event_log_pref_1_idx,,1395894712319.8f3224e7893d218b7215caf57d897b22. state=CLOSED, ts=1395894724363, server=ubuntu-cache,60020,1395883954547 2014-03-27 12:31:44,941 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x144f193670500fe Creating (or updating) unassigned node for 8f3224e7893d218b7215caf57d897b22 with OFFLINE state 2014-03-27 12:31:44,949 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=ubuntu-hnn,60020,1395883934550, region=d1e2c0e6ee7ca3c8c84f3b5fe83264f1 2014-03-27 12:31:44,961 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=M_ZK_REGION_OFFLINE, server=ubuntu-hmaster,60000,1395884494516, region=8f3224e7893d218b7215caf57d897b22 2014-03-27 12:31:44,961 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for event_log_pref_1_idx,,1395894712319.8f3224e7893d218b7215caf57d897b22. destination server is ubuntu-hnn,60020,1395883934550 2014-03-27 12:31:44,962 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan for region event_log_pref_1_idx,,1395894712319.8f3224e7893d218b7215caf57d897b22.; plan=hri=event_log_pref_1_idx,,1395894712319.8f3224e7893d218b7215caf57d897b22., src=ubuntu-cache,60020,1395883954547, dest=ubuntu-hnn,60020,1395883934550 2014-03-27 12:31:44,962 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Assigning region event_log_pref_1_idx,,1395894712319.8f3224e7893d218b7215caf57d897b22. to ubuntu-hnn,60020,1395883934550 2014-03-27 12:31:45,000 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=ubuntu-hnn,60020,1395883934550, region=8f3224e7893d218b7215caf57d897b22 2014-03-27 12:31:45,014 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=ubuntu-hnn,60020,1395883934550, region=d1e2c0e6ee7ca3c8c84f3b5fe83264f1 2014-03-27 12:31:45,016 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=ubuntu-hnn,60020,1395883934550, region=8f3224e7893d218b7215caf57d897b22 2014-03-27 12:31:45,064 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENED, server=ubuntu-hnn,60020,1395883934550, region=8f3224e7893d218b7215caf57d897b22 2014-03-27 12:31:45,065 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Handling OPENED event for event_log_pref_1_idx,,1395894712319.8f3224e7893d218b7215caf57d897b22. from ubuntu-hnn,60020,1395883934550; deleting unassigned node 2014-03-27 12:31:45,065 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x144f193670500fe Deleting existing unassigned node for 8f3224e7893d218b7215caf57d897b22 that is in expected state RS_ZK_REGION_OPENED 2014-03-27 12:31:45,067 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENED, server=ubuntu-hnn,60020,1395883934550, region=d1e2c0e6ee7ca3c8c84f3b5fe83264f1 2014-03-27 12:31:45,067 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Handling OPENED event for event_log_pref_1,,1395894708084.d1e2c0e6ee7ca3c8c84f3b5fe83264f1. from ubuntu-hnn,60020,1395883934550; deleting unassigned node 2014-03-27 12:31:45,067 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x144f193670500fe Deleting existing unassigned node for d1e2c0e6ee7ca3c8c84f3b5fe83264f1 that is in expected state RS_ZK_REGION_OPENED 2014-03-27 12:31:45,082 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: The znode of region event_log_pref_1_idx,,1395894712319.8f3224e7893d218b7215caf57d897b22. has been deleted. 2014-03-27 12:31:45,082 INFO org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the region event_log_pref_1_idx,,1395894712319.8f3224e7893d218b7215caf57d897b22. that was online on ubuntu-hnn,60020,1395883934550 2014-03-27 12:31:45,086 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x144f193670500fe Successfully deleted unassigned node for region 8f3224e7893d218b7215caf57d897b22 in expected state RS_ZK_REGION_OPENED 2014-03-27 12:31:45,090 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x144f193670500fe Successfully deleted unassigned node for region d1e2c0e6ee7ca3c8c84f3b5fe83264f1 in expected state RS_ZK_REGION_OPENED 2014-03-27 12:31:45,099 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: The znode of region event_log_pref_1,,1395894708084.d1e2c0e6ee7ca3c8c84f3b5fe83264f1. has been deleted. 2014-03-27 12:31:45,100 INFO org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the region event_log_pref_1,,1395894708084.d1e2c0e6ee7ca3c8c84f3b5fe83264f1. that was online on ubuntu-hnn,60020,1395883934550 2014-03-27 12:36:43,274 DEBUG org.apache.hadoop.hbase.client.MetaScanner: Scanning .META. starting at row= for max=2147483647 rows using org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@2c1450bd 2014-03-27 12:36:43,323 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor: Deleting region event_log_pref_1_idx,,1395893984556.02a4a8a9d864b9504a258397de8564f3. because daughter splits no longer hold references 2014-03-27 12:36:43,323 DEBUG org.apache.hadoop.hbase.index.SecIndexLoadBalancer: The regioninfo {NAME => 'event_log_pref_1_idx,,1395893984556.02a4a8a9d864b9504a258397de8564f3.', STARTKEY => '', ENDKEY => '', ENCODED => 02a4a8a9d864b9504a258397de8564f3, OFFLINE => true, SPLIT => true,} removed from the region plan 2014-03-27 12:36:43,323 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: ARCHIVING region hdfs://ubuntu-hnn:9000/hbase/event_log_pref_1_idx/02a4a8a9d864b9504a258397de8564f3 2014-03-27 12:36:43,323 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: Have an archive directory, preparing to move files 2014-03-27 12:36:43,326 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: Archiving:[class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://ubuntu-hnn:9000/hbase/event_log_pref_1_idx/02a4a8a9d864b9504a258397de8564f3/d] 2014-03-27 12:36:43,326 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: Starting to archive files:[class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://ubuntu-hnn:9000/hbase/event_log_pref_1_idx/02a4a8a9d864b9504a258397de8564f3/d] 2014-03-27 12:36:43,326 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: moving files to the archive directory: hdfs://ubuntu-hnn:9000/hbase/.archive/event_log_pref_1_idx/02a4a8a9d864b9504a258397de8564f3 2014-03-27 12:36:43,354 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: Created archive directory:hdfs://ubuntu-hnn:9000/hbase/.archive/event_log_pref_1_idx/02a4a8a9d864b9504a258397de8564f3 2014-03-27 12:36:43,354 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: Archiving:class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://ubuntu-hnn:9000/hbase/event_log_pref_1_idx/02a4a8a9d864b9504a258397de8564f3/d 2014-03-27 12:36:43,355 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://ubuntu-hnn:9000/hbase/event_log_pref_1_idx/02a4a8a9d864b9504a258397de8564f3/d is a directory, archiving children files 2014-03-27 12:36:43,356 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: moving files to the archive directory: hdfs://ubuntu-hnn:9000/hbase/.archive/event_log_pref_1_idx/02a4a8a9d864b9504a258397de8564f3/d 2014-03-27 12:36:43,373 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: Created archive directory:hdfs://ubuntu-hnn:9000/hbase/.archive/event_log_pref_1_idx/02a4a8a9d864b9504a258397de8564f3/d 2014-03-27 12:36:43,374 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: Archiving:class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://ubuntu-hnn:9000/hbase/event_log_pref_1_idx/02a4a8a9d864b9504a258397de8564f3/d/b1c82766c9c54e94ad28f9d140a7bb08 2014-03-27 12:36:43,375 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: No existing file in archive for:hdfs://ubuntu-hnn:9000/hbase/.archive/event_log_pref_1_idx/02a4a8a9d864b9504a258397de8564f3/d/b1c82766c9c54e94ad28f9d140a7bb08, free to archive original file. 2014-03-27 12:36:43,385 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: Finished archiving file from: class org.apache.hadoop.hbase.backup.HFileArchiver$FileablePath, file:hdfs://ubuntu-hnn:9000/hbase/event_log_pref_1_idx/02a4a8a9d864b9504a258397de8564f3/d/b1c82766c9c54e94ad28f9d140a7bb08, to: hdfs://ubuntu-hnn:9000/hbase/.archive/event_log_pref_1_idx/02a4a8a9d864b9504a258397de8564f3/d/b1c82766c9c54e94ad28f9d140a7bb08 2014-03-27 12:36:43,385 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: Successfully resolved and archived, now can just delete region. 2014-03-27 12:36:43,398 DEBUG org.apache.hadoop.hbase.backup.HFileArchiver: Deleted all region files in: hdfs://ubuntu-hnn:9000/hbase/event_log_pref_1_idx/02a4a8a9d864b9504a258397de8564f3 2014-03-27 12:36:43,487 INFO org.apache.hadoop.hbase.catalog.MetaEditor: Deleted region event_log_pref_1_idx,,1395893984556.02a4a8a9d864b9504a258397de8564f3. from META 2014-03-27 12:36:43,487 INFO org.apache.hadoop.hbase.master.CatalogJanitor: Scanned 173 catalog row(s) and gc'd 1 unreferenced parent region(s) 2014-03-27 12:36:44,739 DEBUG org.apache.hadoop.hbase.master.HMaster: Not running balancer because 2 region(s) in transition: {863d53f7f4e88d667131151c4266acfd=event_log_pref_1,b800094060ab267603d3e50f688fb17a_gszg4,1395894708084.863d53f7f4e88d667131151c4266acfd. state=PENDING_CLOSE, ts=1395894706273, server=null, e4eabfe71644616730beca129a7b7138=event_log_p...

hy2014 commented 10 years ago

Actually, I do the performance last week, but it can be ok. The different is that,

  1. on the last week, I create table with IndexedHTableDescriptor.
  2. But this week, i first create the table in the hbase shell, then use IndexUtil.createIndexTable to modify the table (because our hbase table already has some data in production before we use hindex).

But i found it always occur "Regions in Transition"(the region is blocked in Region.doClose() method).
and i found this region and its index region is not in the same region server(but if I use the first way to create table, everything is ok)

So I concern after I run the TableIndexer Job(create table first, then modify table), does it also occur this problem?

Thanks