Hbase&Phoenix - Githubissues

nanne007 commented 7 years ago

Hbase BlockCache

http://www.n10k.com/blog/blockcache-101/ http://www.n10k.com/blog/blockcache-showdown/

nanne007 commented 6 years ago

Phoenix 4.7 BUG 集锦

    @Override
    public HTableDescriptor getTableDescriptor(byte[] tableName) throws SQLException {
        HTableInterface htable = getTable(tableName);
        try {
            return htable.getTableDescriptor();
        } catch (IOException e) {
            if(e instanceof org.apache.hadoop.hbase.TableNotFoundException ||
                e.getCause() instanceof org.apache.hadoop.hbase.TableNotFoundException) {
              byte[][] schemaAndTableName = new byte[2][];
              // BUG here
              SchemaUtil.getVarChars(tableName, schemaAndTableName);
              throw new TableNotFoundException(Bytes.toString(schemaAndTableName[0]), Bytes.toString(schemaAndTableName[1]));
            }
            throw new RuntimeException(e);
        } finally {
            Closeables.closeQuietly(htable);
        }
    }

nanne007 commented 5 years ago

ExpressionCompiler

[ ] TODO

ParseNodeVisitor

Query Plan

Select

fromTable
select
where
groupBy
having
limit
orderBy
isDistinct
hint

isAggregate # 也会基于 groupBy de stateless 和 having
bindCount
hasSequence
udfParseNodes
selects, # sub query

hasWildcard # 基于 select 出来的字段

先把 sub select 拍平，
然后做normalize, resolve column,

之后把 sub query 变成 join, 见 SubqueryRewriter，

/*
* Class for rewriting where-clause sub-queries into join queries.
* 
* If the where-clause sub-query is one of those top-node conditions (being 
* the only condition node or direct descendant of AND nodes), we convert the
* sub-query directly into semi-joins, anti-joins or inner-joins, and meanwhile
* remove the original condition node from the where clause.
* Otherwise, we convert the sub-query into left-joins and change the original
* condition node into a null test of a join table field (ONE if matched, NULL 
* if not matched).
*/

HBase Index write

Indexer, RegionObserver，

监听更新的 WALEdit，用 IndexBuilder build 出要索引更新的 WALEdit，写WAL，然后 IndexWriter 写盘。
Replay WAL 的时候，重写索引，如果索引表不存在，则跳过。

MetaDataClient

创建表

statement 由 CreateTableCompiler compile 成 plan，plan 执行方法中调用 metadata client 的 createTable 方法。

CREATE TABLE => UPSERT INTO SYSTEM.CATALOG(...)

update index state

MatadataClient -> ConnectionService -> MetaDataService -> acquire row lock

MedaDataRegionObserver

找出当前 disable 的 index 表，一次性更新 data 表相关的所有 index。

index from disable to inactive
[ ] IndexMaintainer
[ ] dataHTable batch ? 写回原表，让整个流程重新来一遍
index from inactive to active

Meta Data 相关

MedaDataClient -> PMetaData -> PMetaDataCache

SYSTEM.CATALOG 存储细节

每个表（包括 data 表，index 表）在 hbase 里面会以 tenantId, schemaName, tableName 为前缀存储一系列 rows。

包括：

tenantId+schemaName+tableName: 表相关的属性，包括 tableType, pkName 等等。
tenantId+schemaName+tableName+indexName: 表关联的索引或者视图：只会存在一个属性，用来判断是索引还是视图。
tenantId+schemaName+tableName+columnName+columnFamilyName: 表的字段，以及字段在 hbase 中存储用的 familyName。

\x00RAW\x00LOAN_CONSUMER_APPLICATION/0:BASE_COLUMN_COUNT/1504788098542/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00\x00LOAN_CONSUMER_APPLICATION_ADDRESS/0:LINK_TYPE/1504788098544/Put/vlen=1/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00\x00LOAN_CONSUMER_APPLICATION_DEVICE_ID/0:LINK_TYPE/1504788098545/Put/vlen=1/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00\x00LOAN_CONSUMER_APPLICATION_EMAIL/0:LINK_TYPE/1504788098547/Put/vlen=1/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00\x00LOAN_CONSUMER_APPLICATION_ID_NO_MOB/0:LINK_TYPE/1504788098549/Put/vlen=1/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00\x00LOAN_CONSUMER_APPLICATION_IP/0:LINK_TYPE/1504788098552/Put/vlen=1/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00\x00LOAN_CONSUMER_APPLICATION_MAIN_CONTACT_TEL/0:LINK_TYPE/1504788098554/Put/vlen=1/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00\x00LOAN_CONSUMER_APPLICATION_MOB/0:LINK_TYPE/1504788098557/Put/vlen=1/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00ADDRESS_CITY\x000/0:DATA_TYPE/1504788098559/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00ADDRESS_COUNTY\x000/0:DATA_TYPE/1504788098560/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00ADDRESS_DETAIL\x000/0:DATA_TYPE/1504788098564/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00ADDRESS_PROV\x000/0:DATA_TYPE/1504788098566/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00AGREEMENT_ID\x000/0:DATA_TYPE/1504788098567/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00APPLICATION_NO\x000/0:DATA_TYPE/1504788098570/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00APPLICATION_STATUS\x000/0:DATA_TYPE/1504788098573/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00APPLY_AMOUNT\x000/0:DATA_TYPE/1504788098576/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00APPLY_INSTALLMENT\x000/0:DATA_TYPE/1504788098578/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00APPLY_STATUS\x000/0:DATA_TYPE/1504788098581/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00APPLY_TIME\x000/0:DATA_TYPE/1504788098582/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00APPROVE_COMMENT\x000/0:DATA_TYPE/1504788098584/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00APPROVE_MODEL_STATUS\x000/0:DATA_TYPE/1504788098586/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00APPROVE_REJECT_CODE\x000/0:DATA_TYPE/1504788098587/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00APPROVE_STATUS\x000/0:DATA_TYPE/1504788098589/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00APPROVE_TIME\x000/0:DATA_TYPE/1504788098590/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00APPROVE_USER\x000/0:DATA_TYPE/1504788098592/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00APP_VER\x000/0:DATA_TYPE/1504788098593/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00ASSET_DESC\x000/0:DATA_TYPE/1504788098595/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00AUTO_REFUSE_TIME\x000/0:DATA_TYPE/1504788098596/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00BIND_MOB\x000/0:DATA_TYPE/1504788098598/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00BINLOG_TIME\x000/0:DATA_TYPE/1504788098600/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00CONTACT_STATUS\x000/0:DATA_TYPE/1504788098603/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00CREATE_TIME\x000/0:DATA_TYPE/1504788098605/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00DEBIT_CARD_BANK\x000/0:DATA_TYPE/1504788098607/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00DEBIT_CARD_CHECK_ORDER_NO\x000/0:DATA_TYPE/1504788098609/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00DEBIT_CARD_CHECK_TIME\x000/0:DATA_TYPE/1504788098612/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00DEBIT_CARD_CITY\x000/0:DATA_TYPE/1504788098614/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00DEBIT_CARD_NO\x000/0:DATA_TYPE/1504788098615/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00DEBIT_CARD_PROV\x000/0:DATA_TYPE/1504788098617/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00DEBIT_CARD_STATUS\x000/0:DATA_TYPE/1504788098620/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00DEVICE_ID\x000/0:DATA_TYPE/1504788098623/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00DONGHUA_APP_NO\x000/0:DATA_TYPE/1504788098626/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00DONGHUA_CALL_TIME\x000/0:DATA_TYPE/1504788098629/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00DONGHUA_COMMENT\x000/0:DATA_TYPE/1504788098631/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00DONGHUA_STATUS\x000/0:DATA_TYPE/1504788098632/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00EMAIL\x000/0:DATA_TYPE/1504788098634/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00FUND_USAGE\x000/0:DATA_TYPE/1504788098635/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00HOUSE_LOAN\x000/0:DATA_TYPE/1504788098637/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00ID\x000/0:DATA_TYPE/1504788098639/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00ID_NO\x000/0:DATA_TYPE/1504788098640/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00INCOME_SOURCE\x000/0:DATA_TYPE/1504788098642/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00INSTALLMENT\x000/0:DATA_TYPE/1504788098643/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00INTEREST_RATE\x000/0:DATA_TYPE/1504788098645/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00IP\x000/0:DATA_TYPE/1504788098646/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00LAST_MOD_TIME\x000/0:DATA_TYPE/1504788098648/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00LATITUDE\x000/0:DATA_TYPE/1504788098650/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00LEADS_STATUS\x000/0:DATA_TYPE/1504788098652/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00LOAN_AMOUT\x000/0:DATA_TYPE/1504788098655/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00LOAN_FEE\x000/0:DATA_TYPE/1504788098657/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00LOAN_MSG_READ\x000/0:DATA_TYPE/1504788098659/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00LOAN_STATUS\x000/0:DATA_TYPE/1504788098660/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00LOAN_TIME\x000/0:DATA_TYPE/1504788098664/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00LOAN_TYPE\x000/0:DATA_TYPE/1504788098665/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00LONGITUDE\x000/0:DATA_TYPE/1504788098667/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00MAIN_CONTACT_NAME\x000/0:DATA_TYPE/1504788098669/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00MAIN_CONTACT_RELA\x000/0:DATA_TYPE/1504788098672/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00MAIN_CONTACT_TEL\x000/0:DATA_TYPE/1504788098676/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00MC\x000/0:DATA_TYPE/1504788098679/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00MOB\x000/0:DATA_TYPE/1504788098681/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00MONTHLY_EXPENDITURE\x000/0:DATA_TYPE/1504788098683/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00OVERDUE_RATE\x000/0:DATA_TYPE/1504788098685/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00PAY_REQUEST_ID\x000/0:DATA_TYPE/1504788098687/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00PAY_REQUEST_STATUS\x000/0:DATA_TYPE/1504788098689/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00PAY_REQUEST_TIME\x000/0:DATA_TYPE/1504788098690/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00PK/0:DATA_TYPE/1504788098692/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00PLATFORM\x000/0:DATA_TYPE/1504788098694/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00REAL_NAME\x000/0:DATA_TYPE/1504788098696/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00REJECT_CODE_LEVEL1\x000/0:DATA_TYPE/1504788098697/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00REJECT_CODE_LEVEL2\x000/0:DATA_TYPE/1504788098699/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00REPAY_DAY\x000/0:DATA_TYPE/1504788098702/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00REPAY_STATUS\x000/0:DATA_TYPE/1504788098705/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00REPAY_TIME\x000/0:DATA_TYPE/1504788098707/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00RISKER_SCORE\x000/0:DATA_TYPE/1504788098709/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00SALARY\x000/0:DATA_TYPE/1504788098711/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00SERVICE_RATE\x000/0:DATA_TYPE/1504788098715/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00TERM_DAY\x000/0:DATA_TYPE/1504788098716/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00UID\x000/0:DATA_TYPE/1504788098719/Put/vlen=4/seqid=0
\x00RAW\x00LOAN_CONSUMER_APPLICATION\x00WF\x000/0:DATA_TYPE/1504788098722/Put/vlen=4/seqid=0

        byte[] key = SchemaUtil.getTableKey(null, "RAW", "LOAN_CONSUMER_APPLICATION");
        Scan scan = MetaDataUtil.newTableRowsScan(key, 0, HConstants.LATEST_TIMESTAMP);
        ResultScanner scanner = table.getScanner(scan);
        while (true) {
            Result result = scanner.next();
            if (result == null) {
                break;
            }

            List<Cell> cells = result.listCells();
            if (cells == null) {
                continue;
            }

            System.out.println(CellUtil.getCellKeyAsString(cells.get(0)));
        }

nanne007 / notes

Hbase&Phoenix #68