backend/local: optimize local writer concurrency and memory usage

[glorv]

What problem does this PR solve?

Optimize local writer performance and memory usage.

What is changed and how it works?

Optimize local writer by:

• Open an index LocalWriter for each chunk to avoid the bottle neck at write index kvs
• Replace the async ApplendRows with sync operation, to optimize the memory usage and simplify the logic
• decrease the encode/deliverLoop chan to decrease memory usage
• use a memory buffer to restore the temp kv pairs before write them to SST file
• Batch ingest SST files into pebble to avoid the sync overhead in pebble ingest operation

Bench result: The benchmark tests were run on a 40core machine with an NVME disk. and based on the following three data and table schema:

• DataSet1. 14k warehouse tpcc data
• DataSet2. 1k warehouse order_line table with 3 indexes. Thus each row generates 4 kvs.

  PRIMARY KEY (`ol_w_id`,`ol_d_id`,`ol_o_id`,`ol_number`),
  KEY `idx_d_i` (`ol_d_id`, `ol_i_id`),
  KEY `idx_d_w_supply` (`ol_d_id`, `ol_w_id`, `ol_supply_w_id`)

• DataSet3. 1k warehouse order_line table with 3 indexes. Thus each row generates 8 kvs.

  PRIMARY KEY (`ol_w_id`,`ol_d_id`,`ol_o_id`,`ol_number`),
  KEY `idx_d_i` (`ol_d_id`,`ol_i_id`),
  KEY `idx_d_w_supply` (`ol_d_id`,`ol_w_id`,`ol_supply_w_id`),
  KEY `idx_o_amount` (`ol_o_id`,`ol_amount`),
  KEY `idx_d_supply` (`ol_d_id`,`ol_supply_w_id`),
  KEY `idx_o_d_i` (`ol_o_id`,`ol_d_id`,`ol_i_id`),
  KEY `idx_i_id` (`ol_i_id`)

Bench Result:	DataSet	Branch	Peak Memory	Cost Time
DataSet1	master	34GB	2h25m
DataSet1	opt-local-writer	29GB	1h46m
DataSet2	master	30GB	24m57s
DataSet2	opt-local-writer	30GB	9m32s
DataSet3	master	64GB	1h22m
DataSet3	opt-local-writer	33GB	33m

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

Related changes

[glorv]

/run-all-tests

[glorv]

/run-all-tests

[glorv]

closed in favor of https://github.com/pingcap/br/pull/753