Open kunga opened 7 months ago
https://nda.ya.ru/t/n6PUIdLH7595hj B-Tree + Flat, Flat, B-Tree
2024-03-26 05:41 UTC tpcc_ydb: Loading data done in 1854 seconds
2024-03-26 05:53 UTC tpcc_ydb: Compaction done in 713 seconds
2024-03-26 06:06 UTC tpcc_ydb: Built index in 747 seconds
tpmC: 202416
2024-03-26 01:03 UTC tpcc_ydb: Loading data done in 1854 seconds
2024-03-26 01:13 UTC tpcc_ydb: Compaction done in 556 seconds
2024-03-26 01:24 UTC tpcc_ydb: Built index in 683 seconds
tpmC: 202205
https://nda.ya.ru/t/jHlSyKq3759KBW B-Tree, Flat
2024-03-26 12:18 UTC zipfian workload a: 345.4K Op/s, latency 99% 133.119 ms: 414848.7K oks, 0 not_found, 448 errors, 1201089 ms run time
2024-03-26 12:39 UTC zipfian workload b: 501.3K Op/s, latency 99% 18.655 ms: 600000.1K oks, 0 not_found, 0 errors, 1200882 ms run time
2024-03-26 13:00 UTC zipfian workload c: 548.6K Op/s, latency 99% 11.159 ms: 600000.1K oks, 0 not_found, 0 errors, 1099871 ms run time
2024-03-26 13:19 UTC zipfian workload f: 261.8K Op/s, latency 99% 19.631 ms: 471487.2K oks, 0 not_found, 259 errors, 1201088 ms run time
2024-03-26 13:40 UTC latest workload d: 61.9K Op/s, latency 99% 832.511 ms: 74310.4K oks, 9.9K not_found, 1.9K errors, 1201119 ms run time
2024-03-26 14:12 UTC zipfian workload e: 65.3K Op/s, latency 99% 99.007 ms: 39999.9K oks, 0 not_found, 128 errors, 612947 ms run time
2024-03-26 14:43 UTC zipfian workload a: 354.8K Op/s, latency 99% 28.479 ms: 426058.9K oks, 0 not_found, 344 errors, 1201099 ms run time
2024-03-26 15:04 UTC zipfian workload b: 494.7K Op/s, latency 99% 22.751 ms: 594172.8K oks, 0 not_found, 12 errors, 1201091 ms run time
2024-03-26 15:25 UTC zipfian workload c: 532.5K Op/s, latency 99% 14.367 ms: 600000.1K oks, 0 not_found, 0 errors, 1132455 ms run time
2024-03-26 15:45 UTC zipfian workload f: 260.9K Op/s, latency 99% 23.839 ms: 469954.6K oks, 0 not_found, 135 errors, 1201086 ms run time
2024-03-26 16:06 UTC latest workload d: 66.8K Op/s, latency 99% 577.023 ms: 80221.9K oks, 2.2K not_found, 601 errors, 1201123 ms run time
2024-03-26 16:37 UTC zipfian workload e: 63.6K Op/s, latency 99% 474.367 ms: 39999.8K oks, 0 not_found, 264 errors, 629655 ms run time
https://nda.ya.ru/t/u4oX-oPx759rBA
https://nda.ya.ru/t/G8iQ0UAh759rCW
https://nda.ya.ru/t/KHu9a09t75gmqt
2024-04-30 11:38 UTC tpcc_ydb: Loading data done in 1831 seconds
2024-04-30 11:47 UTC tpcc_ydb: Compaction done in 562 seconds
2024-04-30 12:00 UTC tpcc_ydb: Built index in 757 seconds
Result: Total
Time: 7200 seconds
Start measure: 1714480513
Warehouses: 16000
New orders: 24262986
tpmC: 202191
Efficiency: 98.27
Throughput: 7497
Goodput: 8721
NewOrder: OK: 24262986, FAILED: 208 (0.0%)
Payment: OK: 23236184, FAILED: 10 (0.0%)
OrderStatus: OK: 2162490, FAILED: 0
Delivery: OK: 2156203, FAILED: 2988 (0.14%)
StockLevel: OK: 2162091, FAILED: 0
NewOrder:
50%: 320 ms
90%: 512 ms
95%: 512 ms
99%: 1250 ms
99.9%: 2000 ms
Payment:
50%: 144 ms
90%: 224 ms
95%: 256 ms
99%: 480 ms
99.9%: 512 ms
OrderStatus:
50%: 96 ms
90%: 192 ms
95%: 224 ms
99%: 320 ms
99.9%: 480 ms
Delivery:
50%: 1250 ms
90%: 2500 ms
95%: 3000 ms
99%: 4500 ms
99.9%: 6000 ms
StockLevel:
50%: 72 ms
90%: 144 ms
95%: 192 ms
99%: 352 ms
99.9%: 512 ms
2024-04-30 17:26 UTC tpcc_ydb: Loading data done in 1825 seconds
2024-04-30 17:35 UTC tpcc_ydb: Compaction done in 566 seconds
2024-04-30 17:47 UTC tpcc_ydb: Built index in 725 seconds
Result: Total
Time: 7200 seconds
Start measure: 1714501358
Warehouses: 16000
New orders: 24303007
tpmC: 202524
Efficiency: 98.43
Throughput: 7508
Goodput: 8738
NewOrder: OK: 24303007, FAILED: 136 (0.0%)
Payment: OK: 23273075, FAILED: 6 (0.0%)
OrderStatus: OK: 2166365, FAILED: 0
Delivery: OK: 2163922, FAILED: 2020 (0.09%)
StockLevel: OK: 2165109, FAILED: 0
NewOrder:
50%: 288 ms
90%: 512 ms
95%: 512 ms
99%: 1000 ms
99.9%: 1500 ms
Payment:
50%: 128 ms
90%: 192 ms
95%: 224 ms
99%: 416 ms
99.9%: 512 ms
OrderStatus:
50%: 88 ms
90%: 176 ms
95%: 224 ms
99%: 288 ms
99.9%: 384 ms
Delivery:
50%: 1250 ms
90%: 2250 ms
95%: 2750 ms
99%: 4000 ms
99.9%: 5500 ms
StockLevel:
50%: 64 ms
90%: 128 ms
95%: 192 ms
99%: 320 ms
99.9%: 512 ms
https://nda.ya.ru/t/sMAtUlme75hHrn
2024-04-30 22:59 UTC zipfian workload a: 355.1K Op/s, latency 99% 190.079 ms: 426468.1K oks, 0 not_found, 115 errors, 1201085 ms run time
2024-04-30 23:20 UTC zipfian workload b: 490.8K Op/s, latency 99% 18.271 ms: 589394.8K oks, 0 not_found, 0 errors, 1201091 ms run time
2024-04-30 23:41 UTC zipfian workload c: 520.5K Op/s, latency 99% 11.383 ms: 600000.1K oks, 0 not_found, 0 errors, 1153658 ms run time
2024-05-01 00:01 UTC zipfian workload f: 263.6K Op/s, latency 99% 27.871 ms: 474887.7K oks, 0 not_found, 274 errors, 1201085 ms run time
2024-05-01 00:22 UTC latest workload d: 66.6K Op/s, latency 99% 594.431 ms: 80124K oks, 0 not_found, 0 errors, 1204251 ms run time
2024-05-01 00:53 UTC zipfian workload e: 64.4K Op/s, latency 99% 83.263 ms: 40000K oks, 0 not_found, 98 errors, 621939 ms run time
2024-05-01 04:02 UTC zipfian workload a: 372.8K Op/s, latency 99% 214.527 ms: 447685.1K oks, 0 not_found, 157 errors, 1201088 ms run time
2024-05-01 08:09 UTC zipfian workload a: 363.4K Op/s, latency 99% 202.239 ms: 436449K oks, 0 not_found, 224 errors, 1201078 ms run time
2024-05-01 04:23 UTC zipfian workload b: 499.7K Op/s, latency 99% 16.015 ms: 600000.1K oks, 0 not_found, 0 errors, 1200871 ms run time
2024-05-01 08:30 UTC zipfian workload b: 496.7K Op/s, latency 99% 16.199 ms: 596508.7K oks, 0 not_found, 1 errors, 1201091 ms run time
2024-05-01 04:44 UTC zipfian workload c: 519.7K Op/s, latency 99% 11.415 ms: 600000.1K oks, 0 not_found, 0 errors, 1157306 ms run time
2024-05-01 08:52 UTC zipfian workload c: 517.9K Op/s, latency 99% 11.263 ms: 600000.1K oks, 0 not_found, 0 errors, 1161620 ms run time
2024-05-01 05:04 UTC zipfian workload f: 268.6K Op/s, latency 99% 27.999 ms: 483795.6K oks, 0 not_found, 421 errors, 1201087 ms run time
2024-05-01 09:12 UTC zipfian workload f: 265.3K Op/s, latency 99% 50.847 ms: 477805K oks, 0 not_found, 734 errors, 1201083 ms run time
2024-05-01 05:25 UTC latest workload d: 63.2K Op/s, latency 99% 635.903 ms: 75794.2K oks, 0 not_found, 0 errors, 1201124 ms run time
2024-05-01 09:33 UTC latest workload d: 64.2K Op/s, latency 99% 647.679 ms: 77055.7K oks, 0 not_found, 0 errors, 1201127 ms run time
2024-05-01 05:56 UTC zipfian workload e: 65.2K Op/s, latency 99% 145.407 ms: 40000K oks, 0 not_found, 7 errors, 614401 ms run time
2024-05-01 10:03 UTC zipfian workload e: 64.2K Op/s, latency 99% 77.759 ms: 40000K oks, 0 not_found, 25 errors, 623347 ms run time
https://nda.ya.ru/t/E-eHq6GZ75hyB5
2024-05-01 13:17 UTC tpcc_ydb: Loading data done in 1405 seconds
2024-05-01 13:24 UTC tpcc_ydb: Compaction done in 411 seconds
2024-05-01 13:33 UTC tpcc_ydb: Built index in 555 seconds
Result: Total
Time: 7200 seconds
Start measure: 1714572511
Warehouses: 12000
New orders: 18355322
tpmC: 152959
Efficiency: 99.12
Throughput: 5670
Goodput: 6601
NewOrder: OK: 18355322, FAILED: 230 (0.0%)
Payment: OK: 17574233, FAILED: 1 (0.0%)
OrderStatus: OK: 1635884, FAILED: 0
Delivery: OK: 1636733, FAILED: 54 (0.0%)
StockLevel: OK: 1635779, FAILED: 2 (0.0%)
NewOrder:
50%: 128 ms
90%: 288 ms
95%: 384 ms
99%: 512 ms
99.9%: 1000 ms
Payment:
50%: 48 ms
90%: 112 ms
95%: 160 ms
99%: 256 ms
99.9%: 416 ms
OrderStatus:
50%: 40 ms
90%: 144 ms
95%: 176 ms
99%: 256 ms
99.9%: 384 ms
Delivery:
50%: 512 ms
90%: 512 ms
95%: 1000 ms
99%: 1500 ms
99.9%: 2250 ms
StockLevel:
50%: 32 ms
90%: 176 ms
95%: 256 ms
99%: 448 ms
99.9%: 512 ms
2024-05-01 16:41 UTC tpcc_ydb: Loading data done in 1369 seconds
2024-05-01 16:49 UTC tpcc_ydb: Compaction done in 441 seconds
2024-05-01 16:57 UTC tpcc_ydb: Built index in 513 seconds
Result: Total
Time: 7200 seconds
Start measure: 1714584764
Warehouses: 12000
New orders: 18358584
tpmC: 152986
Efficiency: 99.14
Throughput: 5669
Goodput: 6600
NewOrder: OK: 18358584, FAILED: 67 (0.0%)
Payment: OK: 17568479, FAILED: 8 (0.0%)
OrderStatus: OK: 1633702, FAILED: 0
Delivery: OK: 1634485, FAILED: 77 (0.0%)
StockLevel: OK: 1635871, FAILED: 0
NewOrder:
50%: 144 ms
90%: 288 ms
95%: 384 ms
99%: 512 ms
99.9%: 1000 ms
Payment:
50%: 48 ms
90%: 112 ms
95%: 160 ms
99%: 256 ms
99.9%: 448 ms
OrderStatus:
50%: 40 ms
90%: 144 ms
95%: 192 ms
99%: 256 ms
99.9%: 416 ms
Delivery:
50%: 512 ms
90%: 512 ms
95%: 1000 ms
99%: 1500 ms
99.9%: 2250 ms
StockLevel:
50%: 32 ms
90%: 192 ms
95%: 288 ms
99%: 480 ms
99.9%: 512 ms
https://nda.ya.ru/t/oyxKo3hD75iN9U
2024-05-02 08:57 UTC tpcc_ydb: Loading data done in 1478 seconds
2024-05-02 09:05 UTC tpcc_ydb: Compaction done in 443 seconds
2024-05-02 09:14 UTC tpcc_ydb: Built index in 566 seconds
Result: Total
Time: 7200 seconds
Start measure: 1714643384
Warehouses: 12000
New orders: 18178342
tpmC: 151485
Efficiency: 98.16
Throughput: 5616
Goodput: 6544
NewOrder: OK: 18178342, FAILED: 1617 (0.01%)
Payment: OK: 17405059, FAILED: 25 (0.0%)
OrderStatus: OK: 1621301, FAILED: 15 (0.0%)
Delivery: OK: 1620295, FAILED: 1185 (0.07%)
StockLevel: OK: 1618387, FAILED: 2 (0.0%)
NewOrder:
50%: 352 ms
90%: 512 ms
95%: 1000 ms
99%: 1500 ms
99.9%: 3000 ms
Payment:
50%: 96 ms
90%: 224 ms
95%: 288 ms
99%: 448 ms
99.9%: 512 ms
OrderStatus:
50%: 128 ms
90%: 288 ms
95%: 352 ms
99%: 512 ms
99.9%: 512 ms
Delivery:
50%: 1250 ms
90%: 2500 ms
95%: 3000 ms
99%: 4000 ms
99.9%: 5750 ms
StockLevel:
50%: 224 ms
90%: 512 ms
95%: 512 ms
99%: 1250 ms
99.9%: 2000 ms
2024-05-02 13:18 UTC tpcc_ydb: Loading data done in 1431 seconds
2024-05-02 13:26 UTC tpcc_ydb: Compaction done in 455 seconds
2024-05-02 13:34 UTC tpcc_ydb: Built index in 534 seconds
Result: Total
Time: 7200 seconds
Start measure: 1714659003
Warehouses: 12000
New orders: 18172437
tpmC: 151435
Efficiency: 98.13
Throughput: 5613
Goodput: 6540
NewOrder: OK: 18172437, FAILED: 564 (0.0%)
Payment: OK: 17402606, FAILED: 18 (0.0%)
OrderStatus: OK: 1620966, FAILED: 0
Delivery: OK: 1616072, FAILED: 1003 (0.06%)
StockLevel: OK: 1618266, FAILED: 0
NewOrder:
50%: 352 ms
90%: 512 ms
95%: 1000 ms
99%: 1500 ms
99.9%: 2750 ms
Payment:
50%: 96 ms
90%: 224 ms
95%: 288 ms
99%: 480 ms
99.9%: 512 ms
OrderStatus:
50%: 144 ms
90%: 320 ms
95%: 384 ms
99%: 512 ms
99.9%: 512 ms
Delivery:
50%: 1250 ms
90%: 2500 ms
95%: 3000 ms
99%: 4250 ms
99.9%: 5750 ms
StockLevel:
50%: 256 ms
90%: 512 ms
95%: 1000 ms
99%: 1250 ms
99.9%: 2000 ms
https://nda.ya.ru/t/MYwKhca_75iuBR
2024-05-02 19:10 UTC tpcc_ydb: Loading data done in 1501 seconds
2024-05-02 19:18 UTC tpcc_ydb: Compaction done in 450 seconds
2024-05-02 19:27 UTC tpcc_ydb: Built index in 555 seconds
Result: Total
Time: 7200 seconds
Start measure: 1714680150
Warehouses: 12000
New orders: 13898018
tpmC: 115815
Efficiency: 75.05
Throughput: 4328
Goodput: 5228
NewOrder: OK: 13898018, FAILED: 105094 (0.75%)
Payment: OK: 13374895, FAILED: 48410 (0.36%)
OrderStatus: OK: 1245056, FAILED: 3990 (0.32%)
Delivery: OK: 1215879, FAILED: 32209 (2.58%)
StockLevel: OK: 1244627, FAILED: 3817 (0.31%)
NewOrder:
50%: 9000 ms
90%: 15000 ms
95%: 20000 ms
99%: 20000 ms
99.9%: 20000 ms
Payment:
50%: 2750 ms
90%: 4000 ms
95%: 5250 ms
99%: 9000 ms
99.9%: 15000 ms
OrderStatus:
50%: 3000 ms
90%: 4000 ms
95%: 5250 ms
99%: 8000 ms
99.9%: 15000 ms
Delivery:
50%: 9000 ms
90%: 20000 ms
95%: 20000 ms
99%: 20000 ms
99.9%: 20000 ms
StockLevel:
50%: 3750 ms
90%: 5500 ms
95%: 6000 ms
99%: 9000 ms
99.9%: 15000 ms
2024-05-02 23:01 UTC tpcc_ydb: Loading data done in 1492 seconds
2024-05-02 23:09 UTC tpcc_ydb: Compaction done in 451 seconds
2024-05-02 23:17 UTC tpcc_ydb: Built index in 491 seconds
Result: Total
Time: 7200 seconds
Start measure: 1714693941
Warehouses: 12000
New orders: 14137163
tpmC: 117808
Efficiency: 76.34
Throughput: 4400
Goodput: 5299
NewOrder: OK: 14137163, FAILED: 90520 (0.64%)
Payment: OK: 13614464, FAILED: 37143 (0.27%)
OrderStatus: OK: 1268181, FAILED: 2875 (0.23%)
Delivery: OK: 1247545, FAILED: 18966 (1.5%)
StockLevel: OK: 1266308, FAILED: 2891 (0.23%)
NewOrder:
50%: 8000 ms
90%: 14000 ms
95%: 20000 ms
99%: 20000 ms
99.9%: 20000 ms
Payment:
50%: 2500 ms
90%: 4000 ms
95%: 5250 ms
99%: 8000 ms
99.9%: 15000 ms
OrderStatus:
50%: 2500 ms
90%: 4000 ms
95%: 5250 ms
99%: 8000 ms
99.9%: 15000 ms
Delivery:
50%: 8000 ms
90%: 15000 ms
95%: 20000 ms
99%: 20000 ms
99.9%: 20000 ms
StockLevel:
50%: 3250 ms
90%: 4750 ms
95%: 5750 ms
99%: 9000 ms
99.9%: 15000 ms
https://nda.ya.ru/t/lJkYsNo375joSi
2024-05-03 10:52 UTC tpcc_ydb: Loading data done in 1526 seconds
2024-05-03 10:59 UTC tpcc_ydb: Compaction done in 434 seconds
2024-05-03 11:08 UTC tpcc_ydb: Built index in 546 seconds
Result: Total
Time: 7200 seconds
Start measure: 1714736605
Warehouses: 12000
New orders: 8644037
tpmC: 72032
Efficiency: 46.68
Throughput: 3465
Goodput: 3453
NewOrder: OK: 8644037, FAILED: 2567753 (22.9%)
Payment: OK: 8832887, FAILED: 1915043 (17.82%)
OrderStatus: OK: 826607, FAILED: 174705 (17.45%)
Delivery: OK: 563554, FAILED: 433429 (43.47%)
StockLevel: OK: 830724, FAILED: 170401 (17.02%)
NewOrder:
50%: 15000 ms
90%: 20000 ms
95%: 20000 ms
99%: 20000 ms
99.9%: 20000 ms
Payment:
50%: 6000 ms
90%: 14000 ms
95%: 15000 ms
99%: 15000 ms
99.9%: 15000 ms
OrderStatus:
50%: 6000 ms
90%: 14000 ms
95%: 15000 ms
99%: 15000 ms
99.9%: 15000 ms
Delivery:
50%: 15000 ms
90%: 20000 ms
95%: 20000 ms
99%: 20000 ms
99.9%: 20000 ms
StockLevel:
50%: 7000 ms
90%: 14000 ms
95%: 15000 ms
99%: 15000 ms
99.9%: 20000 ms
2024-05-03 17:04 UTC tpcc_ydb: Loading data done in 1546 seconds
2024-05-03 17:12 UTC tpcc_ydb: Compaction done in 452 seconds
2024-05-03 17:21 UTC tpcc_ydb: Built index in 545 seconds
Result: Total
Time: 7200 seconds
Start measure: 1714759001
Warehouses: 12000
New orders: 9081395
tpmC: 75677
Efficiency: 49.04
Throughput: 3515
Goodput: 3611
NewOrder: OK: 9081395, FAILED: 2289951 (20.14%)
Payment: OK: 9290304, FAILED: 1616212 (14.82%)
OrderStatus: OK: 867382, FAILED: 147384 (14.52%)
Delivery: OK: 638883, FAILED: 373551 (36.9%)
StockLevel: OK: 870645, FAILED: 144406 (14.23%)
NewOrder:
50%: 15000 ms
90%: 20000 ms
95%: 20000 ms
99%: 20000 ms
99.9%: 20000 ms
Payment:
50%: 6000 ms
90%: 13000 ms
95%: 15000 ms
99%: 15000 ms
99.9%: 15000 ms
OrderStatus:
50%: 6000 ms
90%: 13000 ms
95%: 15000 ms
99%: 15000 ms
99.9%: 15000 ms
Delivery:
50%: 15000 ms
90%: 20000 ms
95%: 20000 ms
99%: 20000 ms
99.9%: 20000 ms
StockLevel:
50%: 6000 ms
90%: 13000 ms
95%: 15000 ms
99%: 15000 ms
99.9%: 15000 ms
https://nda.ya.ru/t/llFWtuzo75kKft
2024-05-04 11:33 UTC zipfian workload a: 369.1K Op/s, latency 99% 158.335 ms: 443248.7K oks, 0 not_found, 422 errors, 1201090 ms run time
2024-05-04 11:54 UTC zipfian workload b: 503.5K Op/s, latency 99% 18.815 ms: 600000.1K oks, 0 not_found, 0 errors, 1191820 ms run time
2024-05-04 12:15 UTC zipfian workload c: 527.8K Op/s, latency 99% 11.335 ms: 600000.1K oks, 0 not_found, 0 errors, 1142053 ms run time
2024-05-04 12:35 UTC zipfian workload f: 276K Op/s, latency 99% 32.607 ms: 497156K oks, 0 not_found, 335 errors, 1201087 ms run time
2024-05-04 12:56 UTC latest workload d: 63.6K Op/s, latency 99% 600.063 ms: 76383.2K oks, 0 not_found, 0 errors, 1201122 ms run time
2024-05-04 13:28 UTC zipfian workload e: 65K Op/s, latency 99% 244.735 ms: 40000K oks, 0 not_found, 33 errors, 615966 ms run time
2024-05-04 15:25 UTC zipfian workload a: 375.7K Op/s, latency 99% 55.167 ms: 451246.9K oks, 0 not_found, 413 errors, 1201087 ms run time
2024-05-04 15:47 UTC zipfian workload b: 500.3K Op/s, latency 99% 17.599 ms: 600000.1K oks, 0 not_found, 2 errors, 1199313 ms run time
2024-05-04 16:08 UTC zipfian workload c: 524.9K Op/s, latency 99% 11.559 ms: 600000.1K oks, 0 not_found, 0 errors, 1145535 ms run time
2024-05-04 16:28 UTC zipfian workload f: 275.4K Op/s, latency 99% 25.727 ms: 496070.9K oks, 0 not_found, 358 errors, 1201091 ms run time
2024-05-04 16:49 UTC latest workload d: 62.5K Op/s, latency 99% 591.359 ms: 75009.4K oks, 0 not_found, 0 errors, 1201149 ms run time
2024-05-04 17:21 UTC zipfian workload e: 62.9K Op/s, latency 99% 85.119 ms: 40000K oks, 0 not_found, 19 errors, 636709 ms run time
Description
Implement a readonly B-Tree SST index instead of the current flat one.
This improvement opens us the possibility of not always needing to keep SST's indexes in memory. We will be able to store a large amount of "cold" data without dozens of compute nodes to load needed parts of indexes on demand. Also enhances stability of Shared Cache and removes (actually reduces) passive bytes segment.
As we write a SST during the compaction process, its indexes will be readonly. However, the B-Tree structure minimizes needed disk I/O operations.
Also we will extend the current index format with additional information about data size and erased row count which will allow us to navigate easily and count statistics without the need of fully loading the whole index into memory.
Internal design doc.
Steps
1. Support simple flat index loading in all the usages
Instead of loading all SST's indexes on a tablet start, do it on the first usage. Before that loading keep memory free.
IIndexIter
interface that loads index on the first usage https://github.com/ydb-platform/ydb/commit/66a3bbb229cb89b2af275fd583259d3a47e215b4TRunIt
https://github.com/ydb-platform/ydb/commit/66a3bbb229cb89b2af275fd583259d3a47e215b4IPages
implementations https://github.com/ydb-platform/ydb/commit/aab08a6fe3bcdbe14fa7e61c38825fecd152b7a0 https://github.com/ydb-platform/ydb/commit/10fe9530d0a2328e2b7bd4c081dfaf58fc2ee525TCharge
https://github.com/ydb-platform/ydb/commit/55d8ae0f29a754250bb75958428dfe3b543cdaf9TKeysLoader
https://github.com/ydb-platform/ydb/commit/0f2a2ff24b46998fe341a18f0ac90a2d722249fa https://github.com/ydb-platform/ydb/commit/bac30c7bd7a0129d6f9d8b76dfee813f615573feBuildStats
https://github.com/ydb-platform/ydb/commit/287b4bb4d2239bb7555407d03950d323ebf0edce https://github.com/ydb-platform/ydb/commit/a18f54585040aedbeeebd9d1cdcc1950146ae81aTForward
https://github.com/ydb-platform/ydb/commit/5ad1dcc7936525dc6742bcbf855e291cc71dbda4 https://github.com/ydb-platform/ydb/pull/657TDump
https://github.com/ydb-platform/ydb/commit/cc271cdcd8792ac27caefcbe45c7349ccd5b5765TPart
https://github.com/ydb-platform/ydb/commit/16b0f8d9f0fba37ed662509e8f20577aa073ca482. Implement readonly B-Tree index
Implement B-Tree data structure, build and write them with an SST, implement searches.
TPartWriter
https://github.com/ydb-platform/ydb/commit/1ccc8508988675ed855c7cf27474d5b660098822TRunIt
with a new B-Tree index iterator (TPartBtreeIndexIt
) https://github.com/ydb-platform/ydb/commit/2f7d8b6e692bd7584a66686b5504381631dd4522EnableLocalDBBtreeIndex
setting https://github.com/ydb-platform/ydb/commit/ba6b7d6c666b8a304518ccc1c8e9a0daec8df47aTKeysLoader
with a new B-Tree index iterator https://github.com/ydb-platform/ydb/commit/897ada7941ae63332514c27e676d2f70edc3c13b3. Implement Precharge over readonly B-Tree index
Implement Precharge that works over tree structured index instead of the old flat one.
ICharge
interface and extractChargeRange
methods https://github.com/ydb-platform/ydb/commit/e7f64cc383ca087378ff3edb8b833360561445a7 https://github.com/ydb-platform/ydb/pull/6824. Implement Build Stats over readonly B-Tree index
Implement Build Stats that works over tree structured index instead of the old flat one. We expect that it will no longer need to fully load index and only use upper-level B-Tree index nodes.
5. Implement Scan over readonly B-Tree index
Implement Scan that works over over tree structured index.
6. Make fixes
Manually check suspicious places and fix what is broken with B-Tree index.
TSchemeShard::Clear
method https://github.com/ydb-platform/ydb/pull/655EnableLocalDBBtreeIndex
setting https://github.com/ydb-platform/ydb/pull/653 https://github.com/ydb-platform/ydb/pull/671Cache line head don't want to do fetch but should
verify https://github.com/ydb-platform/ydb/pull/31346. Test
Verify that everything works together not dramatically worse than before. Make tests on a slice.
TRunIt
over B-Tree index https://github.com/ydb-platform/ydb/pull/653EnableLocalDBBtreeIndex
andEnableLocalDBFlatIndex
settings work7. Make optimizations
8. Release
EnableLocalDBBtreeIndex
setting on pre-prod clusters (both indexes will be produced) (for some databases first)EnableLocalDBBtreeIndex
setting on prod clusters (for some databases first)EnableLocalDBBtreeIndex
setting by defaultEnableLocalDBFlatIndex
setting