👑 Readonly B-Tree SST index

Description

Implement a readonly B-Tree SST index instead of the current flat one.

This improvement opens us the possibility of not always needing to keep SST's indexes in memory. We will be able to store a large amount of "cold" data without dozens of compute nodes to load needed parts of indexes on demand. Also enhances stability of Shared Cache and removes (actually reduces) passive bytes segment.

As we write a SST during the compaction process, its indexes will be readonly. However, the B-Tree structure minimizes needed disk I/O operations.

Also we will extend the current index format with additional information about data size and erased row count which will allow us to navigate easily and count statistics without the need of fully loading the whole index into memory.

Internal design doc.

Steps

1. Support simple flat index loading in all the usages

Instead of loading all SST's indexes on a tablet start, do it on the first usage. Before that loading keep memory free.

[x] Add and implement an index IIndexIter interface that loads index on the first usage https://github.com/ydb-platform/ydb/commit/66a3bbb229cb89b2af275fd583259d3a47e215b4
[x] Support index loading in TRunIt https://github.com/ydb-platform/ydb/commit/66a3bbb229cb89b2af275fd583259d3a47e215b4
[x] Support index loading from IPages implementations https://github.com/ydb-platform/ydb/commit/aab08a6fe3bcdbe14fa7e61c38825fecd152b7a0 https://github.com/ydb-platform/ydb/commit/10fe9530d0a2328e2b7bd4c081dfaf58fc2ee525
[x] Support index loading inTCharge https://github.com/ydb-platform/ydb/commit/55d8ae0f29a754250bb75958428dfe3b543cdaf9
[x] Support index loading in TKeysLoader https://github.com/ydb-platform/ydb/commit/0f2a2ff24b46998fe341a18f0ac90a2d722249fa https://github.com/ydb-platform/ydb/commit/bac30c7bd7a0129d6f9d8b76dfee813f615573fe
[x] Support index loading in BuildStats https://github.com/ydb-platform/ydb/commit/287b4bb4d2239bb7555407d03950d323ebf0edce https://github.com/ydb-platform/ydb/commit/a18f54585040aedbeeebd9d1cdcc1950146ae81a
[x] Support index loading in TForward https://github.com/ydb-platform/ydb/commit/5ad1dcc7936525dc6742bcbf855e291cc71dbda4 https://github.com/ydb-platform/ydb/pull/657
[x] Support index loading in TDump https://github.com/ydb-platform/ydb/commit/cc271cdcd8792ac27caefcbe45c7349ccd5b5765
[x] Remove the rest direct usages of index fields from TPart https://github.com/ydb-platform/ydb/commit/16b0f8d9f0fba37ed662509e8f20577aa073ca48
[x] Do not count flat index pages in transaction usage https://github.com/ydb-platform/ydb/commit/e4592d6bd86384da2667f8399b41ca65858f3ee8
[x] Do not load index on tablet start https://github.com/ydb-platform/ydb/commit/bcae19f0ae90c93d0ab978ef1b4e516053f19535

2. Implement readonly B-Tree index

Implement B-Tree data structure, build and write them with an SST, implement searches.

[x] Implement B-Tree node class with serialization and deserialization to binary pages format https://github.com/ydb-platform/ydb/commit/cf4d301c44398f63b387618472918acfff0dc87c
[x] Implement B-Tree builder that accepts data pages and writes B-Tree nodes online https://github.com/ydb-platform/ydb/commit/523950c2b381cefddbfee36b9d788ea420964b76
[x] Integrate B-Tree builder with TPartWriter https://github.com/ydb-platform/ydb/commit/1ccc8508988675ed855c7cf27474d5b660098822
[x] Optimize B-Tree node format for groups where all row records have the same size https://github.com/ydb-platform/ydb/commit/4c94e44e7808c0b62370bee78c216fd131216d8c https://github.com/ydb-platform/ydb/commit/ebe7a9b148efe82f86ed8ee5efbc3c3d40ef5833
[x] Implement search by row id https://github.com/ydb-platform/ydb/commit/402da95804025c14d23aef3aad037797381bed0d
[x] Implement search by key https://github.com/ydb-platform/ydb/commit/1eda4dd4f9ac1ba457c1b73a9ce9cb450f50b132
[x] Implement next and prev methods that can handle page faults https://github.com/ydb-platform/ydb/commit/2f7d8b6e692bd7584a66686b5504381631dd4522
[x] Integrate TRunIt with a new B-Tree index iterator (TPartBtreeIndexIt) https://github.com/ydb-platform/ydb/commit/2f7d8b6e692bd7584a66686b5504381631dd4522
[x] Add EnableLocalDBBtreeIndex setting https://github.com/ydb-platform/ydb/commit/ba6b7d6c666b8a304518ccc1c8e9a0daec8df47a
[x] Integrate TKeysLoader with a new B-Tree index iterator https://github.com/ydb-platform/ydb/commit/897ada7941ae63332514c27e676d2f70edc3c13b

3. Implement Precharge over readonly B-Tree index

Implement Precharge that works over tree structured index instead of the old flat one.

[x] Add ICharge interface and extract ChargeRange methods https://github.com/ydb-platform/ydb/commit/e7f64cc383ca087378ff3edb8b833360561445a7 https://github.com/ydb-platform/ydb/pull/682
[x] Charge main pages within given row id range https://github.com/ydb-platform/ydb/pull/754
[x] Charge main pages within given key range https://github.com/ydb-platform/ydb/pull/1102
[x] Charge groups pages https://github.com/ydb-platform/ydb/pull/1224
[x] Treat items limit https://github.com/ydb-platform/ydb/pull/1338
[x] Treats bytes limit https://github.com/ydb-platform/ydb/pull/1489
[x] Charge history pages https://github.com/ydb-platform/ydb/pull/1593
[x] Bugfix partially-loaded tree https://github.com/ydb-platform/ydb/pull/1674
[x] Check that Iterator and Charge behaviour are consistent (all needed pages are precharged) https://github.com/ydb-platform/ydb/pull/1869

4. Implement Build Stats over readonly B-Tree index

Implement Build Stats that works over tree structured index instead of the old flat one. We expect that it will no longer need to fully load index and only use upper-level B-Tree index nodes.

[x] Implement basic version https://github.com/ydb-platform/ydb/pull/1949 https://github.com/ydb-platform/ydb/pull/2136
[x] Improve it not to load all index nodes https://github.com/ydb-platform/ydb/pull/2177

5. Implement Scan over readonly B-Tree index

Implement Scan that works over over tree structured index.

[x] Implement Scan over B-Tree index basic version #2260
[x] Forward-load B-Tree index in Scan #3710

6. Make fixes

Manually check suspicious places and fix what is broken with B-Tree index.

[x] Fix sticky pages logic after table compaction https://github.com/ydb-platform/ydb/commit/dda26179eda44bc3694bcfe1e61554baa2f44b72
[x] Keep B-Tree index in memory after compactions https://github.com/ydb-platform/ydb/pull/1165
[x] Fix B-Tree index iterator behaviour with corner pages https://github.com/ydb-platform/ydb/pull/995
[x] Bugfix TSchemeShard::Clear method https://github.com/ydb-platform/ydb/pull/655
[x] Count flat or B-Tree index to index size depending on EnableLocalDBBtreeIndex setting https://github.com/ydb-platform/ydb/pull/653 https://github.com/ydb-platform/ydb/pull/671
[x] Add a possibility to turn off Flat index writing https://github.com/ydb-platform/ydb/pull/2546
[x] Remove flat index usage from most of tests #2547
[x] Fix Cache line head don't want to do fetch but should verify https://github.com/ydb-platform/ydb/pull/3134
[x] ~Limit leaf B-Tree index nodes small enough for skipping them in BuildStats~ Temporary use bigger index resolution to avoid full B-Tree index loading https://github.com/ydb-platform/ydb/pull/3182
[x] https://github.com/ydb-platform/ydb/pull/4969
[x] https://github.com/ydb-platform/ydb/pull/4925

6. Test

Verify that everything works together not dramatically worse than before. Make tests on a slice.

[x] Benchmark TRunIt over B-Tree index https://github.com/ydb-platform/ydb/pull/653
[x] Benchmark Charge over B-Tree index https://github.com/ydb-platform/ydb/pull/1869
[x] Check that EnableLocalDBBtreeIndex and EnableLocalDBFlatIndex settings work
[x] Check large table with a small amount of Shared Cache
[x] Check YCSB benchmark
[x] Check TPC-C benchmark
[x] Check that our changes do not effect Hive balancing (in mem size)

7. Make optimizations

[x] Preload indexes during part switches on followers https://github.com/ydb-platform/ydb/issues/4744
[ ] ❓ Do not allocate in CreateIndexIter https://github.com/ydb-platform/ydb/pull/2514#discussion_r1519564636

8. Release

[x] Wait for a new release tag
[ ] Enable EnableLocalDBBtreeIndex setting on pre-prod clusters (both indexes will be produced) (for some databases first)
[ ] Wait until a new release is installed and won't be rolled back on prod clusters
[ ] Enable EnableLocalDBBtreeIndex setting on prod clusters (for some databases first)
[ ] Enable EnableLocalDBBtreeIndex setting by default
[ ] https://github.com/ydb-platform/ydb/issues/7718
[ ] Disable EnableLocalDBFlatIndex setting
[ ] ❓ Delete flat index support

TPC-C 26-03-24

https://nda.ya.ru/t/n6PUIdLH7595hj B-Tree + Flat, Flat, B-Tree

B-Tree index

2024-03-26 05:41 UTC tpcc_ydb: Loading data done in 1854 seconds
2024-03-26 05:53 UTC tpcc_ydb: Compaction done in 713 seconds
2024-03-26 06:06 UTC tpcc_ydb: Built index in 747 seconds

tpmC: 202416

Flat index

2024-03-26 01:03 UTC tpcc_ydb: Loading data done in 1854 seconds
2024-03-26 01:13 UTC tpcc_ydb: Compaction done in 556 seconds
2024-03-26 01:24 UTC tpcc_ydb: Built index in 683 seconds

tpmC: 202205

YCSB 26-03-24

https://nda.ya.ru/t/jHlSyKq3759KBW B-Tree, Flat

B-Tree index

2024-03-26 12:18 UTC zipfian workload a: 345.4K Op/s, latency 99% 133.119 ms: 414848.7K oks, 0 not_found, 448 errors, 1201089 ms run time
2024-03-26 12:39 UTC zipfian workload b: 501.3K Op/s, latency 99% 18.655 ms: 600000.1K oks, 0 not_found, 0 errors, 1200882 ms run time
2024-03-26 13:00 UTC zipfian workload c: 548.6K Op/s, latency 99% 11.159 ms: 600000.1K oks, 0 not_found, 0 errors, 1099871 ms run time
2024-03-26 13:19 UTC zipfian workload f: 261.8K Op/s, latency 99% 19.631 ms: 471487.2K oks, 0 not_found, 259 errors, 1201088 ms run time
2024-03-26 13:40 UTC latest workload d: 61.9K Op/s, latency 99% 832.511 ms: 74310.4K oks, 9.9K not_found, 1.9K errors, 1201119 ms run time
2024-03-26 14:12 UTC zipfian workload e: 65.3K Op/s, latency 99% 99.007 ms: 39999.9K oks, 0 not_found, 128 errors, 612947 ms run time

Flat index

2024-03-26 14:43 UTC zipfian workload a: 354.8K Op/s, latency 99% 28.479 ms: 426058.9K oks, 0 not_found, 344 errors, 1201099 ms run time
2024-03-26 15:04 UTC zipfian workload b: 494.7K Op/s, latency 99% 22.751 ms: 594172.8K oks, 0 not_found, 12 errors, 1201091 ms run time
2024-03-26 15:25 UTC zipfian workload c: 532.5K Op/s, latency 99% 14.367 ms: 600000.1K oks, 0 not_found, 0 errors, 1132455 ms run time
2024-03-26 15:45 UTC zipfian workload f: 260.9K Op/s, latency 99% 23.839 ms: 469954.6K oks, 0 not_found, 135 errors, 1201086 ms run time
2024-03-26 16:06 UTC latest workload d: 66.8K Op/s, latency 99% 577.023 ms: 80221.9K oks, 2.2K not_found, 601 errors, 1201123 ms run time
2024-03-26 16:37 UTC zipfian workload e: 63.6K Op/s, latency 99% 474.367 ms: 39999.8K oks, 0 not_found, 264 errors, 629655 ms run time

TPC-C 30-04-2024, 16K

https://nda.ya.ru/t/KHu9a09t75gmqt

Flat index

2024-04-30 11:38 UTC tpcc_ydb: Loading data done in 1831 seconds
2024-04-30 11:47 UTC tpcc_ydb: Compaction done in 562 seconds
2024-04-30 12:00 UTC tpcc_ydb: Built index in 757 seconds

Result: Total
  Time: 7200 seconds
  Start measure: 1714480513
  Warehouses: 16000
  New orders: 24262986
  tpmC: 202191
  Efficiency: 98.27
  Throughput: 7497
  Goodput: 8721

NewOrder: OK: 24262986, FAILED: 208 (0.0%)
Payment: OK: 23236184, FAILED: 10 (0.0%)
OrderStatus: OK: 2162490, FAILED: 0
Delivery: OK: 2156203, FAILED: 2988 (0.14%)
StockLevel: OK: 2162091, FAILED: 0
NewOrder:
  50%: 320 ms
  90%: 512 ms
  95%: 512 ms
  99%: 1250 ms
  99.9%: 2000 ms
Payment:
  50%: 144 ms
  90%: 224 ms
  95%: 256 ms
  99%: 480 ms
  99.9%: 512 ms
OrderStatus:
  50%: 96 ms
  90%: 192 ms
  95%: 224 ms
  99%: 320 ms
  99.9%: 480 ms
Delivery:
  50%: 1250 ms
  90%: 2500 ms
  95%: 3000 ms
  99%: 4500 ms
  99.9%: 6000 ms
StockLevel:
  50%: 72 ms
  90%: 144 ms
  95%: 192 ms
  99%: 352 ms
  99.9%: 512 ms

B-Tree index

2024-04-30 17:26 UTC tpcc_ydb: Loading data done in 1825 seconds
2024-04-30 17:35 UTC tpcc_ydb: Compaction done in 566 seconds
2024-04-30 17:47 UTC tpcc_ydb: Built index in 725 seconds

Result: Total
  Time: 7200 seconds
  Start measure: 1714501358
  Warehouses: 16000
  New orders: 24303007
  tpmC: 202524
  Efficiency: 98.43
  Throughput: 7508
  Goodput: 8738

NewOrder: OK: 24303007, FAILED: 136 (0.0%)
Payment: OK: 23273075, FAILED: 6 (0.0%)
OrderStatus: OK: 2166365, FAILED: 0
Delivery: OK: 2163922, FAILED: 2020 (0.09%)
StockLevel: OK: 2165109, FAILED: 0
NewOrder:
  50%: 288 ms
  90%: 512 ms
  95%: 512 ms
  99%: 1000 ms
  99.9%: 1500 ms
Payment:
  50%: 128 ms
  90%: 192 ms
  95%: 224 ms
  99%: 416 ms
  99.9%: 512 ms
OrderStatus:
  50%: 88 ms
  90%: 176 ms
  95%: 224 ms
  99%: 288 ms
  99.9%: 384 ms
Delivery:
  50%: 1250 ms
  90%: 2250 ms
  95%: 2750 ms
  99%: 4000 ms
  99.9%: 5500 ms
StockLevel:
  50%: 64 ms
  90%: 128 ms
  95%: 192 ms
  99%: 320 ms
  99.9%: 512 ms

YSCB 30-04-2024

https://nda.ya.ru/t/sMAtUlme75hHrn

Flat index

2024-04-30 22:59 UTC zipfian workload a: 355.1K Op/s, latency 99% 190.079 ms: 426468.1K oks, 0 not_found, 115 errors, 1201085 ms run time

2024-04-30 23:20 UTC zipfian workload b: 490.8K Op/s, latency 99% 18.271 ms: 589394.8K oks, 0 not_found, 0 errors, 1201091 ms run time

2024-04-30 23:41 UTC zipfian workload c: 520.5K Op/s, latency 99% 11.383 ms: 600000.1K oks, 0 not_found, 0 errors, 1153658 ms run time

2024-05-01 00:01 UTC zipfian workload f: 263.6K Op/s, latency 99% 27.871 ms: 474887.7K oks, 0 not_found, 274 errors, 1201085 ms run time

2024-05-01 00:22 UTC latest workload d: 66.6K Op/s, latency 99% 594.431 ms: 80124K oks, 0 not_found, 0 errors, 1204251 ms run time

2024-05-01 00:53 UTC zipfian workload e: 64.4K Op/s, latency 99% 83.263 ms: 40000K oks, 0 not_found, 98 errors, 621939 ms run time

B-Tree index

2024-05-01 04:02 UTC zipfian workload a: 372.8K Op/s, latency 99% 214.527 ms: 447685.1K oks, 0 not_found, 157 errors, 1201088 ms run time
2024-05-01 08:09 UTC zipfian workload a: 363.4K Op/s, latency 99% 202.239 ms: 436449K oks, 0 not_found, 224 errors, 1201078 ms run time

2024-05-01 04:23 UTC zipfian workload b: 499.7K Op/s, latency 99% 16.015 ms: 600000.1K oks, 0 not_found, 0 errors, 1200871 ms run time
2024-05-01 08:30 UTC zipfian workload b: 496.7K Op/s, latency 99% 16.199 ms: 596508.7K oks, 0 not_found, 1 errors, 1201091 ms run time

2024-05-01 04:44 UTC zipfian workload c: 519.7K Op/s, latency 99% 11.415 ms: 600000.1K oks, 0 not_found, 0 errors, 1157306 ms run time
2024-05-01 08:52 UTC zipfian workload c: 517.9K Op/s, latency 99% 11.263 ms: 600000.1K oks, 0 not_found, 0 errors, 1161620 ms run time

2024-05-01 05:04 UTC zipfian workload f: 268.6K Op/s, latency 99% 27.999 ms: 483795.6K oks, 0 not_found, 421 errors, 1201087 ms run time
2024-05-01 09:12 UTC zipfian workload f: 265.3K Op/s, latency 99% 50.847 ms: 477805K oks, 0 not_found, 734 errors, 1201083 ms run time

2024-05-01 05:25 UTC latest workload d: 63.2K Op/s, latency 99% 635.903 ms: 75794.2K oks, 0 not_found, 0 errors, 1201124 ms run time
2024-05-01 09:33 UTC latest workload d: 64.2K Op/s, latency 99% 647.679 ms: 77055.7K oks, 0 not_found, 0 errors, 1201127 ms run time

2024-05-01 05:56 UTC zipfian workload e: 65.2K Op/s, latency 99% 145.407 ms: 40000K oks, 0 not_found, 7 errors, 614401 ms run time
2024-05-01 10:03 UTC zipfian workload e: 64.2K Op/s, latency 99% 77.759 ms: 40000K oks, 0 not_found, 25 errors, 623347 ms run time

TPC-C 01-05-2024, 12K

https://nda.ya.ru/t/E-eHq6GZ75hyB5

Flat index

2024-05-01 13:17 UTC tpcc_ydb: Loading data done in 1405 seconds
2024-05-01 13:24 UTC tpcc_ydb: Compaction done in 411 seconds
2024-05-01 13:33 UTC tpcc_ydb: Built index in 555 seconds

Result: Total
  Time: 7200 seconds
  Start measure: 1714572511
  Warehouses: 12000
  New orders: 18355322
  tpmC: 152959
  Efficiency: 99.12
  Throughput: 5670
  Goodput: 6601

NewOrder: OK: 18355322, FAILED: 230 (0.0%)
Payment: OK: 17574233, FAILED: 1 (0.0%)
OrderStatus: OK: 1635884, FAILED: 0
Delivery: OK: 1636733, FAILED: 54 (0.0%)
StockLevel: OK: 1635779, FAILED: 2 (0.0%)
NewOrder:
  50%: 128 ms
  90%: 288 ms
  95%: 384 ms
  99%: 512 ms
  99.9%: 1000 ms
Payment:
  50%: 48 ms
  90%: 112 ms
  95%: 160 ms
  99%: 256 ms
  99.9%: 416 ms
OrderStatus:
  50%: 40 ms
  90%: 144 ms
  95%: 176 ms
  99%: 256 ms
  99.9%: 384 ms
Delivery:
  50%: 512 ms
  90%: 512 ms
  95%: 1000 ms
  99%: 1500 ms
  99.9%: 2250 ms
StockLevel:
  50%: 32 ms
  90%: 176 ms
  95%: 256 ms
  99%: 448 ms
  99.9%: 512 ms

B-Tree index

2024-05-01 16:41 UTC tpcc_ydb: Loading data done in 1369 seconds
2024-05-01 16:49 UTC tpcc_ydb: Compaction done in 441 seconds
2024-05-01 16:57 UTC tpcc_ydb: Built index in 513 seconds

Result: Total
  Time: 7200 seconds
  Start measure: 1714584764
  Warehouses: 12000
  New orders: 18358584
  tpmC: 152986
  Efficiency: 99.14
  Throughput: 5669
  Goodput: 6600

NewOrder: OK: 18358584, FAILED: 67 (0.0%)
Payment: OK: 17568479, FAILED: 8 (0.0%)
OrderStatus: OK: 1633702, FAILED: 0
Delivery: OK: 1634485, FAILED: 77 (0.0%)
StockLevel: OK: 1635871, FAILED: 0
NewOrder:
  50%: 144 ms
  90%: 288 ms
  95%: 384 ms
  99%: 512 ms
  99.9%: 1000 ms
Payment:
  50%: 48 ms
  90%: 112 ms
  95%: 160 ms
  99%: 256 ms
  99.9%: 448 ms
OrderStatus:
  50%: 40 ms
  90%: 144 ms
  95%: 192 ms
  99%: 256 ms
  99.9%: 416 ms
Delivery:
  50%: 512 ms
  90%: 512 ms
  95%: 1000 ms
  99%: 1500 ms
  99.9%: 2250 ms
StockLevel:
  50%: 32 ms
  90%: 192 ms
  95%: 288 ms
  99%: 480 ms
  99.9%: 512 ms

TPC-C 02-05-2024, 12K, 54GB cache

https://nda.ya.ru/t/oyxKo3hD75iN9U

Flat index

2024-05-02 08:57 UTC tpcc_ydb: Loading data done in 1478 seconds
2024-05-02 09:05 UTC tpcc_ydb: Compaction done in 443 seconds
2024-05-02 09:14 UTC tpcc_ydb: Built index in 566 seconds

Result: Total
  Time: 7200 seconds
  Start measure: 1714643384
  Warehouses: 12000
  New orders: 18178342
  tpmC: 151485
  Efficiency: 98.16
  Throughput: 5616
  Goodput: 6544

NewOrder: OK: 18178342, FAILED: 1617 (0.01%)
Payment: OK: 17405059, FAILED: 25 (0.0%)
OrderStatus: OK: 1621301, FAILED: 15 (0.0%)
Delivery: OK: 1620295, FAILED: 1185 (0.07%)
StockLevel: OK: 1618387, FAILED: 2 (0.0%)
NewOrder:
  50%: 352 ms
  90%: 512 ms
  95%: 1000 ms
  99%: 1500 ms
  99.9%: 3000 ms
Payment:
  50%: 96 ms
  90%: 224 ms
  95%: 288 ms
  99%: 448 ms
  99.9%: 512 ms
OrderStatus:
  50%: 128 ms
  90%: 288 ms
  95%: 352 ms
  99%: 512 ms
  99.9%: 512 ms
Delivery:
  50%: 1250 ms
  90%: 2500 ms
  95%: 3000 ms
  99%: 4000 ms
  99.9%: 5750 ms
StockLevel:
  50%: 224 ms
  90%: 512 ms
  95%: 512 ms
  99%: 1250 ms
  99.9%: 2000 ms

B-Tree index

2024-05-02 13:18 UTC tpcc_ydb: Loading data done in 1431 seconds
2024-05-02 13:26 UTC tpcc_ydb: Compaction done in 455 seconds
2024-05-02 13:34 UTC tpcc_ydb: Built index in 534 seconds

Result: Total
  Time: 7200 seconds
  Start measure: 1714659003
  Warehouses: 12000
  New orders: 18172437
  tpmC: 151435
  Efficiency: 98.13
  Throughput: 5613
  Goodput: 6540

NewOrder: OK: 18172437, FAILED: 564 (0.0%)
Payment: OK: 17402606, FAILED: 18 (0.0%)
OrderStatus: OK: 1620966, FAILED: 0
Delivery: OK: 1616072, FAILED: 1003 (0.06%)
StockLevel: OK: 1618266, FAILED: 0
NewOrder:
  50%: 352 ms
  90%: 512 ms
  95%: 1000 ms
  99%: 1500 ms
  99.9%: 2750 ms
Payment:
  50%: 96 ms
  90%: 224 ms
  95%: 288 ms
  99%: 480 ms
  99.9%: 512 ms
OrderStatus:
  50%: 144 ms
  90%: 320 ms
  95%: 384 ms
  99%: 512 ms
  99.9%: 512 ms
Delivery:
  50%: 1250 ms
  90%: 2500 ms
  95%: 3000 ms
  99%: 4250 ms
  99.9%: 5750 ms
StockLevel:
  50%: 256 ms
  90%: 512 ms
  95%: 1000 ms
  99%: 1250 ms
  99.9%: 2000 ms

TPC-C 02-05-2024, 12K, 18GB cache

https://nda.ya.ru/t/MYwKhca_75iuBR

Flat index

2024-05-02 19:10 UTC tpcc_ydb: Loading data done in 1501 seconds
2024-05-02 19:18 UTC tpcc_ydb: Compaction done in 450 seconds
2024-05-02 19:27 UTC tpcc_ydb: Built index in 555 seconds

Result: Total
  Time: 7200 seconds
  Start measure: 1714680150
  Warehouses: 12000
  New orders: 13898018
  tpmC: 115815
  Efficiency: 75.05
  Throughput: 4328
  Goodput: 5228

NewOrder: OK: 13898018, FAILED: 105094 (0.75%)
Payment: OK: 13374895, FAILED: 48410 (0.36%)
OrderStatus: OK: 1245056, FAILED: 3990 (0.32%)
Delivery: OK: 1215879, FAILED: 32209 (2.58%)
StockLevel: OK: 1244627, FAILED: 3817 (0.31%)
NewOrder:
  50%: 9000 ms
  90%: 15000 ms
  95%: 20000 ms
  99%: 20000 ms
  99.9%: 20000 ms
Payment:
  50%: 2750 ms
  90%: 4000 ms
  95%: 5250 ms
  99%: 9000 ms
  99.9%: 15000 ms
OrderStatus:
  50%: 3000 ms
  90%: 4000 ms
  95%: 5250 ms
  99%: 8000 ms
  99.9%: 15000 ms
Delivery:
  50%: 9000 ms
  90%: 20000 ms
  95%: 20000 ms
  99%: 20000 ms
  99.9%: 20000 ms
StockLevel:
  50%: 3750 ms
  90%: 5500 ms
  95%: 6000 ms
  99%: 9000 ms
  99.9%: 15000 ms

B-Tree index

2024-05-02 23:01 UTC tpcc_ydb: Loading data done in 1492 seconds
2024-05-02 23:09 UTC tpcc_ydb: Compaction done in 451 seconds
2024-05-02 23:17 UTC tpcc_ydb: Built index in 491 seconds

Result: Total
  Time: 7200 seconds
  Start measure: 1714693941
  Warehouses: 12000
  New orders: 14137163
  tpmC: 117808
  Efficiency: 76.34
  Throughput: 4400
  Goodput: 5299

NewOrder: OK: 14137163, FAILED: 90520 (0.64%)
Payment: OK: 13614464, FAILED: 37143 (0.27%)
OrderStatus: OK: 1268181, FAILED: 2875 (0.23%)
Delivery: OK: 1247545, FAILED: 18966 (1.5%)
StockLevel: OK: 1266308, FAILED: 2891 (0.23%)
NewOrder:
  50%: 8000 ms
  90%: 14000 ms
  95%: 20000 ms
  99%: 20000 ms
  99.9%: 20000 ms
Payment:
  50%: 2500 ms
  90%: 4000 ms
  95%: 5250 ms
  99%: 8000 ms
  99.9%: 15000 ms
OrderStatus:
  50%: 2500 ms
  90%: 4000 ms
  95%: 5250 ms
  99%: 8000 ms
  99.9%: 15000 ms
Delivery:
  50%: 8000 ms
  90%: 15000 ms
  95%: 20000 ms
  99%: 20000 ms
  99.9%: 20000 ms
StockLevel:
  50%: 3250 ms
  90%: 4750 ms
  95%: 5750 ms
  99%: 9000 ms
  99.9%: 15000 ms

TPC-C 03-05-2024, 12K, 2GB cache

https://nda.ya.ru/t/lJkYsNo375joSi

Flat index

2024-05-03 10:52 UTC tpcc_ydb: Loading data done in 1526 seconds
2024-05-03 10:59 UTC tpcc_ydb: Compaction done in 434 seconds
2024-05-03 11:08 UTC tpcc_ydb: Built index in 546 seconds

Result: Total
  Time: 7200 seconds
  Start measure: 1714736605
  Warehouses: 12000
  New orders: 8644037
  tpmC: 72032
  Efficiency: 46.68
  Throughput: 3465
  Goodput: 3453

NewOrder: OK: 8644037, FAILED: 2567753 (22.9%)
Payment: OK: 8832887, FAILED: 1915043 (17.82%)
OrderStatus: OK: 826607, FAILED: 174705 (17.45%)
Delivery: OK: 563554, FAILED: 433429 (43.47%)
StockLevel: OK: 830724, FAILED: 170401 (17.02%)
NewOrder:
  50%: 15000 ms
  90%: 20000 ms
  95%: 20000 ms
  99%: 20000 ms
  99.9%: 20000 ms
Payment:
  50%: 6000 ms
  90%: 14000 ms
  95%: 15000 ms
  99%: 15000 ms
  99.9%: 15000 ms
OrderStatus:
  50%: 6000 ms
  90%: 14000 ms
  95%: 15000 ms
  99%: 15000 ms
  99.9%: 15000 ms
Delivery:
  50%: 15000 ms
  90%: 20000 ms
  95%: 20000 ms
  99%: 20000 ms
  99.9%: 20000 ms
StockLevel:
  50%: 7000 ms
  90%: 14000 ms
  95%: 15000 ms
  99%: 15000 ms
  99.9%: 20000 ms

B-Tree index

2024-05-03 17:04 UTC tpcc_ydb: Loading data done in 1546 seconds
2024-05-03 17:12 UTC tpcc_ydb: Compaction done in 452 seconds
2024-05-03 17:21 UTC tpcc_ydb: Built index in 545 seconds

Result: Total
  Time: 7200 seconds
  Start measure: 1714759001
  Warehouses: 12000
  New orders: 9081395
  tpmC: 75677
  Efficiency: 49.04
  Throughput: 3515
  Goodput: 3611

NewOrder: OK: 9081395, FAILED: 2289951 (20.14%)
Payment: OK: 9290304, FAILED: 1616212 (14.82%)
OrderStatus: OK: 867382, FAILED: 147384 (14.52%)
Delivery: OK: 638883, FAILED: 373551 (36.9%)
StockLevel: OK: 870645, FAILED: 144406 (14.23%)
NewOrder:
  50%: 15000 ms
  90%: 20000 ms
  95%: 20000 ms
  99%: 20000 ms
  99.9%: 20000 ms
Payment:
  50%: 6000 ms
  90%: 13000 ms
  95%: 15000 ms
  99%: 15000 ms
  99.9%: 15000 ms
OrderStatus:
  50%: 6000 ms
  90%: 13000 ms
  95%: 15000 ms
  99%: 15000 ms
  99.9%: 15000 ms
Delivery:
  50%: 15000 ms
  90%: 20000 ms
  95%: 20000 ms
  99%: 20000 ms
  99.9%: 20000 ms
StockLevel:
  50%: 6000 ms
  90%: 13000 ms
  95%: 15000 ms
  99%: 15000 ms
  99.9%: 15000 ms

YCSB 04-05-2024, 12K, 2GB cache

https://nda.ya.ru/t/llFWtuzo75kKft

Flat index

2024-05-04 11:33 UTC zipfian workload a: 369.1K Op/s, latency 99% 158.335 ms: 443248.7K oks, 0 not_found, 422 errors, 1201090 ms run time

2024-05-04 11:54 UTC zipfian workload b: 503.5K Op/s, latency 99% 18.815 ms: 600000.1K oks, 0 not_found, 0 errors, 1191820 ms run time

2024-05-04 12:15 UTC zipfian workload c: 527.8K Op/s, latency 99% 11.335 ms: 600000.1K oks, 0 not_found, 0 errors, 1142053 ms run time

2024-05-04 12:35 UTC zipfian workload f: 276K Op/s, latency 99% 32.607 ms: 497156K oks, 0 not_found, 335 errors, 1201087 ms run time

2024-05-04 12:56 UTC latest workload d: 63.6K Op/s, latency 99% 600.063 ms: 76383.2K oks, 0 not_found, 0 errors, 1201122 ms run time

2024-05-04 13:28 UTC zipfian workload e: 65K Op/s, latency 99% 244.735 ms: 40000K oks, 0 not_found, 33 errors, 615966 ms run time

B-Tree index

2024-05-04 15:25 UTC zipfian workload a: 375.7K Op/s, latency 99% 55.167 ms: 451246.9K oks, 0 not_found, 413 errors, 1201087 ms run time

2024-05-04 15:47 UTC zipfian workload b: 500.3K Op/s, latency 99% 17.599 ms: 600000.1K oks, 0 not_found, 2 errors, 1199313 ms run time

2024-05-04 16:08 UTC zipfian workload c: 524.9K Op/s, latency 99% 11.559 ms: 600000.1K oks, 0 not_found, 0 errors, 1145535 ms run time

2024-05-04 16:28 UTC zipfian workload f: 275.4K Op/s, latency 99% 25.727 ms: 496070.9K oks, 0 not_found, 358 errors, 1201091 ms run time

2024-05-04 16:49 UTC latest workload d: 62.5K Op/s, latency 99% 591.359 ms: 75009.4K oks, 0 not_found, 0 errors, 1201149 ms run time

2024-05-04 17:21 UTC zipfian workload e: 62.9K Op/s, latency 99% 85.119 ms: 40000K oks, 0 not_found, 19 errors, 636709 ms run time

ydb-platform / ydb