search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
8.69k stars 1.04k forks source link

Implement log multiplexing #3549

Open ddorian opened 4 years ago

ddorian commented 4 years ago

Jira Link: DB-1984 We can end up with 1k+ tablets worth of logs, min 2 files, 64MB each at peak, leaves a lot of space and time spent on bootstrap. It also converts sequential-append log to a random one (when multiple logs are fsyncing to disk).

example: rocksdb column families does https://github.com/facebook/rocksdb/wiki/Column-Families#implementation along with complex wal

Analytics

ddorian commented 4 years ago

Quick example on my 12vcore nvme laptop.

durable_wal_write=1 (see when it drops to 280qps):

./bin/yb-ctl start --ysql_num_shards_per_tserver 64 --tserver_flags="yb_num_shards_per_tserver=64,durable_wal_write=1"

java -jar yb-sample-apps.jar --workload CassandraKeyValue --nodes 127.0.0.1:9042 --num_threads_read 0 --num_threads_write 12

5954 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 5446.86 ops/sec (2.08 ms/op), 27270 total ops  |  Uptime: 5008 ms | maxWrittenKey: 27254 | maxGeneratedKey: 27281 | 
10955 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 5636.21 ops/sec (2.13 ms/op), 55455 total ops  |  Uptime: 10009 ms | maxWrittenKey: 55455 | maxGeneratedKey: 55469 | 
15955 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 5042.77 ops/sec (2.38 ms/op), 80671 total ops  |  Uptime: 15009 ms | maxWrittenKey: 80668 | maxGeneratedKey: 80683 | 
20955 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 5439.42 ops/sec (2.20 ms/op), 107870 total ops  |  Uptime: 20009 ms | maxWrittenKey: 107867 | maxGeneratedKey: 107881 | 
25956 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 2950.11 ops/sec (4.05 ms/op), 122622 total ops  |  Uptime: 25010 ms | maxWrittenKey: 122621 | maxGeneratedKey: 122633 | 
30957 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 229.17 ops/sec (52.13 ms/op), 123768 total ops  |  Uptime: 30010 ms | maxWrittenKey: 123763 | maxGeneratedKey: 123779 | 
35957 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 222.57 ops/sec (54.01 ms/op), 124881 total ops  |  Uptime: 35011 ms | maxWrittenKey: 124876 | maxGeneratedKey: 124892 | 
40958 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 1129.90 ops/sec (10.69 ms/op), 130531 total ops  |  Uptime: 40012 ms | maxWrittenKey: 130528 | maxGeneratedKey: 130542 | 
45958 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 5850.14 ops/sec (2.05 ms/op), 159784 total ops  |  Uptime: 45012 ms | maxWrittenKey: 159780 | maxGeneratedKey: 159796 | 
50959 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 5468.61 ops/sec (2.19 ms/op), 187130 total ops  |  Uptime: 50013 ms | maxWrittenKey: 187129 | maxGeneratedKey: 187141 | 
55959 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 5478.79 ops/sec (2.19 ms/op), 214527 total ops  |  Uptime: 55013 ms | maxWrittenKey: 214526 | maxGeneratedKey: 214538 | 
60959 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 4926.24 ops/sec (2.43 ms/op), 239160 total ops  |  Uptime: 60013 ms | maxWrittenKey: 239158 | maxGeneratedKey: 239171 | 
65960 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 5177.21 ops/sec (2.32 ms/op), 265048 total ops  |  Uptime: 65014 ms | maxWrittenKey: 265041 | maxGeneratedKey: 265059 | 
70960 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 5423.06 ops/sec (2.21 ms/op), 292165 total ops  |  Uptime: 70014 ms | maxWrittenKey: 292163 | maxGeneratedKey: 292176 | 
75960 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 4968.95 ops/sec (2.41 ms/op), 317011 total ops  |  Uptime: 75014 ms | maxWrittenKey: 317010 | maxGeneratedKey: 317022 | 
80961 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 5763.43 ops/sec (2.08 ms/op), 345830 total ops  |  Uptime: 80015 ms | maxWrittenKey: 345825 | maxGeneratedKey: 345842 | 
85961 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 4931.87 ops/sec (2.43 ms/op), 370491 total ops  |  Uptime: 85015 ms | maxWrittenKey: 370490 | maxGeneratedKey: 370502 | 
90961 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 5340.02 ops/sec (2.25 ms/op), 397193 total ops  |  Uptime: 90015 ms | maxWrittenKey: 397188 | maxGeneratedKey: 397205 | 
95962 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 4797.64 ops/sec (2.50 ms/op), 421183 total ops  |  Uptime: 95016 ms | maxWrittenKey: 421182 | maxGeneratedKey: 421194 | 
100962 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 5747.38 ops/sec (2.09 ms/op), 449922 total ops  |  Uptime: 100016 ms | maxWrittenKey: 449920 | maxGeneratedKey: 449933 | 
105963 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 4869.96 ops/sec (2.46 ms/op), 474274 total ops  |  Uptime: 105017 ms | maxWrittenKey: 474270 | maxGeneratedKey: 474285 | 
110963 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 5215.04 ops/sec (2.30 ms/op), 500352 total ops  |  Uptime: 110017 ms | maxWrittenKey: 500351 | maxGeneratedKey: 500363 | 
115964 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 2582.47 ops/sec (4.64 ms/op), 513266 total ops  |  Uptime: 115018 ms | maxWrittenKey: 513262 | maxGeneratedKey: 513277 | 
120965 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 266.76 ops/sec (44.75 ms/op), 514600 total ops  |  Uptime: 120019 ms | maxWrittenKey: 514596 | maxGeneratedKey: 514611 | 
125965 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 810.09 ops/sec (14.91 ms/op), 518651 total ops  |  Uptime: 125019 ms | maxWrittenKey: 518650 | maxGeneratedKey: 518667 | 
130966 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 5414.29 ops/sec (2.22 ms/op), 545725 total ops  |  Uptime: 130020 ms | maxWrittenKey: 545716 | maxGeneratedKey: 545736 | 
135966 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 5398.75 ops/sec (2.22 ms/op), 572722 total ops  |  Uptime: 135020 ms | maxWrittenKey: 572716 | maxGeneratedKey: 572733 |

durable_wal_write=0

./bin/yb-ctl start --ysql_num_shards_per_tserver 64 --tserver_flags="yb_num_shards_per_tserver=64,durable_wal_write=0"

java -jar yb-sample-apps.jar --workload CassandraKeyValue --nodes 127.0.0.1:9042 --num_threads_read 0 --num_threads_write 12

6021 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 19969.62 ops/sec (0.57 ms/op), 99879 total ops  |  Uptime: 5002 ms | maxWrittenKey: 99872 | maxGeneratedKey: 99898 | 
11022 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 24995.42 ops/sec (0.48 ms/op), 224870 total ops  |  Uptime: 10003 ms | maxWrittenKey: 224868 | maxGeneratedKey: 224884 | 
16022 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 25202.52 ops/sec (0.48 ms/op), 350893 total ops  |  Uptime: 15003 ms | maxWrittenKey: 350893 | maxGeneratedKey: 350906 | 
21022 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 24146.82 ops/sec (0.50 ms/op), 471634 total ops  |  Uptime: 20003 ms | maxWrittenKey: 471630 | maxGeneratedKey: 471649 | 
26023 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 23546.96 ops/sec (0.51 ms/op), 589375 total ops  |  Uptime: 25004 ms | maxWrittenKey: 589360 | maxGeneratedKey: 589389 | 
31023 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 23310.41 ops/sec (0.51 ms/op), 705934 total ops  |  Uptime: 30004 ms | maxWrittenKey: 705925 | maxGeneratedKey: 705946 | 
36023 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 23508.24 ops/sec (0.51 ms/op), 823483 total ops  |  Uptime: 35004 ms | maxWrittenKey: 823479 | maxGeneratedKey: 823494 | 
41024 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 23536.72 ops/sec (0.51 ms/op), 941176 total ops  |  Uptime: 40005 ms | maxWrittenKey: 941177 | maxGeneratedKey: 941189 | 
46024 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 23445.28 ops/sec (0.51 ms/op), 1058409 total ops  |  Uptime: 45005 ms | maxWrittenKey: 999999 | maxGeneratedKey: 1000013 | 
51024 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 23833.87 ops/sec (0.50 ms/op), 1177586 total ops  |  Uptime: 50005 ms | maxWrittenKey: 999999 | maxGeneratedKey: 1000013 | 
56025 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 23718.05 ops/sec (0.51 ms/op), 1296183 total ops  |  Uptime: 55006 ms | maxWrittenKey: 999999 | maxGeneratedKey: 1000013 | 
61025 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 23740.36 ops/sec (0.51 ms/op), 1414892 total ops  |  Uptime: 60006 ms | maxWrittenKey: 999999 | maxGeneratedKey: 1000013 | 
66025 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 23756.07 ops/sec (0.50 ms/op), 1533680 total ops  |  Uptime: 65006 ms | maxWrittenKey: 999999 | maxGeneratedKey: 1000013 | 
71026 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 23677.73 ops/sec (0.51 ms/op), 1652076 total ops  |  Uptime: 70007 ms | maxWrittenKey: 999999 | maxGeneratedKey: 1000013 | 
76026 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 23669.41 ops/sec (0.51 ms/op), 1770432 total ops  |  Uptime: 75007 ms | maxWrittenKey: 999999 | maxGeneratedKey: 1000013 | 
81026 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 23701.02 ops/sec (0.51 ms/op), 1888945 total ops  |  Uptime: 80007 ms | maxWrittenKey: 999999 | maxGeneratedKey: 1000013 | 
86027 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 23617.15 ops/sec (0.51 ms/op), 2007038 total ops  |  Uptime: 85008 ms | maxWrittenKey: 999999 | maxGeneratedKey: 1000013 | 
91029 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 23569.32 ops/sec (0.51 ms/op), 2124944 total ops  |  Uptime: 90010 ms | maxWrittenKey: 999999 | maxGeneratedKey: 1000013 | 
96029 [Thread-1] INFO com.yugabyte.sample.common.metrics.MetricsTracker  - Read: 0.00 ops/sec (0.00 ms/op), 0 total ops  |  Write: 23555.39 ops/sec (0.51 ms/op), 2242728 total ops  |  Uptime: 95010 ms | maxWrittenKey: 999999 | maxGeneratedKey: 1000013 |

The reason the write ops falls may be do to memtable flush or running compactions.

nnsgmsone commented 3 years ago

If the number of raft groups is particularly large, io will also deteriorate. How is it more appropriate to handle the situation where the amount of data is particularly large?

ddorian commented 3 years ago

How is it more appropriate to handle the situation where the amount of data is particularly large?

Note that a tablet can grow very big, 200GB+. And with auto-splitting in production, it'll be easy to start with few tablets and still keep few of them even when your datasets grows very large.