search

apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
11.8k stars 3.11k forks source link

[improvement](segmentcache) limit segment cache by memory or segment num #37026

Closed dataroaring closed 1 day ago

dataroaring commented 3 days ago

Also enlarge columns per segment.

doris-robot commented 3 days ago

Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.

dataroaring commented 3 days ago

run buildall

github-actions[bot] commented 3 days ago

clang-tidy review says "All clean, LGTM! :+1:"

github-actions[bot] commented 3 days ago

clang-tidy review says "All clean, LGTM! :+1:"

github-actions[bot] commented 3 days ago

PR approved by at least one committer and no changes requested.

github-actions[bot] commented 3 days ago

PR approved by anyone and no changes requested.

dataroaring commented 2 days ago

run buildall

github-actions[bot] commented 2 days ago

clang-tidy review says "All clean, LGTM! :+1:"

dataroaring commented 2 days ago

run buildall

github-actions[bot] commented 2 days ago

clang-tidy review says "All clean, LGTM! :+1:"

doris-robot commented 2 days ago
TPC-H: Total hot run time: 39929 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit b8ee16a18416ede3ceeafc01af509783c039dac7, data reload: false ------ Round 1 ---------------------------------- q1 17632 4368 4257 4257 q2 2019 194 198 194 q3 10464 1164 1211 1164 q4 10191 836 755 755 q5 7504 2675 2672 2672 q6 223 137 136 136 q7 955 595 601 595 q8 9218 2116 2098 2098 q9 8841 6537 6466 6466 q10 8945 3714 3728 3714 q11 453 233 233 233 q12 426 231 220 220 q13 17768 2959 3001 2959 q14 267 220 219 219 q15 524 483 484 483 q16 515 372 374 372 q17 982 672 712 672 q18 8182 7431 7431 7431 q19 7522 1569 1477 1477 q20 657 317 339 317 q21 4964 3155 3917 3155 q22 387 340 340 340 Total cold run time: 118639 ms Total hot run time: 39929 ms ----- Round 2, with runtime_filter_mode=off ----- q1 4450 4206 4226 4206 q2 373 276 267 267 q3 3039 2826 2922 2826 q4 2002 1745 1756 1745 q5 5556 5536 5433 5433 q6 230 130 130 130 q7 2224 1884 1846 1846 q8 3249 3481 3429 3429 q9 8770 8784 8814 8784 q10 4279 3720 3824 3720 q11 589 479 510 479 q12 820 637 633 633 q13 17094 3145 3201 3145 q14 301 260 281 260 q15 538 480 495 480 q16 467 430 406 406 q17 1849 1548 1534 1534 q18 8094 7980 7824 7824 q19 3532 1732 1615 1615 q20 2172 1850 1848 1848 q21 5143 4882 4777 4777 q22 641 536 542 536 Total cold run time: 75412 ms Total hot run time: 55923 ms ```
doris-robot commented 2 days ago
TPC-DS: Total hot run time: 174542 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit b8ee16a18416ede3ceeafc01af509783c039dac7, data reload: false query1 907 380 371 371 query2 6449 2517 2305 2305 query3 6645 204 218 204 query4 19387 17670 17292 17292 query5 3758 495 493 493 query6 253 155 160 155 query7 4584 291 289 289 query8 322 296 297 296 query9 8469 2416 2388 2388 query10 572 314 286 286 query11 10932 10165 10089 10089 query12 114 88 87 87 query13 1644 358 374 358 query14 9298 7775 7582 7582 query15 256 188 195 188 query16 7932 289 275 275 query17 1869 564 547 547 query18 2062 282 277 277 query19 203 154 154 154 query20 86 83 91 83 query21 211 144 136 136 query22 4542 4177 4214 4177 query23 33821 33588 33581 33581 query24 11140 2879 2974 2879 query25 663 409 404 404 query26 775 160 158 158 query27 2355 320 319 319 query28 6564 2129 2127 2127 query29 923 716 628 628 query30 249 162 160 160 query31 996 755 761 755 query32 104 60 55 55 query33 776 326 294 294 query34 965 475 472 472 query35 790 637 641 637 query36 1141 980 986 980 query37 147 82 82 82 query38 2952 2904 2843 2843 query39 898 832 833 832 query40 215 132 130 130 query41 58 55 56 55 query42 110 111 104 104 query43 607 513 532 513 query44 1227 728 742 728 query45 188 164 170 164 query46 1090 699 737 699 query47 1871 1787 1791 1787 query48 380 294 302 294 query49 912 517 419 419 query50 773 377 384 377 query51 6827 6685 6728 6685 query52 110 93 93 93 query53 357 293 287 287 query54 869 454 438 438 query55 73 76 73 73 query56 292 257 256 256 query57 1117 1049 1030 1030 query58 247 234 243 234 query59 3260 3337 3198 3198 query60 302 288 310 288 query61 96 92 94 92 query62 598 429 442 429 query63 311 282 287 282 query64 8754 2251 1732 1732 query65 3135 3079 3186 3079 query66 729 325 314 314 query67 15882 14892 15125 14892 query68 7935 532 543 532 query69 708 398 334 334 query70 1218 1131 1145 1131 query71 562 284 274 274 query72 8592 5184 5535 5184 query73 806 323 317 317 query74 5920 5557 5617 5557 query75 5043 2663 2694 2663 query76 4908 914 916 914 query77 769 298 303 298 query78 10573 9879 9772 9772 query79 8202 522 520 520 query80 1044 483 488 483 query81 545 219 224 219 query82 743 111 110 110 query83 325 175 170 170 query84 265 88 87 87 query85 1310 276 272 272 query86 443 304 311 304 query87 3279 3088 3086 3086 query88 4856 2352 2381 2352 query89 561 391 392 391 query90 2032 189 190 189 query91 132 100 101 100 query92 66 48 50 48 query93 6289 504 492 492 query94 1260 188 192 188 query95 405 323 318 318 query96 617 268 271 268 query97 3236 3036 3015 3015 query98 216 201 191 191 query99 1162 856 853 853 Total cold run time: 290166 ms Total hot run time: 174542 ms ```
doris-robot commented 2 days ago
ClickBench: Total hot run time: 30.88 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit b8ee16a18416ede3ceeafc01af509783c039dac7, data reload: false query1 0.04 0.03 0.03 query2 0.09 0.04 0.04 query3 0.22 0.05 0.06 query4 1.67 0.08 0.07 query5 0.51 0.50 0.50 query6 1.14 0.73 0.72 query7 0.02 0.01 0.01 query8 0.05 0.05 0.05 query9 0.56 0.50 0.51 query10 0.56 0.54 0.54 query11 0.14 0.11 0.12 query12 0.14 0.12 0.12 query13 0.60 0.60 0.59 query14 0.80 0.78 0.77 query15 0.86 0.81 0.82 query16 0.36 0.38 0.33 query17 0.97 0.98 1.03 query18 0.21 0.24 0.22 query19 1.76 1.68 1.65 query20 0.02 0.01 0.01 query21 15.44 0.76 0.66 query22 4.42 6.77 2.39 query23 18.30 1.42 1.16 query24 2.14 0.22 0.22 query25 0.15 0.09 0.08 query26 0.26 0.18 0.19 query27 0.08 0.08 0.09 query28 13.16 1.01 1.00 query29 12.62 3.32 3.28 query30 0.25 0.06 0.06 query31 2.85 0.38 0.39 query32 3.29 0.48 0.49 query33 2.90 2.89 2.90 query34 16.90 4.39 4.40 query35 4.51 4.49 4.50 query36 0.65 0.46 0.47 query37 0.18 0.15 0.16 query38 0.16 0.14 0.15 query39 0.04 0.03 0.04 query40 0.16 0.14 0.14 query41 0.09 0.05 0.04 query42 0.05 0.05 0.04 query43 0.05 0.04 0.04 Total cold run time: 109.37 s Total hot run time: 30.88 s ```
github-actions[bot] commented 1 day ago

PR approved by at least one committer and no changes requested.