apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
12.38k stars 3.22k forks source link

[fix] (inverted index ) Fix the incorrect index size during compaction #37232

Closed csun5285 closed 2 months ago

csun5285 commented 3 months ago

Proposed changes

Index compaction didn't update the total size and index size.

Issue Number: close #xxx

doris-robot commented 3 months ago

Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.

csun5285 commented 3 months ago

run buildall

github-actions[bot] commented 3 months ago

clang-tidy review says "All clean, LGTM! :+1:"

doris-robot commented 3 months ago
TPC-H: Total hot run time: 39635 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 34c4d94003133eba5a6bd33f0ba5095188b1487e, data reload: false ------ Round 1 ---------------------------------- q1 17628 4355 4303 4303 q2 2019 194 191 191 q3 10466 1187 1118 1118 q4 10195 798 666 666 q5 7486 2692 2685 2685 q6 221 139 136 136 q7 964 609 610 609 q8 9256 2101 2086 2086 q9 9032 6510 6501 6501 q10 8932 3680 3735 3680 q11 449 237 238 237 q12 487 232 225 225 q13 17760 2950 3026 2950 q14 273 223 221 221 q15 517 497 494 494 q16 504 394 376 376 q17 969 630 649 630 q18 8130 7477 7368 7368 q19 5935 1481 1409 1409 q20 694 323 330 323 q21 4903 3091 3180 3091 q22 388 346 336 336 Total cold run time: 117208 ms Total hot run time: 39635 ms ----- Round 2, with runtime_filter_mode=off ----- q1 4370 4210 4196 4196 q2 364 258 281 258 q3 2977 2864 2851 2851 q4 2013 1699 1718 1699 q5 5669 5541 5493 5493 q6 224 126 128 126 q7 2249 1877 1896 1877 q8 3277 3411 3439 3411 q9 8743 8608 8867 8608 q10 4100 3903 3717 3717 q11 571 513 500 500 q12 818 640 621 621 q13 17154 3143 3202 3143 q14 309 281 289 281 q15 521 482 479 479 q16 500 440 437 437 q17 1810 1545 1498 1498 q18 8047 7894 7819 7819 q19 1837 1618 1641 1618 q20 2187 1899 1911 1899 q21 5190 4901 4829 4829 q22 642 549 562 549 Total cold run time: 73572 ms Total hot run time: 55909 ms ```
doris-robot commented 3 months ago
TPC-DS: Total hot run time: 170839 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 34c4d94003133eba5a6bd33f0ba5095188b1487e, data reload: false query1 927 378 374 374 query2 6438 2481 2303 2303 query3 6637 212 224 212 query4 19218 17459 17265 17265 query5 3675 468 458 458 query6 288 175 160 160 query7 4595 300 312 300 query8 330 297 318 297 query9 8435 2375 2365 2365 query10 579 300 299 299 query11 10551 9912 10090 9912 query12 116 95 81 81 query13 1645 385 370 370 query14 9354 7769 6857 6857 query15 229 177 185 177 query16 7750 266 263 263 query17 1846 534 508 508 query18 1946 268 267 267 query19 188 146 154 146 query20 86 82 80 80 query21 215 134 125 125 query22 4267 4131 4043 4043 query23 33607 33733 33688 33688 query24 10718 2936 2932 2932 query25 607 383 367 367 query26 700 150 154 150 query27 2233 321 324 321 query28 6225 2123 2130 2123 query29 909 634 631 631 query30 254 154 162 154 query31 962 748 748 748 query32 94 52 55 52 query33 665 305 295 295 query34 891 492 487 487 query35 738 639 623 623 query36 1136 1005 968 968 query37 141 79 82 79 query38 2879 2836 2810 2810 query39 918 853 858 853 query40 207 129 127 127 query41 56 56 54 54 query42 149 94 101 94 query43 557 545 542 542 query44 1065 734 720 720 query45 206 169 161 161 query46 1099 717 736 717 query47 1844 1754 1810 1754 query48 359 297 308 297 query49 826 399 412 399 query50 763 377 374 374 query51 6952 6799 6681 6681 query52 102 91 96 91 query53 362 298 285 285 query54 862 444 435 435 query55 76 72 72 72 query56 288 270 263 263 query57 1111 1023 1072 1023 query58 249 246 250 246 query59 3265 3289 3106 3106 query60 309 281 281 281 query61 97 90 126 90 query62 610 442 436 436 query63 325 291 293 291 query64 8532 2242 1709 1709 query65 3149 3106 3098 3098 query66 745 316 326 316 query67 15282 14990 15029 14990 query68 4558 535 540 535 query69 568 411 319 319 query70 1191 1051 1170 1051 query71 380 333 270 270 query72 7096 5424 2737 2737 query73 746 324 322 322 query74 5903 5577 5517 5517 query75 3324 2647 2686 2647 query76 2236 942 927 927 query77 540 305 294 294 query78 10225 9846 9792 9792 query79 2331 514 506 506 query80 920 478 472 472 query81 579 225 218 218 query82 744 106 101 101 query83 262 180 165 165 query84 232 87 82 82 query85 1513 280 267 267 query86 471 326 327 326 query87 3255 3141 3138 3138 query88 3706 2347 2374 2347 query89 480 393 378 378 query90 1750 184 186 184 query91 125 98 103 98 query92 58 51 48 48 query93 2422 497 495 495 query94 1093 189 184 184 query95 408 317 315 315 query96 602 270 263 263 query97 3155 3023 3037 3023 query98 230 210 186 186 query99 1154 837 845 837 Total cold run time: 264696 ms Total hot run time: 170839 ms ```
doris-robot commented 3 months ago
ClickBench: Total hot run time: 30.27 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 34c4d94003133eba5a6bd33f0ba5095188b1487e, data reload: false query1 0.05 0.03 0.04 query2 0.09 0.04 0.04 query3 0.22 0.05 0.05 query4 1.68 0.07 0.07 query5 0.50 0.52 0.49 query6 1.15 0.73 0.72 query7 0.02 0.01 0.02 query8 0.05 0.04 0.04 query9 0.56 0.49 0.49 query10 0.54 0.54 0.54 query11 0.16 0.11 0.11 query12 0.15 0.13 0.12 query13 0.59 0.59 0.60 query14 0.75 0.78 0.76 query15 0.86 0.81 0.81 query16 0.35 0.36 0.38 query17 1.00 1.02 1.00 query18 0.22 0.25 0.25 query19 1.86 1.73 1.75 query20 0.01 0.00 0.00 query21 15.43 0.84 0.74 query22 4.68 7.43 1.45 query23 18.22 1.38 1.33 query24 2.09 0.25 0.22 query25 0.17 0.09 0.09 query26 0.27 0.18 0.17 query27 0.09 0.08 0.08 query28 13.25 1.02 1.00 query29 12.64 3.33 3.28 query30 0.26 0.06 0.05 query31 2.86 0.40 0.38 query32 3.28 0.46 0.47 query33 2.87 2.87 2.94 query34 17.12 4.36 4.44 query35 4.47 4.49 4.50 query36 0.65 0.49 0.48 query37 0.18 0.16 0.15 query38 0.16 0.15 0.15 query39 0.04 0.03 0.04 query40 0.17 0.16 0.15 query41 0.09 0.05 0.05 query42 0.06 0.04 0.05 query43 0.04 0.04 0.04 Total cold run time: 109.9 s Total hot run time: 30.27 s ```
github-actions[bot] commented 3 months ago

clang-tidy review says "All clean, LGTM! :+1:"

github-actions[bot] commented 3 months ago

clang-tidy review says "All clean, LGTM! :+1:"

github-actions[bot] commented 2 months ago

PR approved by at least one committer and no changes requested.

github-actions[bot] commented 2 months ago

PR approved by anyone and no changes requested.