search

apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
12.78k stars 3.29k forks source link

[opt](log) Optimize compaction log to match output version #44375

Open gavinchou opened 4 days ago

gavinchou commented 4 days ago

What problem does this PR solve?

Problem Summary:

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

doris-robot commented 4 days ago

Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?
gavinchou commented 4 days ago

run buildall

github-actions[bot] commented 4 days ago

clang-tidy review says "All clean, LGTM! :+1:"

doris-robot commented 4 days ago
TPC-H: Total hot run time: 39932 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 04d905d781fda60f50e0001065135aa60c758f2d, data reload: false ------ Round 1 ---------------------------------- q1 17599 7486 7261 7261 q2 2051 182 168 168 q3 10618 1160 1187 1160 q4 10245 709 671 671 q5 7615 2882 2721 2721 q6 239 147 148 147 q7 974 629 602 602 q8 9234 1808 1893 1808 q9 6611 6399 6399 6399 q10 6961 2336 2344 2336 q11 459 255 256 255 q12 432 214 210 210 q13 17797 2999 3027 2999 q14 242 216 218 216 q15 564 528 534 528 q16 654 582 591 582 q17 980 536 551 536 q18 7391 6751 6644 6644 q19 1340 1036 1011 1011 q20 472 178 184 178 q21 4215 3191 3333 3191 q22 364 309 321 309 Total cold run time: 107057 ms Total hot run time: 39932 ms ----- Round 2, with runtime_filter_mode=off ----- q1 7244 7243 7218 7218 q2 330 222 230 222 q3 2932 2780 3046 2780 q4 2112 1742 1752 1742 q5 5607 5740 5647 5647 q6 226 142 141 141 q7 2187 1790 1768 1768 q8 3397 3530 3467 3467 q9 8922 8848 8871 8848 q10 3583 3569 3562 3562 q11 605 517 510 510 q12 837 648 624 624 q13 13304 3178 3220 3178 q14 293 267 267 267 q15 570 530 509 509 q16 689 655 639 639 q17 1808 1575 1576 1575 q18 7947 7384 7394 7384 q19 1647 1578 1580 1578 q20 2031 1866 1803 1803 q21 5360 5216 5266 5216 q22 612 561 571 561 Total cold run time: 72243 ms Total hot run time: 59239 ms ```
doris-robot commented 4 days ago

TeamCity be ut coverage result: Function Coverage: 38.03% (9902/26035) Line Coverage: 29.21% (82817/283525) Region Coverage: 28.35% (42549/150086) Branch Coverage: 24.90% (21565/86596) Coverage Report: http://coverage.selectdb-in.cc/coverage/04d905d781fda60f50e0001065135aa60c758f2d_04d905d781fda60f50e0001065135aa60c758f2d/report/index.html

doris-robot commented 4 days ago
TPC-DS: Total hot run time: 191283 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 04d905d781fda60f50e0001065135aa60c758f2d, data reload: false query1 976 384 383 383 query2 6521 2142 2133 2133 query3 6710 221 224 221 query4 34175 23612 23647 23612 query5 4341 483 465 465 query6 275 176 167 167 query7 4612 295 302 295 query8 286 210 228 210 query9 9505 2662 2650 2650 query10 478 254 238 238 query11 18353 15381 15269 15269 query12 160 100 111 100 query13 1653 434 410 410 query14 9068 7375 6653 6653 query15 272 180 188 180 query16 8390 429 455 429 query17 1658 585 568 568 query18 2143 306 297 297 query19 258 156 154 154 query20 119 114 115 114 query21 203 103 106 103 query22 4656 4280 4451 4280 query23 35068 34042 34045 34042 query24 11418 2462 2444 2444 query25 670 387 414 387 query26 1828 160 151 151 query27 2903 275 276 275 query28 8336 2439 2437 2437 query29 1015 401 402 401 query30 300 154 148 148 query31 1016 815 848 815 query32 101 62 64 62 query33 763 283 278 278 query34 985 496 527 496 query35 854 726 725 725 query36 1129 957 954 954 query37 194 70 75 70 query38 4365 4260 4232 4232 query39 1502 1426 1439 1426 query40 290 97 97 97 query41 48 44 42 42 query42 104 97 96 96 query43 551 507 512 507 query44 1247 819 814 814 query45 184 165 162 162 query46 1144 682 688 682 query47 1954 1874 1863 1863 query48 407 309 317 309 query49 1291 380 392 380 query50 805 384 388 384 query51 7264 7132 7140 7132 query52 95 93 90 90 query53 250 177 177 177 query54 1158 397 405 397 query55 76 78 78 78 query56 272 230 244 230 query57 1342 1154 1144 1144 query58 242 231 219 219 query59 3343 3348 3118 3118 query60 277 252 247 247 query61 110 108 111 108 query62 917 668 690 668 query63 210 187 189 187 query64 5197 660 628 628 query65 3285 3270 3268 3268 query66 1432 311 315 311 query67 15892 15706 15696 15696 query68 4829 552 559 552 query69 429 246 258 246 query70 1185 1144 1147 1144 query71 343 253 254 253 query72 6407 4034 4029 4029 query73 761 362 360 360 query74 10560 9078 9109 9078 query75 3424 2755 2672 2672 query76 3018 1104 1023 1023 query77 393 279 271 271 query78 10423 9332 9424 9332 query79 2493 610 610 610 query80 1192 432 445 432 query81 544 230 227 227 query82 654 121 118 118 query83 242 147 154 147 query84 235 69 73 69 query85 1781 306 296 296 query86 478 299 262 262 query87 4731 4471 4737 4471 query88 4084 2251 2212 2212 query89 415 300 294 294 query90 2028 197 188 188 query91 142 106 102 102 query92 65 55 50 50 query93 1677 542 549 542 query94 902 284 274 274 query95 358 248 248 248 query96 617 269 275 269 query97 2910 2689 2719 2689 query98 216 201 196 196 query99 1571 1320 1319 1319 Total cold run time: 305753 ms Total hot run time: 191283 ms ```
doris-robot commented 4 days ago
ClickBench: Total hot run time: 32.58 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 04d905d781fda60f50e0001065135aa60c758f2d, data reload: false query1 0.04 0.03 0.03 query2 0.07 0.03 0.03 query3 0.23 0.07 0.07 query4 1.62 0.10 0.11 query5 0.43 0.40 0.42 query6 1.16 0.65 0.65 query7 0.02 0.01 0.01 query8 0.05 0.04 0.03 query9 0.58 0.51 0.51 query10 0.55 0.55 0.56 query11 0.14 0.10 0.10 query12 0.14 0.12 0.11 query13 0.61 0.61 0.60 query14 2.77 2.72 2.73 query15 0.91 0.81 0.83 query16 0.39 0.38 0.39 query17 1.06 1.01 1.05 query18 0.23 0.22 0.22 query19 1.98 1.81 1.95 query20 0.01 0.01 0.01 query21 15.38 0.62 0.61 query22 2.48 2.36 1.61 query23 17.06 0.82 0.77 query24 2.93 1.26 2.08 query25 0.26 0.18 0.05 query26 0.54 0.13 0.13 query27 0.05 0.04 0.04 query28 9.76 1.10 1.07 query29 12.54 3.29 3.24 query30 0.25 0.06 0.06 query31 2.88 0.38 0.37 query32 3.30 0.46 0.47 query33 3.09 3.07 3.02 query34 16.89 4.45 4.44 query35 4.49 4.58 4.52 query36 0.64 0.49 0.50 query37 0.09 0.06 0.05 query38 0.05 0.03 0.03 query39 0.04 0.03 0.02 query40 0.17 0.13 0.12 query41 0.07 0.02 0.02 query42 0.04 0.02 0.02 query43 0.04 0.04 0.03 Total cold run time: 106.03 s Total hot run time: 32.58 s ```
github-actions[bot] commented 3 days ago

clang-tidy review says "All clean, LGTM! :+1:"

github-actions[bot] commented 3 days ago

PR approved by anyone and no changes requested.

dataroaring commented 2 days ago

run buildall

github-actions[bot] commented 2 days ago

PR approved by at least one committer and no changes requested.

doris-robot commented 2 days ago
TPC-H: Total hot run time: 39750 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit b0ef8512f34c9145d69a9d426ba6d990d9fc60fe, data reload: false ------ Round 1 ---------------------------------- q1 17612 7768 7249 7249 q2 2047 178 182 178 q3 10862 1111 1189 1111 q4 10266 704 723 704 q5 7613 2684 2762 2684 q6 238 151 148 148 q7 985 639 604 604 q8 9241 1909 1935 1909 q9 6646 6376 6434 6376 q10 7004 2272 2278 2272 q11 471 266 262 262 q12 411 220 221 220 q13 17788 2988 3053 2988 q14 249 207 209 207 q15 575 528 518 518 q16 687 603 609 603 q17 1008 561 569 561 q18 7254 6637 6722 6637 q19 1336 1048 1023 1023 q20 467 191 189 189 q21 4353 3299 2994 2994 q22 389 322 313 313 Total cold run time: 107502 ms Total hot run time: 39750 ms ----- Round 2, with runtime_filter_mode=off ----- q1 7231 7211 7209 7209 q2 320 227 230 227 q3 2944 2832 2955 2832 q4 2038 1810 1842 1810 q5 5710 5704 5698 5698 q6 236 149 152 149 q7 2271 1859 1827 1827 q8 3444 3553 3555 3553 q9 8792 8877 8897 8877 q10 3598 3519 3558 3519 q11 598 513 506 506 q12 812 618 632 618 q13 10733 3289 3256 3256 q14 302 284 270 270 q15 577 523 531 523 q16 704 653 655 653 q17 1891 1635 1653 1635 q18 8380 7774 7718 7718 q19 2746 1526 1473 1473 q20 2135 1901 1877 1877 q21 5487 5336 5459 5336 q22 665 593 566 566 Total cold run time: 71614 ms Total hot run time: 60132 ms ```
doris-robot commented 2 days ago

TeamCity be ut coverage result: Function Coverage: 38.32% (9981/26049) Line Coverage: 29.43% (83523/283833) Region Coverage: 28.59% (42992/150360) Branch Coverage: 25.17% (21837/86746) Coverage Report: http://coverage.selectdb-in.cc/coverage/b0ef8512f34c9145d69a9d426ba6d990d9fc60fe_b0ef8512f34c9145d69a9d426ba6d990d9fc60fe/report/index.html

doris-robot commented 2 days ago
TPC-DS: Total hot run time: 196845 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit b0ef8512f34c9145d69a9d426ba6d990d9fc60fe, data reload: false query1 1238 927 938 927 query2 6236 2127 2044 2044 query3 10775 4038 4015 4015 query4 67373 29255 23562 23562 query5 4865 459 474 459 query6 428 185 194 185 query7 5624 301 298 298 query8 321 231 228 228 query9 9312 2780 2750 2750 query10 436 260 254 254 query11 17277 15319 15868 15319 query12 155 104 104 104 query13 1625 440 428 428 query14 10771 7447 7060 7060 query15 209 178 196 178 query16 7126 453 404 404 query17 1370 592 608 592 query18 1796 310 305 305 query19 203 172 161 161 query20 136 122 113 113 query21 196 104 109 104 query22 4860 4661 4382 4382 query23 34837 34326 34400 34326 query24 5527 2625 2506 2506 query25 495 406 406 406 query26 638 145 140 140 query27 1807 288 295 288 query28 4559 2506 2501 2501 query29 679 424 413 413 query30 221 156 148 148 query31 1031 790 833 790 query32 69 54 57 54 query33 461 299 288 288 query34 927 518 515 515 query35 841 721 713 713 query36 1083 956 995 956 query37 124 76 78 76 query38 4467 4473 4468 4468 query39 1544 1536 1456 1456 query40 214 99 98 98 query41 67 45 42 42 query42 110 101 102 101 query43 525 488 502 488 query44 1153 845 829 829 query45 194 164 172 164 query46 1126 703 690 690 query47 2041 1890 1899 1890 query48 421 314 345 314 query49 732 421 429 421 query50 826 415 395 395 query51 7578 7349 7122 7122 query52 99 90 88 88 query53 248 178 178 178 query54 520 409 426 409 query55 83 80 77 77 query56 236 242 234 234 query57 1302 1157 1169 1157 query58 225 220 219 219 query59 3231 3021 2978 2978 query60 269 244 247 244 query61 110 108 107 107 query62 810 664 656 656 query63 213 190 180 180 query64 1357 681 681 681 query65 3237 3211 3192 3192 query66 717 304 303 303 query67 15939 15704 15921 15704 query68 4021 570 598 570 query69 423 256 266 256 query70 1215 1092 1144 1092 query71 365 256 262 256 query72 6382 4106 4084 4084 query73 756 359 362 359 query74 9936 8999 8949 8949 query75 3420 2649 2673 2649 query76 1806 1077 1101 1077 query77 473 278 287 278 query78 10496 9385 9356 9356 query79 1456 586 591 586 query80 874 436 441 436 query81 506 244 230 230 query82 1252 120 115 115 query83 260 152 169 152 query84 276 69 66 66 query85 890 323 296 296 query86 342 284 293 284 query87 4792 4576 4518 4518 query88 3447 2263 2229 2229 query89 411 289 295 289 query90 2063 187 185 185 query91 130 105 104 104 query92 69 50 49 49 query93 1860 543 542 542 query94 859 276 249 249 query95 351 252 253 252 query96 631 273 294 273 query97 2814 2723 2672 2672 query98 227 215 212 212 query99 1581 1303 1314 1303 Total cold run time: 319609 ms Total hot run time: 196845 ms ```
doris-robot commented 2 days ago
ClickBench: Total hot run time: 32.38 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit b0ef8512f34c9145d69a9d426ba6d990d9fc60fe, data reload: false query1 0.04 0.04 0.03 query2 0.06 0.03 0.03 query3 0.24 0.07 0.07 query4 1.61 0.10 0.11 query5 0.41 0.40 0.43 query6 1.15 0.66 0.65 query7 0.02 0.02 0.02 query8 0.04 0.03 0.03 query9 0.57 0.51 0.51 query10 0.55 0.56 0.56 query11 0.14 0.10 0.10 query12 0.14 0.11 0.11 query13 0.61 0.60 0.60 query14 2.71 2.76 2.71 query15 0.89 0.82 0.84 query16 0.39 0.40 0.39 query17 1.07 1.08 1.01 query18 0.23 0.21 0.22 query19 1.96 1.91 2.04 query20 0.01 0.01 0.01 query21 15.36 0.62 0.58 query22 2.89 2.03 2.33 query23 17.17 0.90 0.84 query24 2.83 0.46 0.98 query25 0.30 0.09 0.11 query26 0.40 0.14 0.14 query27 0.04 0.04 0.04 query28 11.43 1.10 1.07 query29 12.57 3.25 3.24 query30 0.27 0.07 0.06 query31 2.86 0.38 0.39 query32 3.28 0.46 0.47 query33 2.98 3.11 3.02 query34 16.92 4.43 4.44 query35 4.46 4.45 4.49 query36 0.66 0.49 0.50 query37 0.09 0.07 0.06 query38 0.05 0.04 0.05 query39 0.03 0.02 0.02 query40 0.17 0.13 0.13 query41 0.07 0.03 0.03 query42 0.04 0.02 0.02 query43 0.03 0.03 0.04 Total cold run time: 107.74 s Total hot run time: 32.38 s ```