search

apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
12.78k stars 3.29k forks source link

[profile](runtime filter) Add necessary metrics in runtime filter #44367

Closed Gabriel39 closed 3 days ago

Gabriel39 commented 4 days ago

What problem does this PR solve?

Currently, we lost wait time and local merge time for each runtime filter. This PR complete some necessary metrics.

Release note

None

Check List (For Author)

Check List (For Reviewer who merge this PR)

doris-robot commented 4 days ago

Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?
Gabriel39 commented 4 days ago

run buildall

github-actions[bot] commented 4 days ago

clang-tidy review says "All clean, LGTM! :+1:"

doris-robot commented 4 days ago

TeamCity be ut coverage result: Function Coverage: 38.03% (9902/26035) Line Coverage: 29.21% (82808/283520) Region Coverage: 28.35% (42546/150078) Branch Coverage: 24.90% (21559/86588) Coverage Report: http://coverage.selectdb-in.cc/coverage/25e05043d376f3d2bd6e0e3addd9ada0b5168e54_25e05043d376f3d2bd6e0e3addd9ada0b5168e54/report/index.html

doris-robot commented 4 days ago
TPC-H: Total hot run time: 39892 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 25e05043d376f3d2bd6e0e3addd9ada0b5168e54, data reload: false ------ Round 1 ---------------------------------- q1 17629 7506 7324 7324 q2 2052 200 167 167 q3 10545 1120 1187 1120 q4 10582 691 811 691 q5 7627 2737 2728 2728 q6 239 148 150 148 q7 966 629 598 598 q8 9247 1903 1916 1903 q9 6553 6376 6394 6376 q10 6981 2323 2327 2323 q11 472 266 257 257 q12 423 213 215 213 q13 17767 3065 3094 3065 q14 237 212 218 212 q15 579 519 524 519 q16 668 596 612 596 q17 994 558 526 526 q18 7408 6694 6589 6589 q19 1332 992 961 961 q20 478 187 179 179 q21 4010 3098 3086 3086 q22 375 330 311 311 Total cold run time: 107164 ms Total hot run time: 39892 ms ----- Round 2, with runtime_filter_mode=off ----- q1 7204 7228 7211 7211 q2 336 242 232 232 q3 2974 2805 3014 2805 q4 2093 1827 1797 1797 q5 5715 5736 5657 5657 q6 230 142 144 142 q7 2268 1874 1815 1815 q8 3474 3561 3558 3558 q9 8814 8927 8969 8927 q10 3574 3564 3551 3551 q11 620 519 505 505 q12 857 630 609 609 q13 13170 3243 3249 3243 q14 309 281 279 279 q15 570 523 523 523 q16 702 643 656 643 q17 1896 1614 1615 1614 q18 8328 7741 7636 7636 q19 1705 1581 1582 1581 q20 2153 1918 1880 1880 q21 5540 5434 5420 5420 q22 654 539 565 539 Total cold run time: 73186 ms Total hot run time: 60167 ms ```
doris-robot commented 3 days ago
TPC-DS: Total hot run time: 198322 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 25e05043d376f3d2bd6e0e3addd9ada0b5168e54, data reload: false query1 1247 959 919 919 query2 6265 2146 2077 2077 query3 10856 4053 3961 3961 query4 67564 29227 23631 23631 query5 4956 460 449 449 query6 419 186 182 182 query7 5486 314 295 295 query8 302 215 215 215 query9 8317 2688 2670 2670 query10 432 264 242 242 query11 17064 15347 15951 15347 query12 159 103 105 103 query13 1439 453 431 431 query14 10524 7896 7738 7738 query15 217 189 192 189 query16 7124 472 537 472 query17 1117 590 600 590 query18 1836 314 320 314 query19 206 166 170 166 query20 123 112 119 112 query21 210 110 106 106 query22 4657 4507 4572 4507 query23 35044 34682 34633 34633 query24 5488 2547 2550 2547 query25 494 386 389 386 query26 680 163 153 153 query27 1869 292 295 292 query28 4462 2501 2485 2485 query29 692 431 424 424 query30 213 156 149 149 query31 1029 847 859 847 query32 65 56 55 55 query33 430 302 324 302 query34 970 523 549 523 query35 881 750 727 727 query36 1123 968 978 968 query37 123 77 74 74 query38 4576 4423 4505 4423 query39 1535 1491 1473 1473 query40 214 101 98 98 query41 44 42 41 41 query42 113 98 101 98 query43 565 509 531 509 query44 1212 885 842 842 query45 188 173 171 171 query46 1178 759 709 709 query47 2060 1897 1954 1897 query48 421 340 320 320 query49 747 412 394 394 query50 855 413 409 409 query51 7521 7350 7106 7106 query52 95 88 84 84 query53 252 178 175 175 query54 521 420 394 394 query55 79 73 74 73 query56 269 240 235 235 query57 1311 1200 1151 1151 query58 219 210 211 210 query59 3421 3249 3054 3054 query60 268 245 238 238 query61 106 108 114 108 query62 850 670 662 662 query63 213 211 193 193 query64 1471 657 636 636 query65 3308 3224 3302 3224 query66 720 299 302 299 query67 16170 15795 15630 15630 query68 3721 575 558 558 query69 436 255 260 255 query70 1215 1156 1130 1130 query71 380 246 256 246 query72 6366 4067 3979 3979 query73 786 369 360 360 query74 10249 9030 9055 9030 query75 3439 2678 2735 2678 query76 2161 1092 1103 1092 query77 478 275 276 275 query78 10633 9508 9437 9437 query79 2176 591 611 591 query80 1362 424 434 424 query81 548 231 232 231 query82 1309 117 121 117 query83 261 152 146 146 query84 279 75 76 75 query85 1032 309 297 297 query86 412 296 306 296 query87 4824 4637 4583 4583 query88 3755 2244 2181 2181 query89 429 291 300 291 query90 1984 185 195 185 query91 139 103 101 101 query92 68 48 49 48 query93 2878 547 554 547 query94 820 310 291 291 query95 355 255 250 250 query96 630 285 281 281 query97 2832 2711 2714 2711 query98 218 200 198 198 query99 2039 1301 1350 1301 Total cold run time: 322799 ms Total hot run time: 198322 ms ```
Gabriel39 commented 3 days ago

run buildall

doris-robot commented 3 days ago
TPC-H: Total hot run time: 40434 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 5b1f5f5402a5a12ffe038ca768ebb5af95d78e26, data reload: false ------ Round 1 ---------------------------------- q1 17769 7546 7298 7298 q2 2045 180 187 180 q3 10528 1155 1134 1134 q4 10221 807 790 790 q5 7622 2765 2704 2704 q6 242 150 147 147 q7 997 620 616 616 q8 9242 1908 1966 1908 q9 6616 6452 6469 6452 q10 6974 2337 2340 2337 q11 454 269 268 268 q12 440 230 215 215 q13 17761 3036 3081 3036 q14 245 210 208 208 q15 568 521 517 517 q16 679 573 607 573 q17 1020 578 577 577 q18 7436 6699 6681 6681 q19 1329 1040 1060 1040 q20 494 183 184 183 q21 4060 3355 3266 3266 q22 383 304 317 304 Total cold run time: 107125 ms Total hot run time: 40434 ms ----- Round 2, with runtime_filter_mode=off ----- q1 7311 7263 7254 7254 q2 323 238 236 236 q3 2957 2896 3121 2896 q4 2176 1859 1797 1797 q5 5627 5730 5671 5671 q6 221 139 140 139 q7 2177 1778 1811 1778 q8 3442 3547 3551 3547 q9 8937 8927 9105 8927 q10 3616 3575 3570 3570 q11 606 510 508 508 q12 847 643 625 625 q13 13850 3187 3185 3185 q14 305 259 267 259 q15 560 509 506 506 q16 684 622 621 621 q17 1814 1578 1580 1578 q18 7854 7410 7556 7410 q19 1670 1559 1647 1559 q20 2071 1844 1890 1844 q21 5482 5233 5113 5113 q22 637 541 571 541 Total cold run time: 73167 ms Total hot run time: 59564 ms ```
doris-robot commented 3 days ago
TPC-DS: Total hot run time: 191861 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 5b1f5f5402a5a12ffe038ca768ebb5af95d78e26, data reload: false query1 955 403 364 364 query2 6529 2063 1995 1995 query3 6709 209 214 209 query4 34083 23634 23576 23576 query5 4383 456 443 443 query6 274 178 181 178 query7 4624 297 306 297 query8 295 240 225 225 query9 9687 2676 2653 2653 query10 480 260 251 251 query11 18067 15240 15280 15240 query12 148 103 101 101 query13 1651 416 427 416 query14 9500 7175 7362 7175 query15 248 180 187 180 query16 8148 444 448 444 query17 1642 571 561 561 query18 2143 290 283 283 query19 372 155 141 141 query20 119 112 110 110 query21 207 103 98 98 query22 4668 4271 4324 4271 query23 34876 34289 34217 34217 query24 10514 2554 2517 2517 query25 639 375 394 375 query26 1523 150 152 150 query27 2843 301 282 282 query28 8098 2427 2419 2419 query29 963 406 402 402 query30 297 147 154 147 query31 1047 802 798 798 query32 101 57 60 57 query33 771 292 292 292 query34 980 517 534 517 query35 898 707 725 707 query36 1113 946 964 946 query37 142 74 79 74 query38 4374 4304 4218 4218 query39 1467 1417 1454 1417 query40 284 100 100 100 query41 47 45 45 45 query42 112 96 98 96 query43 536 484 488 484 query44 1316 816 807 807 query45 185 165 164 164 query46 1131 693 695 693 query47 1968 1819 1904 1819 query48 427 310 316 310 query49 1276 392 406 392 query50 823 378 391 378 query51 7196 7078 7173 7078 query52 100 94 89 89 query53 255 183 190 183 query54 1378 410 405 405 query55 89 75 83 75 query56 275 254 236 236 query57 1298 1213 1168 1168 query58 260 218 232 218 query59 3122 3015 3014 3014 query60 284 253 275 253 query61 139 130 132 130 query62 895 685 713 685 query63 221 196 198 196 query64 5375 763 731 731 query65 3319 3229 3261 3229 query66 1449 388 331 331 query67 16274 15665 15632 15632 query68 4539 557 576 557 query69 434 264 273 264 query70 1165 1137 1142 1137 query71 331 250 246 246 query72 6487 4185 4009 4009 query73 758 368 370 368 query74 10252 9067 9006 9006 query75 3437 2632 2673 2632 query76 2913 1033 1076 1033 query77 381 279 264 264 query78 10535 9450 9496 9450 query79 2497 595 613 595 query80 1235 427 430 427 query81 541 232 233 232 query82 881 122 114 114 query83 248 150 207 150 query84 237 71 72 71 query85 1342 318 309 309 query86 417 296 302 296 query87 4754 4645 4631 4631 query88 4074 2298 2212 2212 query89 402 305 293 293 query90 2024 190 189 189 query91 139 111 104 104 query92 69 49 51 49 query93 1796 542 558 542 query94 924 286 294 286 query95 360 251 263 251 query96 637 279 289 279 query97 2870 2672 2746 2672 query98 231 219 191 191 query99 1515 1325 1315 1315 Total cold run time: 304037 ms Total hot run time: 191861 ms ```
doris-robot commented 3 days ago
ClickBench: Total hot run time: 33.25 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 5b1f5f5402a5a12ffe038ca768ebb5af95d78e26, data reload: false query1 0.04 0.03 0.02 query2 0.06 0.03 0.03 query3 0.23 0.09 0.07 query4 1.60 0.11 0.11 query5 0.43 0.40 0.41 query6 1.16 0.65 0.65 query7 0.02 0.02 0.01 query8 0.04 0.03 0.03 query9 0.57 0.53 0.51 query10 0.55 0.56 0.55 query11 0.15 0.10 0.10 query12 0.14 0.12 0.11 query13 0.62 0.60 0.60 query14 2.70 2.73 2.75 query15 0.90 0.83 0.84 query16 0.39 0.38 0.39 query17 1.02 1.02 1.07 query18 0.21 0.21 0.20 query19 1.90 1.86 1.93 query20 0.02 0.01 0.02 query21 15.36 0.59 0.58 query22 3.13 2.72 1.93 query23 17.14 1.13 0.84 query24 2.74 1.36 2.27 query25 0.12 0.20 0.23 query26 0.53 0.14 0.13 query27 0.05 0.04 0.04 query28 9.58 1.09 1.08 query29 12.57 3.23 3.24 query30 0.24 0.06 0.06 query31 2.86 0.38 0.37 query32 3.29 0.46 0.46 query33 2.97 3.02 3.17 query34 17.68 4.53 4.49 query35 4.50 4.45 4.51 query36 0.67 0.49 0.49 query37 0.09 0.06 0.06 query38 0.04 0.04 0.03 query39 0.04 0.02 0.02 query40 0.16 0.12 0.13 query41 0.07 0.02 0.03 query42 0.03 0.02 0.02 query43 0.03 0.03 0.03 Total cold run time: 106.64 s Total hot run time: 33.25 s ```
doris-robot commented 3 days ago

TeamCity be ut coverage result: Function Coverage: 38.04% (9902/26033) Line Coverage: 29.21% (82806/283513) Region Coverage: 28.34% (42538/150078) Branch Coverage: 24.91% (21566/86590) Coverage Report: http://coverage.selectdb-in.cc/coverage/5b1f5f5402a5a12ffe038ca768ebb5af95d78e26_5b1f5f5402a5a12ffe038ca768ebb5af95d78e26/report/index.html

github-actions[bot] commented 3 days ago

PR approved by at least one committer and no changes requested.

github-actions[bot] commented 3 days ago

PR approved by anyone and no changes requested.