search

apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
12.78k stars 3.29k forks source link

[fix](local exchange) Fix unbalance data distribution #44421

Closed Gabriel39 closed 2 days ago

Gabriel39 commented 3 days ago

What problem does this PR solve?

Follow-up for #44137

Release note

None

Check List (For Author)

Check List (For Reviewer who merge this PR)

doris-robot commented 3 days ago

Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?
Gabriel39 commented 3 days ago

run buildall

github-actions[bot] commented 3 days ago

clang-tidy review says "All clean, LGTM! :+1:"

doris-robot commented 3 days ago

TeamCity be ut coverage result: Function Coverage: 38.04% (9902/26032) Line Coverage: 29.21% (82818/283514) Region Coverage: 28.35% (42542/150082) Branch Coverage: 24.91% (21566/86590) Coverage Report: http://coverage.selectdb-in.cc/coverage/1cb462f9bdda38e5f4d0203b382feb30d7eb9d49_1cb462f9bdda38e5f4d0203b382feb30d7eb9d49/report/index.html

doris-robot commented 3 days ago
TPC-H: Total hot run time: 40236 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 1cb462f9bdda38e5f4d0203b382feb30d7eb9d49, data reload: false ------ Round 1 ---------------------------------- q1 17589 7538 7336 7336 q2 2066 179 170 170 q3 10855 1130 1166 1130 q4 10559 765 674 674 q5 7629 2757 2743 2743 q6 248 151 151 151 q7 979 638 627 627 q8 9234 1888 1931 1888 q9 6568 6433 6436 6433 q10 7007 2310 2345 2310 q11 468 265 262 262 q12 419 229 227 227 q13 17774 3046 3061 3046 q14 241 210 218 210 q15 594 545 513 513 q16 664 592 591 591 q17 978 551 521 521 q18 7305 6707 6716 6707 q19 1336 1065 1061 1061 q20 483 186 178 178 q21 4242 3146 3485 3146 q22 374 313 312 312 Total cold run time: 107612 ms Total hot run time: 40236 ms ----- Round 2, with runtime_filter_mode=off ----- q1 7723 7720 7699 7699 q2 323 230 230 230 q3 2902 2965 3038 2965 q4 2127 1865 1857 1857 q5 5598 5666 5672 5666 q6 224 140 144 140 q7 2251 1829 1793 1793 q8 3380 3546 3568 3546 q9 8905 8863 8934 8863 q10 3638 3570 3557 3557 q11 605 503 509 503 q12 837 661 656 656 q13 10787 3250 3275 3250 q14 314 281 288 281 q15 595 517 513 513 q16 690 640 639 639 q17 1845 1643 1620 1620 q18 8428 7774 7568 7568 q19 1714 1599 1630 1599 q20 2122 1857 1972 1857 q21 5673 5609 5524 5524 q22 614 532 564 532 Total cold run time: 71295 ms Total hot run time: 60858 ms ```
doris-robot commented 3 days ago
TPC-DS: Total hot run time: 197842 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 1cb462f9bdda38e5f4d0203b382feb30d7eb9d49, data reload: false query1 1203 946 913 913 query2 6244 2129 2047 2047 query3 10915 4220 4153 4153 query4 67516 27931 23513 23513 query5 4908 460 462 460 query6 411 187 180 180 query7 5546 314 294 294 query8 313 240 219 219 query9 8514 2748 2755 2748 query10 458 252 259 252 query11 17305 15363 15837 15363 query12 155 112 104 104 query13 1457 423 405 405 query14 9962 7797 7575 7575 query15 216 192 189 189 query16 7173 494 530 494 query17 1072 582 577 577 query18 1875 322 318 318 query19 204 157 164 157 query20 119 115 113 113 query21 208 104 108 104 query22 4654 4630 4531 4531 query23 35004 34159 34300 34159 query24 5522 2546 2550 2546 query25 519 383 404 383 query26 681 146 149 146 query27 2099 291 295 291 query28 4802 2479 2481 2479 query29 708 431 429 429 query30 221 145 149 145 query31 995 835 863 835 query32 67 57 57 57 query33 402 295 288 288 query34 936 524 553 524 query35 840 776 772 772 query36 1063 953 980 953 query37 120 74 75 74 query38 4516 4430 4543 4430 query39 1495 1491 1467 1467 query40 202 106 101 101 query41 44 43 43 43 query42 117 103 101 101 query43 549 512 509 509 query44 1170 847 834 834 query45 188 172 169 169 query46 1179 728 725 725 query47 2025 1893 1941 1893 query48 426 325 313 313 query49 738 410 384 384 query50 837 399 407 399 query51 7470 7213 7061 7061 query52 98 89 85 85 query53 263 181 177 177 query54 507 396 397 396 query55 78 73 77 73 query56 266 247 227 227 query57 1300 1182 1152 1152 query58 214 209 213 209 query59 3301 2998 3009 2998 query60 265 251 248 248 query61 112 109 109 109 query62 788 660 689 660 query63 219 183 193 183 query64 1432 650 639 639 query65 3270 3250 3257 3250 query66 724 308 303 303 query67 15982 15661 15681 15661 query68 3858 558 557 557 query69 429 257 258 257 query70 1212 1154 1165 1154 query71 367 244 249 244 query72 6396 4066 3997 3997 query73 782 358 365 358 query74 9483 9061 9064 9061 query75 3391 2706 2662 2662 query76 1802 1052 1084 1052 query77 503 275 286 275 query78 10599 9458 9463 9458 query79 1545 589 577 577 query80 873 463 427 427 query81 503 244 226 226 query82 1262 116 118 116 query83 269 152 154 152 query84 289 71 75 71 query85 898 293 299 293 query86 392 307 303 303 query87 4727 4592 4633 4592 query88 3485 2255 2228 2228 query89 420 292 295 292 query90 2007 191 193 191 query91 135 102 101 101 query92 69 48 50 48 query93 1879 549 545 545 query94 801 293 293 293 query95 361 248 248 248 query96 619 281 271 271 query97 2917 2712 2724 2712 query98 218 193 195 193 query99 1807 1311 1299 1299 Total cold run time: 318480 ms Total hot run time: 197842 ms ```
doris-robot commented 3 days ago
ClickBench: Total hot run time: 32.12 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 1cb462f9bdda38e5f4d0203b382feb30d7eb9d49, data reload: false query1 0.03 0.03 0.03 query2 0.06 0.04 0.02 query3 0.23 0.07 0.07 query4 1.63 0.10 0.11 query5 0.43 0.41 0.42 query6 1.14 0.67 0.67 query7 0.02 0.02 0.02 query8 0.04 0.02 0.03 query9 0.59 0.52 0.51 query10 0.55 0.56 0.54 query11 0.14 0.10 0.11 query12 0.14 0.11 0.12 query13 0.61 0.61 0.59 query14 2.72 2.71 2.81 query15 0.90 0.82 0.82 query16 0.39 0.37 0.38 query17 0.98 1.09 1.05 query18 0.22 0.21 0.22 query19 1.91 1.86 1.95 query20 0.01 0.02 0.01 query21 15.37 0.57 0.59 query22 2.85 2.37 1.81 query23 17.05 0.91 0.92 query24 3.25 0.86 0.19 query25 0.17 0.18 0.10 query26 0.36 0.13 0.13 query27 0.05 0.04 0.04 query28 11.27 1.11 1.06 query29 12.53 3.33 3.34 query30 0.25 0.06 0.06 query31 2.85 0.39 0.38 query32 3.30 0.46 0.46 query33 3.00 3.05 3.07 query34 16.93 4.48 4.52 query35 4.63 4.57 4.55 query36 0.65 0.51 0.48 query37 0.09 0.06 0.06 query38 0.05 0.04 0.04 query39 0.04 0.03 0.02 query40 0.17 0.13 0.12 query41 0.08 0.02 0.02 query42 0.03 0.02 0.02 query43 0.04 0.02 0.03 Total cold run time: 107.75 s Total hot run time: 32.12 s ```
Gabriel39 commented 2 days ago

run buildall

github-actions[bot] commented 2 days ago

clang-tidy review says "All clean, LGTM! :+1:"

doris-robot commented 2 days ago
TPC-H: Total hot run time: 40320 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 66903ba5560fd25ca0264811fac0cd9fd05537fa, data reload: false ------ Round 1 ---------------------------------- q1 17623 7537 7403 7403 q2 2048 184 172 172 q3 10635 1107 1186 1107 q4 10502 715 659 659 q5 7584 2815 2732 2732 q6 240 151 147 147 q7 1005 624 614 614 q8 9249 1880 1935 1880 q9 6583 6450 6458 6450 q10 7012 2295 2332 2295 q11 464 258 270 258 q12 416 223 225 223 q13 17773 3019 3051 3019 q14 237 214 212 212 q15 582 544 525 525 q16 675 592 583 583 q17 993 567 539 539 q18 7404 6808 6752 6752 q19 1332 997 1102 997 q20 469 192 186 186 q21 4299 3407 3255 3255 q22 386 324 312 312 Total cold run time: 107511 ms Total hot run time: 40320 ms ----- Round 2, with runtime_filter_mode=off ----- q1 7296 7281 7311 7281 q2 332 234 236 234 q3 2902 2906 3003 2906 q4 2107 1904 1828 1828 q5 5679 5710 5757 5710 q6 229 143 142 142 q7 2291 1846 1852 1846 q8 3446 3618 3585 3585 q9 8993 8967 8972 8967 q10 3651 3616 3580 3580 q11 610 511 514 511 q12 898 690 618 618 q13 10847 3255 3244 3244 q14 307 277 292 277 q15 586 529 530 529 q16 697 647 675 647 q17 1882 1631 1610 1610 q18 8455 7971 7707 7707 q19 2020 1590 1507 1507 q20 2172 1877 1881 1877 q21 5657 5611 5430 5430 q22 641 564 567 564 Total cold run time: 71698 ms Total hot run time: 60600 ms ```
doris-robot commented 2 days ago

TeamCity be ut coverage result: Function Coverage: 38.32% (9979/26044) Line Coverage: 29.43% (83498/283754) Region Coverage: 28.56% (42932/150301) Branch Coverage: 25.15% (21812/86718) Coverage Report: http://coverage.selectdb-in.cc/coverage/66903ba5560fd25ca0264811fac0cd9fd05537fa_66903ba5560fd25ca0264811fac0cd9fd05537fa/report/index.html

doris-robot commented 2 days ago
TPC-DS: Total hot run time: 198479 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 66903ba5560fd25ca0264811fac0cd9fd05537fa, data reload: false query1 1238 960 925 925 query2 6260 2176 2169 2169 query3 10824 4104 4061 4061 query4 68379 27330 23538 23538 query5 5038 474 454 454 query6 431 184 190 184 query7 5682 310 291 291 query8 345 238 238 238 query9 9234 2680 2674 2674 query10 461 262 257 257 query11 17443 15216 15881 15216 query12 173 106 103 103 query13 1589 428 440 428 query14 11043 7585 7948 7585 query15 228 194 199 194 query16 7127 483 497 483 query17 1244 614 603 603 query18 1838 321 324 321 query19 213 172 163 163 query20 122 123 114 114 query21 216 108 108 108 query22 4760 4563 4489 4489 query23 35088 34690 34618 34618 query24 5448 2586 2558 2558 query25 494 392 401 392 query26 674 151 161 151 query27 1820 293 298 293 query28 4302 2511 2497 2497 query29 738 429 425 425 query30 214 153 153 153 query31 1019 831 856 831 query32 70 59 58 58 query33 431 298 292 292 query34 959 536 538 536 query35 880 749 765 749 query36 1075 988 1014 988 query37 123 78 80 78 query38 4568 4370 4458 4370 query39 1505 1472 1505 1472 query40 213 102 104 102 query41 45 43 66 43 query42 109 101 98 98 query43 574 522 511 511 query44 1258 874 863 863 query45 187 170 187 170 query46 1174 744 729 729 query47 2019 1933 1933 1933 query48 453 327 325 325 query49 736 404 420 404 query50 863 413 420 413 query51 7359 7363 7196 7196 query52 104 87 94 87 query53 268 182 181 181 query54 511 410 402 402 query55 78 76 78 76 query56 270 242 252 242 query57 1380 1180 1169 1169 query58 246 220 213 213 query59 3251 3106 3062 3062 query60 275 252 250 250 query61 121 106 109 106 query62 807 664 668 664 query63 228 197 198 197 query64 1388 682 661 661 query65 3338 3191 3253 3191 query66 701 304 310 304 query67 16070 15628 15647 15628 query68 3917 585 563 563 query69 429 255 253 253 query70 1126 1171 1139 1139 query71 352 256 251 251 query72 6555 4058 4101 4058 query73 790 375 368 368 query74 10278 9014 9052 9014 query75 3441 2724 2701 2701 query76 1834 1049 1130 1049 query77 512 303 300 300 query78 10527 9476 9376 9376 query79 1853 597 625 597 query80 1159 435 450 435 query81 526 243 228 228 query82 276 118 123 118 query83 207 150 156 150 query84 283 72 75 72 query85 978 307 311 307 query86 391 306 266 266 query87 4698 4539 4627 4539 query88 3638 2300 2238 2238 query89 416 298 310 298 query90 1962 196 189 189 query91 143 107 103 103 query92 63 50 51 50 query93 2563 562 555 555 query94 820 297 287 287 query95 357 249 259 249 query96 646 288 284 284 query97 2882 2744 2674 2674 query98 212 191 186 186 query99 1701 1324 1303 1303 Total cold run time: 322857 ms Total hot run time: 198479 ms ```
doris-robot commented 2 days ago
ClickBench: Total hot run time: 32.39 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 66903ba5560fd25ca0264811fac0cd9fd05537fa, data reload: false query1 0.03 0.03 0.04 query2 0.07 0.03 0.04 query3 0.23 0.07 0.07 query4 1.63 0.10 0.11 query5 0.45 0.41 0.41 query6 1.17 0.66 0.65 query7 0.02 0.01 0.01 query8 0.04 0.03 0.03 query9 0.57 0.51 0.50 query10 0.54 0.56 0.55 query11 0.14 0.10 0.10 query12 0.14 0.12 0.11 query13 0.62 0.61 0.60 query14 2.83 2.76 2.70 query15 0.89 0.83 0.83 query16 0.38 0.38 0.39 query17 1.08 1.03 1.07 query18 0.23 0.21 0.20 query19 1.91 1.85 1.90 query20 0.02 0.01 0.01 query21 15.35 0.57 0.56 query22 2.43 2.15 1.58 query23 16.88 0.89 0.76 query24 3.44 1.28 0.95 query25 0.29 0.24 0.17 query26 0.26 0.14 0.14 query27 0.05 0.04 0.05 query28 10.06 1.11 1.08 query29 12.61 3.21 3.17 query30 0.25 0.07 0.06 query31 2.87 0.39 0.38 query32 3.27 0.46 0.46 query33 3.08 3.02 3.04 query34 16.78 4.53 4.57 query35 4.50 4.51 4.54 query36 0.68 0.49 0.49 query37 0.09 0.06 0.06 query38 0.04 0.03 0.03 query39 0.03 0.02 0.02 query40 0.18 0.13 0.12 query41 0.08 0.03 0.03 query42 0.03 0.02 0.02 query43 0.03 0.02 0.02 Total cold run time: 106.27 s Total hot run time: 32.39 s ```
github-actions[bot] commented 2 days ago

PR approved by at least one committer and no changes requested.

github-actions[bot] commented 2 days ago

PR approved by anyone and no changes requested.