[Enhancement] (nereids)implement syncCommand in nereids

Vallishp commented 1 day ago

What problem does this PR solve?

Issue Number: close #42814

Related PR: #xxx

Problem Summary: implement syncCommand in nereids

Release note

None

Check List (For Author)

Test
- [X] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
  - [ ] This is a refactor/code format and no logic has been changed.
  - [ ] Previous test can cover this change.
  - [ ] No code files have been changed.
  - [ ] Other reason
Behavior changed:
- [ ] No.
- [ ] Yes.
Does this need documentation?
- [ ] No.
- [ ] Yes.

Check List (For Reviewer who merge this PR)

[ ] Confirm the release note
[ ] Confirm test cases
[ ] Confirm document
[ ] Add branch pick label

doris-robot commented 1 day ago

Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Vallishp commented 1 day ago

run buildall

Vallishp commented 1 day ago

run buildall

Vallishp commented 1 day ago

run buildall

doris-robot commented 1 day ago

TPC-H: Total hot run time: 39688 ms

``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 6e2a63c24415e993adb0f4d2e9db6026226c3b18, data reload: false ------ Round 1 ---------------------------------- q1 17603 7494 7283 7283 q2 2053 168 169 168 q3 10561 1120 1140 1120 q4 10216 716 692 692 q5 7603 2717 2631 2631 q6 242 150 149 149 q7 987 626 599 599 q8 9242 1865 1863 1863 q9 6533 6354 6455 6354 q10 6985 2269 2317 2269 q11 470 252 267 252 q12 415 214 222 214 q13 17751 3056 3019 3019 q14 242 204 224 204 q15 567 529 521 521 q16 644 594 583 583 q17 972 663 513 513 q18 7400 6788 6671 6671 q19 1351 1025 997 997 q20 507 180 178 178 q21 4191 3197 3092 3092 q22 379 316 319 316 Total cold run time: 106914 ms Total hot run time: 39688 ms ----- Round 2, with runtime_filter_mode=off ----- q1 7274 7284 7229 7229 q2 331 230 235 230 q3 2917 2793 2843 2793 q4 1969 1727 1664 1664 q5 5427 5387 5447 5387 q6 215 137 135 135 q7 2131 1736 1730 1730 q8 3250 3398 3408 3398 q9 8448 8545 8459 8459 q10 3549 3443 3449 3443 q11 593 496 500 496 q12 793 648 610 610 q13 11803 3051 3055 3051 q14 290 285 275 275 q15 563 505 515 505 q16 678 603 628 603 q17 1808 1574 1560 1560 q18 7934 7593 7488 7488 q19 1672 1625 1550 1550 q20 2040 1806 1811 1806 q21 5457 5323 5203 5203 q22 658 578 568 568 Total cold run time: 69800 ms Total hot run time: 58183 ms ```

doris-robot commented 1 day ago

TPC-DS: Total hot run time: 191713 ms

``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 6e2a63c24415e993adb0f4d2e9db6026226c3b18, data reload: false query1 949 389 366 366 query2 6521 2204 2180 2180 query3 6705 216 213 213 query4 33941 23561 23610 23561 query5 4286 457 457 457 query6 275 196 175 175 query7 4623 300 293 293 query8 306 233 227 227 query9 9419 2684 2662 2662 query10 464 255 247 247 query11 18239 15465 15217 15217 query12 148 97 108 97 query13 1633 417 406 406 query14 9862 7518 7054 7054 query15 303 170 183 170 query16 8214 454 450 450 query17 1741 562 548 548 query18 2127 286 290 286 query19 363 148 141 141 query20 128 113 108 108 query21 202 102 100 100 query22 4812 4491 4595 4491 query23 34967 34098 34315 34098 query24 10470 2471 2412 2412 query25 620 380 386 380 query26 1201 147 151 147 query27 2869 291 276 276 query28 7948 2450 2440 2440 query29 833 399 404 399 query30 298 148 160 148 query31 1022 786 826 786 query32 95 64 59 59 query33 779 278 283 278 query34 998 496 512 496 query35 862 719 725 719 query36 1106 964 977 964 query37 135 79 79 79 query38 4371 4287 4290 4287 query39 1560 1415 1467 1415 query40 204 104 105 104 query41 50 47 48 47 query42 114 99 99 99 query43 544 513 515 513 query44 1252 835 810 810 query45 188 166 164 164 query46 1135 716 705 705 query47 1985 1854 1868 1854 query48 438 326 382 326 query49 1174 387 381 381 query50 815 380 388 380 query51 7361 7236 7032 7032 query52 102 88 87 87 query53 252 178 184 178 query54 1347 403 442 403 query55 80 82 78 78 query56 255 242 232 232 query57 1303 1196 1160 1160 query58 236 215 226 215 query59 3203 3135 3354 3135 query60 293 240 252 240 query61 115 105 104 104 query62 889 684 667 667 query63 211 189 178 178 query64 5285 670 639 639 query65 3283 3243 3254 3243 query66 1467 311 309 309 query67 16046 15938 15687 15687 query68 5014 556 571 556 query69 438 249 255 249 query70 1196 1169 1079 1079 query71 340 249 251 249 query72 6535 4136 3989 3989 query73 757 351 365 351 query74 10366 8954 8934 8934 query75 3442 2638 2697 2638 query76 2948 1052 1085 1052 query77 425 284 283 283 query78 10511 9536 9433 9433 query79 2428 606 611 606 query80 1152 428 438 428 query81 558 237 238 237 query82 924 117 120 117 query83 251 151 144 144 query84 238 72 75 72 query85 1497 309 290 290 query86 451 313 301 301 query87 4772 4518 4644 4518 query88 4026 2185 2144 2144 query89 403 292 303 292 query90 2131 244 185 185 query91 139 105 98 98 query92 68 51 50 50 query93 1687 561 552 552 query94 1027 300 305 300 query95 343 243 242 242 query96 617 280 288 280 query97 2850 2748 2737 2737 query98 218 199 197 197 query99 1585 1308 1310 1308 Total cold run time: 304359 ms Total hot run time: 191713 ms ```

doris-robot commented 1 day ago

ClickBench: Total hot run time: 33.25 s

``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 6e2a63c24415e993adb0f4d2e9db6026226c3b18, data reload: false query1 0.03 0.03 0.03 query2 0.06 0.04 0.03 query3 0.23 0.07 0.08 query4 1.62 0.10 0.10 query5 0.43 0.40 0.40 query6 1.16 0.66 0.65 query7 0.03 0.02 0.01 query8 0.04 0.03 0.04 query9 0.58 0.54 0.50 query10 0.55 0.56 0.56 query11 0.14 0.11 0.11 query12 0.13 0.12 0.11 query13 0.62 0.60 0.58 query14 2.81 2.82 2.83 query15 0.90 0.82 0.83 query16 0.38 0.38 0.37 query17 1.06 1.07 1.00 query18 0.23 0.22 0.21 query19 1.92 1.86 2.07 query20 0.01 0.01 0.00 query21 15.37 0.60 0.58 query22 2.55 2.41 1.75 query23 17.25 1.01 0.89 query24 2.82 1.61 1.76 query25 0.20 0.07 0.05 query26 0.76 0.14 0.14 query27 0.04 0.04 0.05 query28 9.73 1.10 1.08 query29 12.54 3.19 3.19 query30 0.27 0.07 0.06 query31 2.88 0.38 0.38 query32 3.25 0.47 0.48 query33 3.00 3.10 3.02 query34 17.08 4.49 4.52 query35 4.49 4.49 4.47 query36 0.67 0.48 0.50 query37 0.09 0.06 0.06 query38 0.05 0.04 0.03 query39 0.03 0.02 0.02 query40 0.16 0.12 0.11 query41 0.08 0.02 0.02 query42 0.03 0.02 0.02 query43 0.04 0.03 0.03 Total cold run time: 106.31 s Total hot run time: 33.25 s ```

github-actions[bot] commented 1 day ago

PR approved by at least one committer and no changes requested.

github-actions[bot] commented 1 day ago

PR approved by anyone and no changes requested.

Vallishp commented 20 hours ago

run buildall

doris-robot commented 20 hours ago

TPC-H: Total hot run time: 40142 ms

``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit b4ff7dd5ffd46a5376ae8eaa5f45927fbae2b009, data reload: false ------ Round 1 ---------------------------------- q1 17660 7470 7344 7344 q2 2039 175 165 165 q3 10616 1106 1167 1106 q4 10576 760 717 717 q5 7622 2714 2682 2682 q6 240 148 144 144 q7 985 626 622 622 q8 9257 1880 1927 1880 q9 6591 6368 6343 6343 q10 6966 2293 2327 2293 q11 461 265 264 264 q12 435 212 217 212 q13 17798 3085 3062 3062 q14 250 214 218 214 q15 587 534 521 521 q16 611 575 584 575 q17 990 551 633 551 q18 7420 6735 6777 6735 q19 1338 1051 1061 1051 q20 494 187 189 187 q21 4001 3160 3287 3160 q22 379 319 314 314 Total cold run time: 107316 ms Total hot run time: 40142 ms ----- Round 2, with runtime_filter_mode=off ----- q1 7325 7278 7398 7278 q2 328 233 226 226 q3 2927 2813 2938 2813 q4 2139 1836 1962 1836 q5 5694 5747 5695 5695 q6 224 148 147 147 q7 2227 1876 1802 1802 q8 3475 3584 3568 3568 q9 8922 8978 9041 8978 q10 3591 3557 3548 3548 q11 600 529 509 509 q12 811 619 596 596 q13 11661 3311 3301 3301 q14 300 271 269 269 q15 573 515 534 515 q16 683 669 641 641 q17 1883 1651 1626 1626 q18 8457 7909 7639 7639 q19 1712 1558 1604 1558 q20 2119 1865 1874 1865 q21 5655 5490 5376 5376 q22 639 558 563 558 Total cold run time: 71945 ms Total hot run time: 60344 ms ```

doris-robot commented 20 hours ago

TPC-DS: Total hot run time: 197848 ms

``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit b4ff7dd5ffd46a5376ae8eaa5f45927fbae2b009, data reload: false query1 1223 952 931 931 query2 6225 2137 2089 2089 query3 10807 3915 3954 3915 query4 67966 26849 23799 23799 query5 4944 473 456 456 query6 429 183 192 183 query7 5479 303 300 300 query8 321 235 240 235 query9 8445 2709 2683 2683 query10 444 268 259 259 query11 16985 15371 15971 15371 query12 159 104 117 104 query13 1506 439 432 432 query14 10773 7422 7829 7422 query15 209 197 198 197 query16 7216 437 461 437 query17 1104 574 605 574 query18 1825 301 305 301 query19 216 182 158 158 query20 133 111 113 111 query21 201 110 109 109 query22 4572 4523 4306 4306 query23 35381 34527 34754 34527 query24 5453 2647 2535 2535 query25 490 398 395 395 query26 652 154 154 154 query27 1869 290 285 285 query28 4617 2512 2461 2461 query29 714 444 423 423 query30 219 150 146 146 query31 1030 839 852 839 query32 75 58 61 58 query33 437 292 298 292 query34 953 536 549 536 query35 893 764 735 735 query36 1109 952 986 952 query37 122 73 86 73 query38 4607 4505 4347 4347 query39 1553 1662 1472 1472 query40 207 106 102 102 query41 46 44 46 44 query42 118 101 105 101 query43 574 514 503 503 query44 1210 834 848 834 query45 192 166 170 166 query46 1175 803 738 738 query47 2010 1898 1920 1898 query48 425 329 327 327 query49 714 405 405 405 query50 852 402 423 402 query51 7591 7276 7157 7157 query52 101 91 89 89 query53 266 191 176 176 query54 525 392 392 392 query55 81 75 72 72 query56 266 242 246 242 query57 1307 1217 1128 1128 query58 224 228 228 228 query59 3260 3035 3166 3035 query60 275 253 256 253 query61 109 106 106 106 query62 802 663 697 663 query63 223 194 197 194 query64 1389 692 632 632 query65 3294 3221 3215 3215 query66 648 300 317 300 query67 16067 15544 15961 15544 query68 3907 551 559 551 query69 425 263 258 258 query70 1230 1153 1158 1153 query71 353 258 263 258 query72 6304 4096 3751 3751 query73 814 364 369 364 query74 10237 9217 9258 9217 query75 3351 2718 2699 2699 query76 1835 1060 1152 1060 query77 482 279 271 271 query78 10612 9463 9443 9443 query79 2016 615 602 602 query80 1401 429 444 429 query81 537 244 236 236 query82 1278 125 116 116 query83 271 152 158 152 query84 280 73 72 72 query85 1029 307 298 298 query86 428 359 290 290 query87 4770 4668 4704 4668 query88 3854 2265 2214 2214 query89 433 295 309 295 query90 1899 191 197 191 query91 140 101 103 101 query92 64 50 51 50 query93 2803 549 547 547 query94 771 305 291 291 query95 360 254 258 254 query96 686 275 276 275 query97 2906 2694 2719 2694 query98 217 197 198 197 query99 1655 1303 1331 1303 Total cold run time: 322655 ms Total hot run time: 197848 ms ```

doris-robot commented 19 hours ago

ClickBench: Total hot run time: 31.68 s

``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit b4ff7dd5ffd46a5376ae8eaa5f45927fbae2b009, data reload: false query1 0.04 0.03 0.02 query2 0.07 0.03 0.03 query3 0.23 0.07 0.07 query4 1.62 0.11 0.10 query5 0.42 0.40 0.43 query6 1.17 0.64 0.64 query7 0.02 0.01 0.02 query8 0.05 0.04 0.04 query9 0.58 0.51 0.50 query10 0.56 0.56 0.56 query11 0.15 0.10 0.10 query12 0.14 0.11 0.11 query13 0.61 0.62 0.60 query14 2.75 2.75 2.86 query15 0.91 0.84 0.83 query16 0.37 0.41 0.40 query17 1.07 1.02 1.05 query18 0.23 0.22 0.22 query19 1.84 1.87 1.96 query20 0.02 0.01 0.01 query21 15.37 0.58 0.58 query22 2.65 2.70 1.68 query23 17.02 1.08 0.75 query24 3.18 1.88 0.21 query25 0.16 0.08 0.22 query26 0.52 0.13 0.14 query27 0.04 0.05 0.04 query28 10.71 1.11 1.08 query29 12.57 3.28 3.25 query30 0.25 0.06 0.06 query31 2.87 0.40 0.36 query32 3.27 0.47 0.48 query33 2.98 3.02 3.08 query34 17.16 4.46 4.47 query35 4.56 4.47 4.47 query36 0.68 0.49 0.48 query37 0.09 0.05 0.06 query38 0.05 0.03 0.03 query39 0.03 0.02 0.03 query40 0.16 0.12 0.12 query41 0.08 0.03 0.02 query42 0.03 0.02 0.02 query43 0.03 0.02 0.02 Total cold run time: 107.31 s Total hot run time: 31.68 s ```

Vallishp commented 12 hours ago

run buildall

doris-robot commented 11 hours ago

TPC-H: Total hot run time: 39859 ms

``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit bf6f837349569b7dcdc37765813c0883e5a2bf27, data reload: false ------ Round 1 ---------------------------------- q1 17593 7445 7296 7296 q2 2065 177 167 167 q3 10551 1111 1159 1111 q4 10214 698 691 691 q5 7592 2736 2662 2662 q6 240 153 154 153 q7 1000 636 619 619 q8 9237 1844 1908 1844 q9 6549 6369 6390 6369 q10 6956 2280 2325 2280 q11 454 261 261 261 q12 418 226 226 226 q13 17751 3062 3041 3041 q14 240 209 215 209 q15 578 520 538 520 q16 658 589 583 583 q17 969 544 632 544 q18 7432 6800 6600 6600 q19 1334 1035 1165 1035 q20 476 190 183 183 q21 4258 3158 3338 3158 q22 389 307 325 307 Total cold run time: 106954 ms Total hot run time: 39859 ms ----- Round 2, with runtime_filter_mode=off ----- q1 7233 7254 7227 7227 q2 326 235 227 227 q3 2944 2741 3040 2741 q4 2140 1861 1872 1861 q5 5583 5721 5681 5681 q6 225 138 140 138 q7 2210 1792 1839 1792 q8 3378 3515 3460 3460 q9 8846 8917 8971 8917 q10 3572 3509 3530 3509 q11 606 518 506 506 q12 830 592 662 592 q13 11529 3212 3188 3188 q14 292 271 253 253 q15 543 511 512 511 q16 674 649 626 626 q17 1820 1578 1576 1576 q18 7950 7511 7506 7506 q19 1679 1574 1486 1486 q20 2066 1810 1860 1810 q21 5287 5295 5128 5128 q22 640 592 546 546 Total cold run time: 70373 ms Total hot run time: 59281 ms ```

doris-robot commented 11 hours ago

TPC-DS: Total hot run time: 191865 ms

``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit bf6f837349569b7dcdc37765813c0883e5a2bf27, data reload: false query1 975 403 384 384 query2 6526 2175 2089 2089 query3 6714 213 218 213 query4 34144 23680 23662 23662 query5 4364 470 487 470 query6 290 193 179 179 query7 4619 297 304 297 query8 283 226 216 216 query9 9445 2679 2690 2679 query10 473 256 244 244 query11 18367 15360 15233 15233 query12 150 103 101 101 query13 1647 441 415 415 query14 9024 6692 7719 6692 query15 295 182 181 181 query16 8206 477 459 459 query17 1721 609 579 579 query18 2170 320 302 302 query19 381 159 154 154 query20 119 112 111 111 query21 211 103 104 103 query22 4514 4141 4268 4141 query23 34898 34648 34710 34648 query24 10376 2494 2471 2471 query25 644 397 391 391 query26 1513 148 151 148 query27 2880 281 284 281 query28 8283 2472 2422 2422 query29 912 411 415 411 query30 299 153 151 151 query31 1029 812 823 812 query32 94 55 59 55 query33 774 287 277 277 query34 981 504 527 504 query35 835 715 728 715 query36 1091 937 929 929 query37 151 74 74 74 query38 4341 4282 4339 4282 query39 1490 1453 1425 1425 query40 276 101 102 101 query41 48 44 48 44 query42 112 100 95 95 query43 545 493 498 493 query44 1290 841 829 829 query45 180 164 167 164 query46 1156 701 695 695 query47 1938 1841 1859 1841 query48 408 309 318 309 query49 1173 399 381 381 query50 857 377 396 377 query51 7289 7162 7100 7100 query52 100 93 89 89 query53 261 186 186 186 query54 1081 407 403 403 query55 82 76 76 76 query56 254 244 237 237 query57 1302 1176 1172 1172 query58 228 215 231 215 query59 3402 3025 3138 3025 query60 258 241 249 241 query61 111 110 111 110 query62 846 667 695 667 query63 218 184 188 184 query64 4620 687 626 626 query65 3277 3191 3274 3191 query66 883 318 313 313 query67 15941 15676 15434 15434 query68 4963 553 550 550 query69 430 257 262 257 query70 1266 1133 1124 1124 query71 319 250 248 248 query72 6437 4043 4017 4017 query73 764 381 373 373 query74 10262 9111 9010 9010 query75 3426 2674 2682 2674 query76 2958 1069 1136 1069 query77 528 281 279 279 query78 10379 10256 9590 9590 query79 1699 603 604 603 query80 802 429 425 425 query81 563 247 226 226 query82 809 130 118 118 query83 279 152 150 150 query84 236 73 73 73 query85 1330 299 302 299 query86 476 299 311 299 query87 4641 4533 4652 4533 query88 4154 2269 2300 2269 query89 401 300 296 296 query90 1949 192 192 192 query91 140 103 105 103 query92 68 52 52 52 query93 1561 551 555 551 query94 726 282 294 282 query95 344 249 245 245 query96 614 295 285 285 query97 2892 2679 2676 2676 query98 220 197 199 197 query99 1527 1338 1302 1302 Total cold run time: 300528 ms Total hot run time: 191865 ms ```

doris-robot commented 11 hours ago

ClickBench: Total hot run time: 32.92 s

``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit bf6f837349569b7dcdc37765813c0883e5a2bf27, data reload: false query1 0.03 0.03 0.03 query2 0.08 0.03 0.03 query3 0.25 0.07 0.07 query4 1.62 0.10 0.11 query5 0.44 0.39 0.42 query6 1.17 0.66 0.65 query7 0.02 0.02 0.01 query8 0.03 0.03 0.03 query9 0.56 0.54 0.49 query10 0.54 0.56 0.55 query11 0.14 0.11 0.10 query12 0.15 0.12 0.10 query13 0.62 0.62 0.60 query14 2.72 2.73 2.74 query15 0.90 0.83 0.84 query16 0.39 0.40 0.41 query17 1.04 1.02 1.08 query18 0.22 0.20 0.22 query19 1.82 1.87 1.98 query20 0.01 0.01 0.01 query21 15.36 0.60 0.60 query22 2.64 2.75 1.99 query23 16.98 0.84 0.92 query24 2.62 1.08 1.03 query25 0.23 0.17 0.10 query26 0.40 0.14 0.15 query27 0.06 0.04 0.04 query28 11.10 1.13 1.09 query29 12.55 3.30 3.28 query30 0.25 0.06 0.06 query31 2.89 0.39 0.39 query32 3.28 0.48 0.48 query33 3.01 3.00 3.04 query34 16.99 4.53 4.44 query35 4.52 4.45 4.56 query36 0.66 0.50 0.48 query37 0.10 0.06 0.07 query38 0.05 0.03 0.04 query39 0.03 0.02 0.03 query40 0.16 0.13 0.12 query41 0.08 0.03 0.02 query42 0.03 0.03 0.02 query43 0.04 0.03 0.03 Total cold run time: 106.78 s Total hot run time: 32.92 s ```

apache / doris