apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
11.8k stars 3.11k forks source link

[opt](binlog) support rename binlog #37031

Open lsy3993 opened 2 days ago

lsy3993 commented 2 days ago

Proposed changes

Rename operator doesn't have binlog now. This PR will create binlog when execute rename. The rename operator means :

  1. rename table : ALTER TABLE table1 RENAME table2;
  2. rename rollup : ALTER TABLE example_table RENAME ROLLUP rollup1 rollup2;
  3. rename partition : ALTER TABLE example_table RENAME PARTITION p1 p2;

After SQL analyzing, we can get the old and new table name (rollup name or partition name) , then record the info to binlog, so we can use the info from binlog.

doris-robot commented 2 days ago

Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.

github-actions[bot] commented 2 days ago

clang-tidy review says "All clean, LGTM! :+1:"

dataroaring commented 1 day ago

run buildall

lsy3993 commented 1 day ago

run buildall

lsy3993 commented 1 day ago

run buildall

github-actions[bot] commented 1 day ago

clang-tidy review says "All clean, LGTM! :+1:"

github-actions[bot] commented 1 day ago

clang-tidy review says "All clean, LGTM! :+1:"

doris-robot commented 1 day ago
TPC-H: Total hot run time: 40067 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 177485f494d4bc1779b5049a997665adbe28b4d0, data reload: false ------ Round 1 ---------------------------------- q1 18291 4907 4326 4326 q2 2043 190 190 190 q3 10550 1222 1136 1136 q4 10230 789 829 789 q5 7482 2668 2630 2630 q6 217 134 134 134 q7 953 599 608 599 q8 9232 2090 2097 2090 q9 9140 6540 6495 6495 q10 8931 3736 3735 3735 q11 450 247 244 244 q12 401 246 228 228 q13 17765 3003 3006 3003 q14 273 225 218 218 q15 536 487 493 487 q16 534 386 375 375 q17 985 709 730 709 q18 8148 7496 7390 7390 q19 5942 1437 1548 1437 q20 673 324 330 324 q21 4933 3180 3989 3180 q22 419 361 348 348 Total cold run time: 118128 ms Total hot run time: 40067 ms ----- Round 2, with runtime_filter_mode=off ----- q1 4478 4266 4243 4243 q2 386 270 262 262 q3 3046 2929 2946 2929 q4 1976 1798 1732 1732 q5 5533 5530 5468 5468 q6 241 132 133 132 q7 2248 1867 1857 1857 q8 3284 3478 3458 3458 q9 8726 8799 8649 8649 q10 4139 3731 3769 3731 q11 617 515 509 509 q12 808 671 654 654 q13 16492 3201 3174 3174 q14 297 279 268 268 q15 527 499 476 476 q16 522 428 427 427 q17 1831 1531 1484 1484 q18 8272 7977 7835 7835 q19 1852 1601 1562 1562 q20 3087 1911 1854 1854 q21 5276 4834 4883 4834 q22 679 558 589 558 Total cold run time: 74317 ms Total hot run time: 56096 ms ```
doris-robot commented 1 day ago
TPC-DS: Total hot run time: 174928 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 177485f494d4bc1779b5049a997665adbe28b4d0, data reload: false query1 932 379 378 378 query2 6568 2366 2474 2366 query3 6630 206 212 206 query4 19172 17761 17344 17344 query5 3702 495 475 475 query6 259 163 162 162 query7 4590 296 285 285 query8 312 297 294 294 query9 8475 2382 2367 2367 query10 566 297 279 279 query11 10857 10340 10103 10103 query12 113 84 92 84 query13 1647 381 370 370 query14 9743 7661 7840 7661 query15 263 192 185 185 query16 7813 275 278 275 query17 1883 556 536 536 query18 1553 289 277 277 query19 205 155 155 155 query20 91 85 85 85 query21 212 128 127 127 query22 4358 4451 4068 4068 query23 33925 33615 33520 33520 query24 10740 2853 2902 2853 query25 593 404 380 380 query26 706 160 159 159 query27 2242 332 323 323 query28 5923 2090 2095 2090 query29 900 646 646 646 query30 271 159 156 156 query31 968 751 776 751 query32 103 57 55 55 query33 737 303 294 294 query34 944 471 484 471 query35 766 645 625 625 query36 1120 983 986 983 query37 155 75 78 75 query38 2992 2824 2877 2824 query39 913 844 821 821 query40 224 132 124 124 query41 60 55 54 54 query42 110 103 106 103 query43 629 585 565 565 query44 1134 736 717 717 query45 196 168 164 164 query46 1076 707 752 707 query47 1911 1823 1802 1802 query48 367 308 307 307 query49 854 504 405 405 query50 752 382 391 382 query51 6909 6874 6695 6695 query52 103 92 91 91 query53 356 282 283 282 query54 931 442 438 438 query55 73 71 70 70 query56 284 252 265 252 query57 1110 1099 1098 1098 query58 241 237 237 237 query59 3719 3313 3098 3098 query60 295 278 286 278 query61 89 88 89 88 query62 601 476 438 438 query63 325 278 279 278 query64 8645 2258 1733 1733 query65 3165 3096 3094 3094 query66 743 327 320 320 query67 15617 15364 15109 15109 query68 8595 551 549 549 query69 741 485 421 421 query70 1411 1094 1139 1094 query71 539 280 278 278 query72 8902 5408 5651 5408 query73 2278 325 319 319 query74 5992 5612 5598 5598 query75 4869 2665 2652 2652 query76 5126 977 907 907 query77 815 302 297 297 query78 10469 9886 9766 9766 query79 9675 524 517 517 query80 1180 506 473 473 query81 553 224 216 216 query82 486 109 104 104 query83 344 169 166 166 query84 274 91 84 84 query85 988 282 267 267 query86 367 308 282 282 query87 3325 3087 3052 3052 query88 4592 2388 2337 2337 query89 523 389 379 379 query90 2014 189 188 188 query91 164 103 98 98 query92 62 49 51 49 query93 6799 534 520 520 query94 1287 187 189 187 query95 413 317 321 317 query96 607 275 267 267 query97 3236 3057 3082 3057 query98 215 211 199 199 query99 1217 843 849 843 Total cold run time: 292519 ms Total hot run time: 174928 ms ```
doris-robot commented 1 day ago
ClickBench: Total hot run time: 30.68 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 177485f494d4bc1779b5049a997665adbe28b4d0, data reload: false query1 0.05 0.03 0.03 query2 0.09 0.04 0.04 query3 0.22 0.06 0.05 query4 1.67 0.09 0.09 query5 0.52 0.49 0.48 query6 1.13 0.72 0.71 query7 0.02 0.02 0.01 query8 0.05 0.04 0.05 query9 0.54 0.50 0.50 query10 0.55 0.54 0.54 query11 0.15 0.12 0.11 query12 0.14 0.13 0.13 query13 0.59 0.59 0.58 query14 0.77 0.78 0.78 query15 0.86 0.82 0.82 query16 0.37 0.35 0.36 query17 1.03 0.94 1.00 query18 0.20 0.24 0.23 query19 1.76 1.83 1.73 query20 0.02 0.01 0.01 query21 15.44 0.75 0.65 query22 3.76 7.70 2.06 query23 18.28 1.48 1.30 query24 2.14 0.23 0.21 query25 0.16 0.09 0.08 query26 0.27 0.18 0.17 query27 0.08 0.07 0.08 query28 13.28 1.03 1.00 query29 12.59 3.27 3.27 query30 0.26 0.06 0.05 query31 2.91 0.39 0.40 query32 3.24 0.47 0.48 query33 2.83 2.89 2.84 query34 17.15 4.42 4.41 query35 4.49 4.48 4.50 query36 0.65 0.46 0.47 query37 0.18 0.17 0.16 query38 0.16 0.14 0.15 query39 0.04 0.04 0.04 query40 0.17 0.16 0.13 query41 0.09 0.05 0.05 query42 0.05 0.05 0.04 query43 0.04 0.04 0.04 Total cold run time: 108.99 s Total hot run time: 30.68 s ```
lsy3993 commented 1 day ago

run buildall

doris-robot commented 1 day ago
TPC-H: Total hot run time: 40014 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 177485f494d4bc1779b5049a997665adbe28b4d0, data reload: false ------ Round 1 ---------------------------------- q1 18317 4711 4408 4408 q2 2550 194 198 194 q3 10991 1270 1220 1220 q4 10850 846 757 757 q5 7991 2743 2688 2688 q6 227 142 140 140 q7 982 603 616 603 q8 9602 2161 2081 2081 q9 8889 6501 6511 6501 q10 9037 3693 3717 3693 q11 454 240 238 238 q12 407 233 236 233 q13 17789 2945 2987 2945 q14 255 240 220 220 q15 534 482 476 476 q16 527 376 367 367 q17 982 677 672 672 q18 8102 7379 7340 7340 q19 5538 1379 1571 1379 q20 666 315 331 315 q21 4939 3205 3851 3205 q22 398 339 347 339 Total cold run time: 120027 ms Total hot run time: 40014 ms ----- Round 2, with runtime_filter_mode=off ----- q1 4362 4213 4267 4213 q2 367 272 266 266 q3 2995 2684 2714 2684 q4 1846 1637 1585 1585 q5 5262 5322 5278 5278 q6 221 127 129 127 q7 2117 1748 1732 1732 q8 3230 3340 3329 3329 q9 8363 8358 8393 8358 q10 3866 3675 3632 3632 q11 575 488 493 488 q12 771 601 608 601 q13 16413 3001 2995 2995 q14 290 252 254 252 q15 530 488 481 481 q16 457 405 422 405 q17 1775 1486 1467 1467 q18 7711 7411 7478 7411 q19 1717 1581 1530 1530 q20 1992 1751 1776 1751 q21 4975 4620 4655 4620 q22 619 535 574 535 Total cold run time: 70454 ms Total hot run time: 53740 ms ```
doris-robot commented 1 day ago
TPC-DS: Total hot run time: 172690 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 177485f494d4bc1779b5049a997665adbe28b4d0, data reload: false query1 920 380 374 374 query2 6454 2464 2477 2464 query3 6654 204 212 204 query4 19832 17426 17330 17330 query5 4208 483 505 483 query6 281 188 159 159 query7 4598 307 309 307 query8 316 289 285 285 query9 8479 2393 2369 2369 query10 611 299 295 295 query11 10594 10087 10022 10022 query12 129 88 86 86 query13 1645 397 371 371 query14 10394 7208 7793 7208 query15 242 189 192 189 query16 7937 280 279 279 query17 1881 581 544 544 query18 1846 291 283 283 query19 256 143 150 143 query20 99 82 79 79 query21 217 127 124 124 query22 4292 4253 4034 4034 query23 33617 32924 33073 32924 query24 11937 2805 2785 2785 query25 649 368 358 358 query26 1671 153 156 153 query27 2863 317 314 314 query28 7475 2075 2050 2050 query29 1002 616 603 603 query30 288 149 148 148 query31 944 722 729 722 query32 93 50 54 50 query33 767 292 293 292 query34 920 470 477 470 query35 774 631 610 610 query36 1063 950 882 882 query37 153 77 76 76 query38 2878 2742 2761 2742 query39 865 811 795 795 query40 276 128 126 126 query41 55 53 51 51 query42 123 103 99 99 query43 582 558 549 549 query44 1172 728 736 728 query45 191 158 159 158 query46 1081 745 714 714 query47 1859 1796 1784 1784 query48 376 305 290 290 query49 1152 431 428 428 query50 762 380 391 380 query51 6872 6805 6822 6805 query52 105 89 90 89 query53 361 299 282 282 query54 880 443 441 441 query55 74 72 73 72 query56 281 256 277 256 query57 1163 1059 1022 1022 query58 260 241 243 241 query59 3552 3093 3124 3093 query60 309 272 301 272 query61 92 89 92 89 query62 678 436 450 436 query63 321 287 289 287 query64 9858 2259 1769 1769 query65 3184 3102 3119 3102 query66 1319 341 331 331 query67 15384 15084 14771 14771 query68 4838 541 536 536 query69 582 436 332 332 query70 1173 1127 1083 1083 query71 432 289 280 280 query72 7352 5107 5443 5107 query73 758 328 324 324 query74 5978 5554 5438 5438 query75 3571 2675 2647 2647 query76 3415 1010 944 944 query77 659 304 308 304 query78 10368 9927 9820 9820 query79 2192 569 511 511 query80 1137 468 462 462 query81 563 228 218 218 query82 773 106 99 99 query83 193 165 165 165 query84 271 82 89 82 query85 1403 274 265 265 query86 465 325 305 305 query87 3271 3109 3061 3061 query88 3982 2382 2380 2380 query89 479 376 379 376 query90 1838 190 187 187 query91 134 100 101 100 query92 63 48 48 48 query93 2101 517 502 502 query94 1174 190 185 185 query95 402 317 367 317 query96 598 280 274 274 query97 3264 3063 3026 3026 query98 229 205 195 195 query99 1360 850 846 846 Total cold run time: 277184 ms Total hot run time: 172690 ms ```
doris-robot commented 1 day ago
ClickBench: Total hot run time: 30.2 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 177485f494d4bc1779b5049a997665adbe28b4d0, data reload: false query1 0.04 0.04 0.03 query2 0.08 0.05 0.04 query3 0.23 0.05 0.06 query4 1.66 0.10 0.10 query5 0.50 0.48 0.49 query6 1.14 0.72 0.73 query7 0.02 0.02 0.01 query8 0.05 0.05 0.05 query9 0.54 0.50 0.48 query10 0.54 0.54 0.54 query11 0.15 0.11 0.11 query12 0.16 0.12 0.12 query13 0.60 0.58 0.58 query14 0.76 0.80 0.79 query15 0.82 0.81 0.81 query16 0.37 0.36 0.36 query17 0.95 0.94 1.00 query18 0.21 0.26 0.25 query19 1.80 1.67 1.75 query20 0.02 0.01 0.01 query21 15.46 0.74 0.64 query22 4.02 7.17 1.77 query23 18.23 1.38 1.16 query24 2.10 0.23 0.22 query25 0.17 0.09 0.08 query26 0.27 0.18 0.18 query27 0.08 0.08 0.08 query28 13.21 1.02 0.98 query29 12.61 3.32 3.30 query30 0.26 0.07 0.06 query31 2.84 0.39 0.38 query32 3.29 0.47 0.47 query33 2.86 2.87 2.95 query34 17.16 4.37 4.49 query35 4.49 4.48 4.44 query36 0.65 0.48 0.47 query37 0.18 0.15 0.16 query38 0.16 0.15 0.15 query39 0.04 0.03 0.03 query40 0.17 0.14 0.13 query41 0.10 0.05 0.05 query42 0.06 0.05 0.04 query43 0.05 0.04 0.04 Total cold run time: 109.1 s Total hot run time: 30.2 s ```
lsy3993 commented 1 day ago

run buildall

doris-robot commented 1 day ago
TPC-H: Total hot run time: 40005 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 177485f494d4bc1779b5049a997665adbe28b4d0, data reload: false ------ Round 1 ---------------------------------- q1 18651 7777 4350 4350 q2 2034 189 184 184 q3 10616 1192 1121 1121 q4 10360 805 901 805 q5 7466 2716 2651 2651 q6 226 141 136 136 q7 948 605 618 605 q8 9249 2087 2073 2073 q9 9025 6524 6479 6479 q10 8993 3677 3699 3677 q11 460 238 241 238 q12 416 226 230 226 q13 18729 2961 2951 2951 q14 283 227 226 226 q15 536 467 483 467 q16 524 391 378 378 q17 986 679 705 679 q18 8183 7833 7493 7493 q19 7285 1433 1418 1418 q20 683 330 320 320 q21 4881 3190 3294 3190 q22 402 338 346 338 Total cold run time: 120936 ms Total hot run time: 40005 ms ----- Round 2, with runtime_filter_mode=off ----- q1 4393 4259 4409 4259 q2 395 281 268 268 q3 3181 2975 2967 2967 q4 1985 1685 1630 1630 q5 5522 5531 5567 5531 q6 235 131 132 131 q7 2238 1838 1858 1838 q8 3282 3436 3458 3436 q9 8743 8765 8759 8759 q10 3963 3793 3881 3793 q11 603 492 497 492 q12 824 669 634 634 q13 15986 3160 3172 3160 q14 302 269 283 269 q15 534 510 478 478 q16 492 417 422 417 q17 1834 1540 1519 1519 q18 8236 7921 7852 7852 q19 1774 1420 1557 1420 q20 3024 1903 1888 1888 q21 5003 4768 4918 4768 q22 602 551 569 551 Total cold run time: 73151 ms Total hot run time: 56060 ms ```
doris-robot commented 1 day ago
TPC-DS: Total hot run time: 171631 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 177485f494d4bc1779b5049a997665adbe28b4d0, data reload: false query1 926 380 376 376 query2 6369 2372 2449 2372 query3 6636 208 214 208 query4 20798 17527 17518 17518 query5 3759 466 488 466 query6 262 164 164 164 query7 4592 297 291 291 query8 328 308 303 303 query9 8478 2372 2356 2356 query10 583 309 284 284 query11 10574 10018 10196 10018 query12 118 88 85 85 query13 1647 374 360 360 query14 9975 7089 7803 7089 query15 260 195 201 195 query16 8055 277 279 277 query17 1894 563 537 537 query18 2092 284 273 273 query19 216 163 155 155 query20 95 83 84 83 query21 219 132 130 130 query22 4355 4041 3912 3912 query23 33884 33818 33559 33559 query24 10828 2927 2934 2927 query25 604 411 400 400 query26 711 165 161 161 query27 2316 334 327 327 query28 6229 2135 2137 2135 query29 922 678 664 664 query30 270 155 159 155 query31 958 778 758 758 query32 96 59 57 57 query33 676 315 305 305 query34 892 479 489 479 query35 824 678 662 662 query36 1134 972 966 966 query37 147 81 79 79 query38 2999 2876 2805 2805 query39 853 826 828 826 query40 204 125 120 120 query41 52 55 54 54 query42 127 100 102 100 query43 596 564 543 543 query44 1050 728 719 719 query45 190 169 162 162 query46 1077 695 723 695 query47 1837 1795 1770 1770 query48 365 290 290 290 query49 880 393 411 393 query50 766 388 388 388 query51 6971 6903 6735 6735 query52 97 104 89 89 query53 360 300 283 283 query54 854 449 438 438 query55 73 74 73 73 query56 281 266 269 266 query57 1144 1046 1057 1046 query58 259 243 248 243 query59 3218 3176 3232 3176 query60 300 324 280 280 query61 90 93 90 90 query62 577 465 478 465 query63 325 291 286 286 query64 8563 2272 1763 1763 query65 3150 3135 3135 3135 query66 737 320 322 320 query67 15803 14997 14880 14880 query68 8541 548 545 545 query69 771 469 342 342 query70 1247 1181 1099 1099 query71 489 275 275 275 query72 9028 2741 2619 2619 query73 1905 320 318 318 query74 5887 5532 5564 5532 query75 5113 2682 2680 2680 query76 4625 965 900 900 query77 793 306 300 300 query78 10461 9862 9830 9830 query79 3312 522 518 518 query80 2081 468 466 466 query81 547 221 216 216 query82 742 110 104 104 query83 291 178 169 169 query84 282 94 94 94 query85 1077 293 270 270 query86 400 321 321 321 query87 3268 3082 3085 3082 query88 4083 2419 2350 2350 query89 491 390 384 384 query90 1920 193 187 187 query91 129 98 103 98 query92 65 47 50 47 query93 4766 508 514 508 query94 1246 193 186 186 query95 410 306 323 306 query96 615 272 263 263 query97 3227 3056 3052 3052 query98 216 200 200 200 query99 1157 851 855 851 Total cold run time: 285705 ms Total hot run time: 171631 ms ```
doris-robot commented 1 day ago
ClickBench: Total hot run time: 30.77 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 177485f494d4bc1779b5049a997665adbe28b4d0, data reload: false query1 0.04 0.03 0.03 query2 0.08 0.04 0.04 query3 0.24 0.05 0.04 query4 1.69 0.08 0.07 query5 0.49 0.47 0.48 query6 1.13 0.74 0.72 query7 0.02 0.02 0.01 query8 0.05 0.05 0.05 query9 0.55 0.49 0.48 query10 0.56 0.53 0.53 query11 0.15 0.11 0.12 query12 0.15 0.12 0.12 query13 0.61 0.60 0.59 query14 0.75 0.81 0.79 query15 0.85 0.85 0.81 query16 0.37 0.37 0.36 query17 0.96 0.96 1.02 query18 0.21 0.26 0.20 query19 1.77 1.73 1.73 query20 0.01 0.01 0.02 query21 15.42 0.75 0.65 query22 4.82 6.91 2.03 query23 18.22 1.46 1.33 query24 2.10 0.23 0.22 query25 0.16 0.09 0.09 query26 0.26 0.18 0.17 query27 0.09 0.08 0.08 query28 13.28 1.04 1.01 query29 12.65 3.32 3.26 query30 0.26 0.06 0.06 query31 2.85 0.40 0.40 query32 3.26 0.48 0.47 query33 2.87 3.00 2.92 query34 17.19 4.43 4.40 query35 4.50 4.53 4.51 query36 0.65 0.46 0.46 query37 0.19 0.15 0.14 query38 0.15 0.15 0.15 query39 0.05 0.03 0.03 query40 0.17 0.14 0.14 query41 0.09 0.05 0.04 query42 0.06 0.05 0.05 query43 0.05 0.04 0.04 Total cold run time: 110.02 s Total hot run time: 30.77 s ```
github-actions[bot] commented 11 hours ago

clang-tidy review says "All clean, LGTM! :+1:"

github-actions[bot] commented 9 hours ago

PR approved by at least one committer and no changes requested.

github-actions[bot] commented 9 hours ago

PR approved by anyone and no changes requested.