apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
12.77k stars 3.29k forks source link

[fix](bloom filter)Fix drop column with bloom filter index #44361

Closed qidaye closed 2 hours ago

qidaye commented 1 day ago

What problem does this PR solve?

  1. When drop column with bloom filter, we modify the bloom filter column info
  2. When replay editLog, we rebuild bloom filter info by table schema.

Related PR: #41369

Problem Summary:

Release note

Fix drop column with bloom filter

Check List (For Author)

Check List (For Reviewer who merge this PR)

doris-robot commented 1 day ago

Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?
qidaye commented 1 day ago

run buildall

doris-robot commented 1 day ago
TPC-H: Total hot run time: 39934 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit b14cd87a133700eaeadc0a17dc78c69c1e1eb82e, data reload: false ------ Round 1 ---------------------------------- q1 17607 7448 7298 7298 q2 2035 184 177 177 q3 10530 1133 1141 1133 q4 10208 709 650 650 q5 7586 2756 2744 2744 q6 244 147 149 147 q7 984 629 612 612 q8 9245 1787 1944 1787 q9 6484 6420 6387 6387 q10 6957 2298 2305 2298 q11 460 260 264 260 q12 411 217 215 215 q13 17751 3067 3085 3067 q14 251 208 209 208 q15 577 532 533 532 q16 663 593 597 593 q17 974 644 585 585 q18 7535 6722 6718 6718 q19 1346 952 970 952 q20 454 189 179 179 q21 4554 3186 3076 3076 q22 387 325 316 316 Total cold run time: 107243 ms Total hot run time: 39934 ms ----- Round 2, with runtime_filter_mode=off ----- q1 7324 7316 7257 7257 q2 314 229 226 226 q3 2899 2739 2816 2739 q4 1968 1684 1688 1684 q5 5412 5404 5478 5404 q6 215 140 141 140 q7 2129 1785 1728 1728 q8 3266 3418 3420 3418 q9 8480 8501 8549 8501 q10 3511 3456 3457 3456 q11 593 507 520 507 q12 808 576 581 576 q13 10398 3065 3051 3051 q14 289 261 289 261 q15 566 515 510 510 q16 691 628 650 628 q17 1805 1615 1606 1606 q18 7853 7585 7388 7388 q19 1691 1597 1562 1562 q20 2108 1837 1833 1833 q21 5471 5400 5084 5084 q22 663 590 567 567 Total cold run time: 68454 ms Total hot run time: 58126 ms ```
doris-robot commented 1 day ago
TPC-DS: Total hot run time: 191241 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit b14cd87a133700eaeadc0a17dc78c69c1e1eb82e, data reload: false query1 981 386 374 374 query2 6524 2241 2156 2156 query3 6727 215 216 215 query4 34278 23510 23787 23510 query5 4353 457 434 434 query6 277 175 173 173 query7 4623 297 294 294 query8 277 221 219 219 query9 9718 2678 2662 2662 query10 482 258 249 249 query11 18236 15233 15176 15176 query12 158 103 103 103 query13 1657 429 430 429 query14 10231 7161 7490 7161 query15 299 177 174 174 query16 8202 455 460 455 query17 1845 588 584 584 query18 2139 301 303 301 query19 364 158 154 154 query20 118 114 115 114 query21 212 103 102 102 query22 4516 4425 4104 4104 query23 35144 34102 34064 34064 query24 11621 2466 2544 2466 query25 672 400 384 384 query26 1794 149 151 149 query27 2869 281 270 270 query28 8026 2459 2466 2459 query29 1020 408 401 401 query30 302 148 150 148 query31 1008 796 844 796 query32 99 54 59 54 query33 772 286 289 286 query34 1011 501 522 501 query35 857 714 728 714 query36 1099 968 960 960 query37 274 73 84 73 query38 4425 4181 4286 4181 query39 1617 1456 1443 1443 query40 284 96 102 96 query41 46 43 44 43 query42 120 96 97 96 query43 562 501 508 501 query44 1268 801 813 801 query45 183 165 162 162 query46 1160 684 711 684 query47 1939 1801 1850 1801 query48 405 313 304 304 query49 1292 391 379 379 query50 815 381 391 381 query51 7343 7128 7018 7018 query52 98 91 88 88 query53 254 177 179 177 query54 1223 400 413 400 query55 78 75 85 75 query56 251 247 232 232 query57 1325 1166 1169 1166 query58 235 244 210 210 query59 3425 3238 3057 3057 query60 275 242 246 242 query61 140 104 109 104 query62 882 665 677 665 query63 211 195 186 186 query64 5145 644 672 644 query65 3329 3204 3242 3204 query66 1411 318 320 318 query67 16321 15830 15709 15709 query68 5006 545 576 545 query69 420 249 259 249 query70 1216 1079 1146 1079 query71 328 242 250 242 query72 6343 4030 4010 4010 query73 769 362 357 357 query74 10539 8990 9150 8990 query75 3483 2697 2677 2677 query76 2986 1157 1153 1153 query77 511 272 277 272 query78 10410 9464 9459 9459 query79 2247 615 606 606 query80 1118 423 443 423 query81 547 229 234 229 query82 908 116 117 116 query83 251 148 144 144 query84 237 73 72 72 query85 1382 293 288 288 query86 433 280 302 280 query87 4739 4646 4481 4481 query88 3779 2224 2179 2179 query89 400 291 295 291 query90 2164 188 191 188 query91 138 104 102 102 query92 62 50 53 50 query93 1602 543 555 543 query94 1124 284 296 284 query95 361 249 247 247 query96 621 280 283 280 query97 2891 2675 2653 2653 query98 220 211 215 211 query99 1547 1306 1318 1306 Total cold run time: 307629 ms Total hot run time: 191241 ms ```
doris-robot commented 1 day ago
ClickBench: Total hot run time: 32.92 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit b14cd87a133700eaeadc0a17dc78c69c1e1eb82e, data reload: false query1 0.03 0.03 0.03 query2 0.07 0.03 0.03 query3 0.23 0.08 0.07 query4 1.62 0.10 0.11 query5 0.44 0.40 0.40 query6 1.14 0.66 0.66 query7 0.02 0.02 0.01 query8 0.04 0.03 0.03 query9 0.56 0.52 0.51 query10 0.56 0.56 0.56 query11 0.14 0.09 0.10 query12 0.14 0.12 0.11 query13 0.62 0.61 0.61 query14 2.81 2.73 2.73 query15 0.90 0.82 0.83 query16 0.39 0.37 0.39 query17 1.00 1.02 1.00 query18 0.24 0.21 0.22 query19 1.95 1.89 2.00 query20 0.02 0.02 0.01 query21 15.38 0.61 0.60 query22 2.76 2.28 1.80 query23 16.98 1.00 0.80 query24 3.24 1.21 2.13 query25 0.28 0.11 0.20 query26 0.50 0.15 0.14 query27 0.04 0.04 0.04 query28 9.62 1.12 1.08 query29 12.52 3.21 3.17 query30 0.26 0.06 0.06 query31 2.87 0.39 0.38 query32 3.27 0.46 0.46 query33 3.07 3.07 3.16 query34 17.04 4.45 4.48 query35 4.54 4.55 4.53 query36 0.64 0.50 0.48 query37 0.09 0.06 0.06 query38 0.05 0.04 0.03 query39 0.03 0.02 0.02 query40 0.15 0.14 0.12 query41 0.08 0.03 0.02 query42 0.04 0.02 0.02 query43 0.03 0.03 0.03 Total cold run time: 106.4 s Total hot run time: 32.92 s ```
qidaye commented 20 hours ago

run buildall

doris-robot commented 19 hours ago
TPC-H: Total hot run time: 40258 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 9f063ffe153f96436faa24cdbcdf3389146ebc60, data reload: false ------ Round 1 ---------------------------------- q1 17619 7564 7316 7316 q2 2049 184 189 184 q3 10527 1162 1220 1162 q4 10209 727 731 727 q5 7604 2854 2772 2772 q6 244 153 152 152 q7 1009 656 625 625 q8 9461 1917 1945 1917 q9 8272 6443 6454 6443 q10 7080 2330 2372 2330 q11 454 269 264 264 q12 544 222 220 220 q13 17787 3033 3002 3002 q14 255 209 207 207 q15 577 548 531 531 q16 695 602 595 595 q17 999 556 602 556 q18 7448 6694 6640 6640 q19 1359 1059 899 899 q20 476 186 188 186 q21 3967 3222 3217 3217 q22 396 313 324 313 Total cold run time: 109031 ms Total hot run time: 40258 ms ----- Round 2, with runtime_filter_mode=off ----- q1 7368 7269 7295 7269 q2 328 226 231 226 q3 2944 2804 2802 2802 q4 1994 1724 1761 1724 q5 5429 5444 5472 5444 q6 224 142 142 142 q7 2116 1693 1750 1693 q8 3308 3431 3424 3424 q9 8533 8557 8585 8557 q10 3518 3467 3467 3467 q11 603 496 509 496 q12 835 567 598 567 q13 11305 3013 3036 3013 q14 303 267 262 262 q15 563 514 505 505 q16 665 661 639 639 q17 1814 1593 1568 1568 q18 7710 7377 7592 7377 q19 1684 1499 1465 1465 q20 2046 1797 1839 1797 q21 5361 5272 5024 5024 q22 603 570 564 564 Total cold run time: 69254 ms Total hot run time: 58025 ms ```
doris-robot commented 19 hours ago
TPC-DS: Total hot run time: 191637 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 9f063ffe153f96436faa24cdbcdf3389146ebc60, data reload: false query1 869 389 392 389 query2 4628 2085 2088 2085 query3 4213 222 207 207 query4 33406 23606 23470 23470 query5 3234 455 445 445 query6 246 177 175 175 query7 3840 297 308 297 query8 273 225 227 225 query9 8165 2670 2665 2665 query10 420 256 256 256 query11 18042 15198 15400 15198 query12 153 107 103 103 query13 1334 414 427 414 query14 9563 7406 7506 7406 query15 289 179 196 179 query16 7404 493 465 465 query17 1391 595 545 545 query18 1951 286 283 283 query19 319 145 146 145 query20 116 113 108 108 query21 215 106 101 101 query22 4607 4558 4455 4455 query23 34929 34020 34159 34020 query24 11403 2394 2490 2394 query25 593 387 379 379 query26 1616 143 149 143 query27 2601 269 288 269 query28 6911 2449 2431 2431 query29 917 412 405 405 query30 281 146 156 146 query31 1038 836 792 792 query32 74 60 56 56 query33 704 286 276 276 query34 942 516 534 516 query35 863 699 725 699 query36 1083 932 941 932 query37 254 80 72 72 query38 4451 4190 4254 4190 query39 1507 1429 1441 1429 query40 276 99 96 96 query41 48 44 44 44 query42 113 95 96 95 query43 535 491 481 481 query44 1218 835 814 814 query45 185 163 163 163 query46 1151 715 704 704 query47 1983 1906 1879 1879 query48 408 318 302 302 query49 1026 382 393 382 query50 835 374 382 374 query51 7268 7114 7117 7114 query52 96 87 92 87 query53 253 181 176 176 query54 740 408 404 404 query55 82 81 96 81 query56 263 248 244 244 query57 1329 1199 1182 1182 query58 236 219 212 212 query59 3250 3136 2926 2926 query60 293 238 250 238 query61 113 111 113 111 query62 840 675 678 675 query63 212 185 185 185 query64 4504 665 635 635 query65 3283 3205 3260 3205 query66 1153 342 309 309 query67 16176 15888 15686 15686 query68 4928 542 560 542 query69 417 253 257 253 query70 1150 1144 1149 1144 query71 349 252 250 250 query72 6221 4068 4230 4068 query73 765 356 372 356 query74 10338 9097 9007 9007 query75 3449 2658 2712 2658 query76 2707 1011 1097 1011 query77 464 316 278 278 query78 10586 9447 9529 9447 query79 1092 589 596 589 query80 721 427 447 427 query81 554 232 230 230 query82 654 118 121 118 query83 209 153 149 149 query84 251 69 68 68 query85 978 294 298 294 query86 325 306 296 296 query87 4666 4619 4495 4495 query88 3146 2258 2221 2221 query89 404 297 305 297 query90 2139 189 194 189 query91 139 105 104 104 query92 62 57 51 51 query93 1040 529 533 529 query94 943 284 258 258 query95 345 253 246 246 query96 618 280 280 280 query97 2859 2697 2697 2697 query98 213 188 192 188 query99 1556 1346 1328 1328 Total cold run time: 287002 ms Total hot run time: 191637 ms ```
doris-robot commented 19 hours ago
ClickBench: Total hot run time: 32.68 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 9f063ffe153f96436faa24cdbcdf3389146ebc60, data reload: false query1 0.03 0.03 0.03 query2 0.06 0.04 0.03 query3 0.24 0.07 0.08 query4 1.59 0.10 0.11 query5 0.41 0.40 0.40 query6 1.14 0.66 0.66 query7 0.02 0.02 0.01 query8 0.04 0.03 0.03 query9 0.56 0.51 0.48 query10 0.55 0.56 0.56 query11 0.14 0.11 0.10 query12 0.15 0.11 0.11 query13 0.61 0.61 0.60 query14 2.80 2.83 2.80 query15 0.89 0.83 0.82 query16 0.38 0.38 0.37 query17 1.04 0.95 0.99 query18 0.23 0.21 0.21 query19 1.98 1.77 1.83 query20 0.02 0.01 0.02 query21 15.40 0.63 0.59 query22 2.84 2.21 2.00 query23 16.90 1.10 0.84 query24 2.99 1.80 0.98 query25 0.25 0.19 0.14 query26 0.44 0.13 0.13 query27 0.04 0.04 0.05 query28 10.28 1.09 1.07 query29 12.57 3.28 3.31 query30 0.25 0.06 0.07 query31 2.88 0.38 0.38 query32 3.27 0.46 0.48 query33 3.01 2.96 3.01 query34 16.94 4.47 4.44 query35 4.53 4.46 4.42 query36 0.67 0.49 0.48 query37 0.09 0.06 0.06 query38 0.04 0.03 0.04 query39 0.03 0.02 0.02 query40 0.16 0.13 0.12 query41 0.08 0.03 0.02 query42 0.04 0.02 0.02 query43 0.04 0.03 0.03 Total cold run time: 106.62 s Total hot run time: 32.68 s ```
github-actions[bot] commented 18 hours ago

PR approved by at least one committer and no changes requested.

github-actions[bot] commented 18 hours ago

PR approved by anyone and no changes requested.