search

apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
12.77k stars 3.28k forks source link

[opt](primary key bf) enhance primary key bloomfilter by fixed slice type #44397

Closed airborne12 closed 10 hours ago

airborne12 commented 16 hours ago

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Currently, the primary key Bloom filter index can be created with any data type. However, when adding values, it only supports slice values. This inconsistency may lead to potential misuse or future issues.

Release note

None

Check List (For Author)

Check List (For Reviewer who merge this PR)

doris-robot commented 16 hours ago

Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?
github-actions[bot] commented 16 hours ago

clang-tidy review says "All clean, LGTM! :+1:"

airborne12 commented 16 hours ago

run buildall

github-actions[bot] commented 16 hours ago

PR approved by at least one committer and no changes requested.

github-actions[bot] commented 16 hours ago

PR approved by anyone and no changes requested.

doris-robot commented 16 hours ago
TPC-H: Total hot run time: 40185 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 75e28013c4a76658a1b043af209d87c54a822aad, data reload: false ------ Round 1 ---------------------------------- q1 17601 7492 7351 7351 q2 2034 177 174 174 q3 10544 1146 1189 1146 q4 10220 783 755 755 q5 7625 2748 2652 2652 q6 246 150 149 149 q7 997 612 625 612 q8 9273 1880 1933 1880 q9 6539 6400 6437 6400 q10 6995 2364 2330 2330 q11 456 258 277 258 q12 423 216 222 216 q13 17772 3009 3082 3009 q14 242 206 210 206 q15 571 534 524 524 q16 650 588 576 576 q17 979 628 542 542 q18 7602 6673 6831 6673 q19 1342 1075 998 998 q20 479 182 180 180 q21 3967 3247 3323 3247 q22 375 307 318 307 Total cold run time: 106932 ms Total hot run time: 40185 ms ----- Round 2, with runtime_filter_mode=off ----- q1 7334 7299 7260 7260 q2 326 235 228 228 q3 2957 2829 3106 2829 q4 2173 1828 1795 1795 q5 5636 5712 5660 5660 q6 223 140 134 134 q7 2221 1765 1847 1765 q8 3410 3531 3503 3503 q9 8903 8840 8967 8840 q10 3608 3536 3556 3536 q11 604 501 504 501 q12 825 618 603 603 q13 14110 3253 3188 3188 q14 293 269 278 269 q15 561 514 515 514 q16 670 649 619 619 q17 1792 1596 1562 1562 q18 7867 7522 7552 7522 q19 1677 1588 1448 1448 q20 2071 1801 1822 1801 q21 5285 5277 5178 5178 q22 628 544 547 544 Total cold run time: 73174 ms Total hot run time: 59299 ms ```
doris-robot commented 15 hours ago

TeamCity be ut coverage result: Function Coverage: 38.04% (9904/26034) Line Coverage: 29.22% (82855/283528) Region Coverage: 28.35% (42551/150087) Branch Coverage: 24.92% (21577/86598) Coverage Report: http://coverage.selectdb-in.cc/coverage/75e28013c4a76658a1b043af209d87c54a822aad_75e28013c4a76658a1b043af209d87c54a822aad/report/index.html

doris-robot commented 15 hours ago
TPC-DS: Total hot run time: 191829 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 75e28013c4a76658a1b043af209d87c54a822aad, data reload: false query1 974 388 377 377 query2 6531 2104 2097 2097 query3 6711 215 206 206 query4 34165 23800 23641 23641 query5 4383 478 442 442 query6 291 186 183 183 query7 4639 299 302 299 query8 302 234 236 234 query9 9204 2683 2681 2681 query10 478 281 252 252 query11 18265 15264 15344 15264 query12 155 104 107 104 query13 1663 429 425 425 query14 8972 7407 7350 7350 query15 293 180 177 177 query16 8193 475 449 449 query17 1827 554 546 546 query18 2140 290 295 290 query19 359 147 151 147 query20 124 114 110 110 query21 208 102 101 101 query22 4614 4446 4476 4446 query23 34932 34638 33914 33914 query24 11383 2477 2362 2362 query25 613 384 385 384 query26 1526 145 146 145 query27 2856 280 281 280 query28 8095 2435 2422 2422 query29 787 404 409 404 query30 309 154 148 148 query31 998 794 816 794 query32 97 61 61 61 query33 771 288 281 281 query34 1034 496 519 496 query35 886 728 716 716 query36 1088 947 942 942 query37 241 76 78 76 query38 4453 4361 4281 4281 query39 1523 1440 1432 1432 query40 277 99 101 99 query41 49 46 44 44 query42 117 102 99 99 query43 533 484 494 484 query44 1221 830 819 819 query45 186 163 165 163 query46 1129 707 680 680 query47 2001 1875 1857 1857 query48 418 322 318 318 query49 1280 391 386 386 query50 800 393 377 377 query51 7260 7156 7030 7030 query52 103 88 88 88 query53 253 186 177 177 query54 1132 407 428 407 query55 85 75 78 75 query56 259 243 246 243 query57 1307 1186 1177 1177 query58 236 220 222 220 query59 3180 2963 2951 2951 query60 287 247 252 247 query61 112 107 108 107 query62 877 701 667 667 query63 213 198 188 188 query64 4942 670 638 638 query65 3303 3206 3266 3206 query66 1299 319 319 319 query67 16331 15990 15776 15776 query68 5325 572 553 553 query69 427 259 270 259 query70 1147 1153 1166 1153 query71 335 246 245 245 query72 6254 4064 4021 4021 query73 778 376 360 360 query74 10392 9002 9118 9002 query75 3441 2676 2674 2674 query76 3176 994 1039 994 query77 507 266 268 266 query78 10649 9425 9393 9393 query79 2480 600 603 600 query80 1237 431 459 431 query81 544 230 281 230 query82 673 116 119 116 query83 252 151 149 149 query84 243 72 75 72 query85 1734 306 297 297 query86 491 304 274 274 query87 4759 4685 4526 4526 query88 4009 2256 2205 2205 query89 411 292 309 292 query90 2105 187 190 187 query91 136 108 107 107 query92 67 51 55 51 query93 1739 552 552 552 query94 996 300 315 300 query95 367 263 254 254 query96 625 279 282 279 query97 2894 2681 2741 2681 query98 217 199 199 199 query99 1590 1323 1306 1306 Total cold run time: 305273 ms Total hot run time: 191829 ms ```
doris-robot commented 15 hours ago
ClickBench: Total hot run time: 32.47 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 75e28013c4a76658a1b043af209d87c54a822aad, data reload: false query1 0.04 0.03 0.03 query2 0.06 0.03 0.03 query3 0.23 0.08 0.07 query4 1.61 0.10 0.10 query5 0.43 0.39 0.41 query6 1.16 0.67 0.65 query7 0.02 0.02 0.02 query8 0.04 0.03 0.03 query9 0.56 0.52 0.51 query10 0.57 0.57 0.56 query11 0.14 0.10 0.11 query12 0.14 0.11 0.12 query13 0.61 0.61 0.61 query14 2.85 2.84 2.86 query15 0.90 0.84 0.82 query16 0.38 0.37 0.36 query17 1.06 1.09 1.00 query18 0.23 0.21 0.22 query19 1.96 1.85 2.02 query20 0.01 0.01 0.01 query21 15.35 0.61 0.58 query22 2.49 2.53 1.92 query23 17.09 0.86 0.78 query24 3.14 0.71 1.59 query25 0.21 0.10 0.14 query26 0.55 0.13 0.13 query27 0.05 0.04 0.04 query28 10.46 1.10 1.07 query29 12.60 3.26 3.21 query30 0.25 0.06 0.05 query31 2.85 0.38 0.38 query32 3.30 0.48 0.47 query33 3.00 3.04 3.18 query34 17.14 4.43 4.50 query35 4.49 4.43 4.49 query36 0.67 0.51 0.51 query37 0.09 0.06 0.06 query38 0.05 0.04 0.04 query39 0.03 0.02 0.02 query40 0.16 0.13 0.12 query41 0.09 0.03 0.03 query42 0.04 0.03 0.02 query43 0.04 0.03 0.03 Total cold run time: 107.14 s Total hot run time: 32.47 s ```