search

apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
12.77k stars 3.29k forks source link

branch-3.0: [opt](scan) Release instances of Segment to avoid consuming a large amount of memory in ParallelScannerBuilder #44189 #44445

Closed github-actions[bot] closed 3 hours ago

github-actions[bot] commented 6 hours ago

Cherry-picked from #44189

doris-robot commented 6 hours ago

Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?
doris-robot commented 6 hours ago

run buildall

github-actions[bot] commented 6 hours ago

clang-tidy review says "All clean, LGTM! :+1:"

doris-robot commented 5 hours ago
TPC-H: Total hot run time: 40213 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 634b02802381fc47fca95f7a00a0124a0c4f9b72, data reload: false ------ Round 1 ---------------------------------- q1 17619 7294 7237 7237 q2 2066 159 160 159 q3 10819 1083 1142 1083 q4 10536 776 670 670 q5 7736 2816 2799 2799 q6 231 142 143 142 q7 982 600 603 600 q8 9578 1925 2004 1925 q9 7794 6387 6370 6370 q10 6957 2287 2301 2287 q11 448 259 258 258 q12 400 209 206 206 q13 17751 2952 3011 2952 q14 250 209 205 205 q15 568 510 519 510 q16 663 601 587 587 q17 974 600 479 479 q18 7240 6520 6586 6520 q19 1379 1074 1038 1038 q20 478 193 194 193 q21 3869 3036 3095 3036 q22 1100 996 957 957 Total cold run time: 109438 ms Total hot run time: 40213 ms ----- Round 2, with runtime_filter_mode=off ----- q1 7272 7194 7172 7172 q2 325 227 222 222 q3 2841 2886 2861 2861 q4 1984 1800 1823 1800 q5 5699 5989 5702 5702 q6 220 142 139 139 q7 2198 1738 1737 1737 q8 3267 3505 3490 3490 q9 8827 8869 8805 8805 q10 3544 3496 3500 3496 q11 603 500 498 498 q12 825 590 622 590 q13 16500 3112 3122 3112 q14 304 259 270 259 q15 558 521 526 521 q16 687 670 655 655 q17 1816 1651 1601 1601 q18 8231 7700 7386 7386 q19 3322 1511 1568 1511 q20 2073 1848 1833 1833 q21 5345 5441 5270 5270 q22 1165 1017 1065 1017 Total cold run time: 77606 ms Total hot run time: 59677 ms ```
doris-robot commented 5 hours ago
TPC-DS: Total hot run time: 195328 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 634b02802381fc47fca95f7a00a0124a0c4f9b72, data reload: false query1 1222 902 905 902 query2 6221 2088 2055 2055 query3 10884 4088 3793 3793 query4 68944 27217 23469 23469 query5 5671 465 452 452 query6 446 191 183 183 query7 6038 310 318 310 query8 317 222 223 222 query9 9039 2682 2672 2672 query10 507 263 246 246 query11 17943 15269 15782 15269 query12 175 110 102 102 query13 1614 424 425 424 query14 10535 7561 6950 6950 query15 212 182 180 180 query16 7359 523 484 484 query17 1084 562 567 562 query18 1924 335 301 301 query19 195 155 151 151 query20 117 107 110 107 query21 226 103 99 99 query22 4827 4453 4536 4453 query23 34512 34006 34279 34006 query24 6061 2940 2884 2884 query25 537 416 426 416 query26 685 169 168 168 query27 1810 295 302 295 query28 4016 2578 2561 2561 query29 725 456 417 417 query30 232 155 152 152 query31 1018 809 800 800 query32 65 58 55 55 query33 426 276 269 269 query34 910 488 503 488 query35 819 760 745 745 query36 1103 925 952 925 query37 114 69 69 69 query38 4054 3960 3999 3960 query39 1532 1469 1472 1469 query40 203 102 94 94 query41 46 51 45 45 query42 112 96 101 96 query43 532 498 491 491 query44 1166 800 795 795 query45 183 173 166 166 query46 1143 745 729 729 query47 2014 1895 1947 1895 query48 464 357 382 357 query49 717 382 366 366 query50 843 406 419 406 query51 7411 7105 6991 6991 query52 96 88 94 88 query53 258 181 186 181 query54 537 439 432 432 query55 83 75 76 75 query56 250 227 223 223 query57 1227 1097 1100 1097 query58 214 199 222 199 query59 3094 2891 2906 2891 query60 255 252 251 251 query61 101 100 101 100 query62 768 622 655 622 query63 210 192 181 181 query64 1776 670 629 629 query65 3269 3153 3158 3153 query66 714 300 299 299 query67 15807 15296 15289 15289 query68 4597 542 545 542 query69 424 248 258 248 query70 1173 1111 1106 1106 query71 408 258 261 258 query72 6505 3892 3911 3892 query73 756 344 339 339 query74 10168 8952 8999 8952 query75 3268 2580 2653 2580 query76 2159 1137 1196 1137 query77 507 260 277 260 query78 10663 9768 9418 9418 query79 8941 590 588 588 query80 2191 425 411 411 query81 555 240 244 240 query82 1406 116 120 116 query83 296 135 134 134 query84 285 75 75 75 query85 1874 291 291 291 query86 474 301 297 297 query87 4483 4283 4258 4258 query88 5795 2343 2384 2343 query89 556 286 293 286 query90 2121 186 180 180 query91 180 143 137 137 query92 59 46 47 46 query93 6755 538 525 525 query94 837 288 311 288 query95 342 242 251 242 query96 623 273 278 273 query97 3354 3172 3144 3144 query98 212 201 205 201 query99 1754 1315 1320 1315 Total cold run time: 341333 ms Total hot run time: 195328 ms ```