[pick20][opt](mow) reduce memory usage for mow table compaction

[cambyzju]

cherry-pick https://github.com/apache/doris/pull/36865 to branch-2.0

[doris-robot]

Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.

[github-actions[bot]]

clang-tidy review says "All clean, LGTM! :+1:"

[github-actions[bot]]

clang-tidy review says "All clean, LGTM! :+1:"

[cambyzju]

run buildall

[github-actions[bot]]

clang-tidy review says "All clean, LGTM! :+1:"

[doris-robot]

TPC-H: Total hot run time: 50189 ms

``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 56be53894a29899bd98c38a11c58e7cbf59977cb, data reload: false ------ Round 1 ---------------------------------- q1 18023 4440 4376 4376 q2 2067 158 149 149 q3 10428 1908 1918 1908 q4 10188 1296 1362 1296 q5 8502 4017 3910 3910 q6 234 125 150 125 q7 2050 1618 1619 1618 q8 9308 2783 2765 2765 q9 10807 10352 10332 10332 q10 8638 3511 3552 3511 q11 426 253 243 243 q12 479 303 301 301 q13 18338 4010 3976 3976 q14 349 323 333 323 q15 505 465 456 456 q16 699 568 585 568 q17 1163 969 978 969 q18 7362 6956 6842 6842 q19 1822 1695 1652 1652 q20 553 312 297 297 q21 4494 4125 4162 4125 q22 525 450 447 447 Total cold run time: 116960 ms Total hot run time: 50189 ms ----- Round 2, with runtime_filter_mode=off ----- q1 4312 4327 4314 4314 q2 323 233 228 228 q3 4239 4170 4158 4158 q4 2774 2753 2751 2751 q5 7171 7088 7057 7057 q6 239 120 119 119 q7 3248 2863 2847 2847 q8 4372 4489 4511 4489 q9 17013 17004 16770 16770 q10 4298 4270 4301 4270 q11 745 705 697 697 q12 1032 855 865 855 q13 7309 3710 3765 3710 q14 453 431 422 422 q15 518 453 467 453 q16 739 684 679 679 q17 3872 3900 3815 3815 q18 8855 8799 8843 8799 q19 1747 1723 1663 1663 q20 2432 2162 2139 2139 q21 8591 8567 8433 8433 q22 1042 1036 972 972 Total cold run time: 85324 ms Total hot run time: 79640 ms ```

[doris-robot]

TeamCity be ut coverage result: Function Coverage: 37.88% (8112/21417) Line Coverage: 29.54% (66432/224861) Region Coverage: 29.00% (34228/118009) Branch Coverage: 24.86% (17567/70658) Coverage Report: http://coverage.selectdb-in.cc/coverage/56be53894a29899bd98c38a11c58e7cbf59977cb_56be53894a29899bd98c38a11c58e7cbf59977cb/report/index.html

[doris-robot]

TPC-DS: Total hot run time: 204334 ms

``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 56be53894a29899bd98c38a11c58e7cbf59977cb, data reload: false query1 941 426 380 380 query2 6524 2717 2668 2668 query3 6923 211 206 206 query4 21417 18121 18092 18092 query5 19730 6587 6543 6543 query6 299 223 227 223 query7 4162 293 296 293 query8 437 432 446 432 query9 3109 2701 2608 2608 query10 422 296 301 296 query11 11342 10805 10700 10700 query12 124 84 73 73 query13 5600 698 681 681 query14 17419 13586 13407 13407 query15 419 242 252 242 query16 6462 296 258 258 query17 1743 1427 902 902 query18 2284 415 412 412 query19 204 155 151 151 query20 80 77 80 77 query21 191 93 102 93 query22 5218 5110 5051 5051 query23 32717 31940 32127 31940 query24 7041 6548 6538 6538 query25 527 418 426 418 query26 517 162 162 162 query27 1817 301 297 297 query28 6164 2370 2322 2322 query29 2906 2713 2671 2671 query30 237 164 169 164 query31 880 740 745 740 query32 71 63 60 60 query33 401 265 249 249 query34 848 467 480 467 query35 1103 921 894 894 query36 1259 1095 1290 1095 query37 89 62 59 59 query38 3123 2917 2932 2917 query39 1372 1349 1341 1341 query40 209 95 92 92 query41 47 45 44 44 query42 88 90 83 83 query43 928 681 651 651 query44 1110 729 738 729 query45 248 235 234 234 query46 1231 978 958 958 query47 1834 1845 1691 1691 query48 1023 708 697 697 query49 625 368 368 368 query50 887 611 589 589 query51 4697 4639 4714 4639 query52 92 84 90 84 query53 448 332 327 327 query54 2723 2454 2437 2437 query55 92 82 87 82 query56 239 225 201 201 query57 1156 1050 1076 1050 query58 218 199 205 199 query59 4251 4113 4104 4104 query60 211 201 206 201 query61 97 94 97 94 query62 829 458 463 458 query63 486 354 341 341 query64 2559 1536 1481 1481 query65 3642 3578 3560 3560 query66 797 386 375 375 query67 17181 15850 15536 15536 query68 8392 653 641 641 query69 583 352 340 340 query70 1600 1437 1305 1305 query71 397 308 339 308 query72 6547 3532 3520 3520 query73 736 317 313 313 query74 6250 5888 5819 5819 query75 4615 3662 3678 3662 query76 4784 1182 1185 1182 query77 661 253 251 251 query78 13093 11851 12764 11851 query79 12135 626 621 621 query80 1180 391 402 391 query81 503 243 231 231 query82 642 100 101 100 query83 167 134 128 128 query84 264 72 69 69 query85 960 325 326 325 query86 337 291 287 287 query87 3322 3013 3068 3013 query88 5657 2310 2304 2304 query89 353 282 264 264 query90 1980 201 220 201 query91 177 141 142 141 query92 60 53 51 51 query93 3478 568 584 568 query94 700 206 201 201 query95 1103 1070 1061 1061 query96 624 334 327 327 query97 6516 6354 6360 6354 query98 196 173 172 172 query99 2964 940 906 906 Total cold run time: 315046 ms Total hot run time: 204334 ms ```

[doris-robot]

ClickBench: Total hot run time: 30.65 s

``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 56be53894a29899bd98c38a11c58e7cbf59977cb, data reload: false query1 0.02 0.02 0.02 query2 0.07 0.02 0.02 query3 0.26 0.05 0.05 query4 1.77 0.08 0.07 query5 0.53 0.53 0.53 query6 1.24 0.61 0.60 query7 0.01 0.01 0.01 query8 0.03 0.02 0.02 query9 0.53 0.50 0.47 query10 0.53 0.54 0.54 query11 0.12 0.09 0.09 query12 0.12 0.09 0.08 query13 0.62 0.61 0.62 query14 0.78 0.79 0.79 query15 0.77 0.75 0.78 query16 0.37 0.38 0.38 query17 1.01 0.99 1.03 query18 0.22 0.25 0.23 query19 1.97 1.78 1.86 query20 0.02 0.01 0.02 query21 15.45 0.56 0.54 query22 2.01 1.94 1.80 query23 17.29 1.08 1.03 query24 4.66 0.64 1.32 query25 0.33 0.08 0.04 query26 0.57 0.15 0.14 query27 0.05 0.03 0.04 query28 8.52 0.73 0.70 query29 12.71 2.37 2.35 query30 0.62 0.54 0.53 query31 2.81 0.39 0.37 query32 3.38 0.49 0.48 query33 3.09 3.11 3.08 query34 15.23 4.79 4.78 query35 4.84 4.84 4.85 query36 1.06 1.01 1.01 query37 0.06 0.05 0.05 query38 0.03 0.02 0.02 query39 0.02 0.02 0.01 query40 0.16 0.14 0.14 query41 0.06 0.01 0.02 query42 0.02 0.02 0.01 query43 0.02 0.01 0.01 Total cold run time: 103.98 s Total hot run time: 30.65 s ```

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 56be53894a29899bd98c38a11c58e7cbf59977cb with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

[cambyzju]

run external

[cambyzju]

run buildall

[github-actions[bot]]

clang-tidy review says "All clean, LGTM! :+1:"

[doris-robot]

TPC-H: Total hot run time: 50341 ms

``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 3f618f87adc4860245daf2f3d0402294526fd903, data reload: false ------ Round 1 ---------------------------------- q1 17697 4980 4392 4392 q2 2076 164 151 151 q3 10529 1897 1939 1897 q4 10347 1274 1338 1274 q5 8460 3984 3965 3965 q6 261 126 125 125 q7 2063 1606 1591 1591 q8 9559 2761 2728 2728 q9 10980 10612 10488 10488 q10 8645 3537 3547 3537 q11 424 247 253 247 q12 470 304 308 304 q13 18346 3959 3995 3959 q14 374 329 328 328 q15 505 464 460 460 q16 653 566 569 566 q17 1148 967 972 967 q18 7295 6987 6963 6963 q19 1800 1616 1630 1616 q20 529 306 317 306 q21 4517 4149 4053 4053 q22 519 440 424 424 Total cold run time: 117197 ms Total hot run time: 50341 ms ----- Round 2, with runtime_filter_mode=off ----- q1 4316 4274 4299 4274 q2 319 226 220 220 q3 4192 4141 4168 4141 q4 2739 2737 2742 2737 q5 7255 7189 7171 7171 q6 244 121 121 121 q7 3296 2849 2800 2800 q8 4370 4493 4540 4493 q9 17467 17058 17270 17058 q10 4254 4272 4256 4256 q11 738 693 688 688 q12 1044 864 869 864 q13 7645 3716 3731 3716 q14 455 429 427 427 q15 505 455 454 454 q16 736 678 667 667 q17 3907 3806 3930 3806 q18 8924 8765 8751 8751 q19 1703 1745 1653 1653 q20 2383 2167 2146 2146 q21 8534 8459 8498 8459 q22 1084 990 953 953 Total cold run time: 86110 ms Total hot run time: 79855 ms ```

[doris-robot]

TeamCity be ut coverage result: Function Coverage: 37.89% (8115/21417) Line Coverage: 29.56% (66467/224862) Region Coverage: 29.03% (34257/118012) Branch Coverage: 24.90% (17591/70660) Coverage Report: http://coverage.selectdb-in.cc/coverage/3f618f87adc4860245daf2f3d0402294526fd903_3f618f87adc4860245daf2f3d0402294526fd903/report/index.html

[doris-robot]

TPC-DS: Total hot run time: 203146 ms

``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 3f618f87adc4860245daf2f3d0402294526fd903, data reload: false query1 932 419 379 379 query2 6524 2840 2668 2668 query3 6920 207 201 201 query4 20421 18090 18015 18015 query5 19743 6525 6528 6525 query6 299 220 233 220 query7 4152 304 318 304 query8 414 471 395 395 query9 3083 2649 2594 2594 query10 405 315 311 311 query11 11434 10621 10701 10621 query12 122 77 75 75 query13 5594 703 698 698 query14 17289 13454 13240 13240 query15 370 240 249 240 query16 6482 287 263 263 query17 1719 1483 869 869 query18 2330 414 405 405 query19 209 151 151 151 query20 80 78 84 78 query21 187 97 94 94 query22 5239 5084 5050 5050 query23 32484 32063 31914 31914 query24 7042 6534 6579 6534 query25 520 428 417 417 query26 541 158 157 157 query27 1879 288 294 288 query28 6158 2371 2308 2308 query29 2922 2711 2663 2663 query30 244 166 167 166 query31 922 709 736 709 query32 67 67 61 61 query33 412 263 242 242 query34 843 480 501 480 query35 1144 959 949 949 query36 1396 1055 1056 1055 query37 91 58 58 58 query38 3129 2939 2982 2939 query39 1375 1335 1324 1324 query40 209 97 92 92 query41 45 44 43 43 query42 80 86 79 79 query43 664 870 659 659 query44 1113 724 731 724 query45 254 235 237 235 query46 1235 947 942 942 query47 1800 1626 1708 1626 query48 1028 718 700 700 query49 617 367 367 367 query50 873 609 622 609 query51 4781 4674 4774 4674 query52 95 72 80 72 query53 443 328 325 325 query54 2651 2490 2486 2486 query55 86 87 82 82 query56 242 218 228 218 query57 1194 1136 1109 1109 query58 226 207 214 207 query59 4216 3767 3915 3767 query60 218 203 203 203 query61 101 96 101 96 query62 889 499 461 461 query63 485 350 340 340 query64 2601 1555 1496 1496 query65 3631 3555 3556 3555 query66 783 379 382 379 query67 18439 15310 15757 15310 query68 8465 686 640 640 query69 576 344 351 344 query70 1563 1481 1339 1339 query71 409 311 324 311 query72 6497 3484 3527 3484 query73 740 324 314 314 query74 6367 5800 5793 5793 query75 4594 3749 3701 3701 query76 4737 1163 1099 1099 query77 623 258 253 253 query78 12721 11718 11958 11718 query79 8819 650 637 637 query80 2763 410 401 401 query81 514 235 230 230 query82 1709 102 97 97 query83 179 135 131 131 query84 265 71 75 71 query85 1361 319 327 319 query86 363 287 291 287 query87 3217 2996 3064 2996 query88 5173 2277 2314 2277 query89 389 303 273 273 query90 1745 209 218 209 query91 204 139 165 139 query92 58 53 52 52 query93 6200 538 552 538 query94 913 218 216 216 query95 1140 1072 1076 1072 query96 649 322 326 322 query97 6557 6354 6398 6354 query98 203 182 178 178 query99 2949 835 955 835 Total cold run time: 316778 ms Total hot run time: 203146 ms ```

[doris-robot]

ClickBench: Total hot run time: 31.31 s

``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 3f618f87adc4860245daf2f3d0402294526fd903, data reload: false query1 0.03 0.03 0.02 query2 0.07 0.02 0.02 query3 0.24 0.04 0.04 query4 1.80 0.08 0.08 query5 0.54 0.53 0.52 query6 1.32 0.64 0.60 query7 0.02 0.01 0.01 query8 0.04 0.02 0.02 query9 0.52 0.48 0.49 query10 0.55 0.54 0.55 query11 0.12 0.08 0.09 query12 0.11 0.08 0.08 query13 0.62 0.62 0.62 query14 0.79 0.78 0.79 query15 0.78 0.76 0.76 query16 0.36 0.36 0.37 query17 0.97 1.01 1.01 query18 0.23 0.24 0.24 query19 1.91 1.85 1.83 query20 0.01 0.00 0.01 query21 15.46 0.57 0.53 query22 2.34 2.04 1.65 query23 17.17 1.10 1.14 query24 6.07 1.50 1.34 query25 0.40 0.07 0.06 query26 0.74 0.16 0.15 query27 0.05 0.03 0.04 query28 6.45 0.77 0.78 query29 12.61 2.19 2.19 query30 0.66 0.53 0.53 query31 2.80 0.39 0.38 query32 3.36 0.49 0.50 query33 3.06 3.06 3.11 query34 15.29 4.81 4.80 query35 4.84 4.87 4.88 query36 1.04 1.03 1.01 query37 0.06 0.05 0.05 query38 0.04 0.02 0.02 query39 0.02 0.01 0.01 query40 0.16 0.15 0.14 query41 0.07 0.01 0.02 query42 0.02 0.01 0.01 query43 0.02 0.02 0.02 Total cold run time: 103.76 s Total hot run time: 31.31 s ```

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 3f618f87adc4860245daf2f3d0402294526fd903 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s