search

apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
12.77k stars 3.29k forks source link

[Fix](full compaction) Full compaction should not do ordered data compaction #44359

Closed Yukang-Lian closed 13 hours ago

Yukang-Lian commented 1 day ago

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Problem: For a duplicate table with the following distribution, if it has already completed cumulative compaction and then undergoes full compaction, it will cause a BE core issue. Check failed: new_point == Tablet::K_INVALID_CUMULATIVE_POINT || new_point >= _cumulative_point Unexpected cumulative point: 1087, origin: 2801. "rowsets": [ "[0-386] 0 DATA NONOVERLAPPING 02000000000198aabe4290f2b0f5f35610c08a233a061892 0", "[387-387] 0 DELETE OVERLAP_UNKNOWN 0200000000541310ac4d76e7580a708a2823a4d7a4f06090 0", "[388-388] 0 DELETE OVERLAP_UNKNOWN 0200000000541d76ac4d76e7580a708a2823a4d7a4f06090 0", "[389-389] 0 DELETE OVERLAP_UNKNOWN 0200000000543b4dac4d76e7580a708a2823a4d7a4f06090 0", "[390-390] 0 DELETE OVERLAP_UNKNOWN 02000000005453aeac4d76e7580a708a2823a4d7a4f06090 0", "[391-391] 0 DELETE OVERLAP_UNKNOWN 0200000000546a44ac4d76e7580a708a2823a4d7a4f06090 0", "[392-392] 0 DELETE OVERLAP_UNKNOWN 02000000005480dbac4d76e7580a708a2823a4d7a4f06090 0", "[393-393] 0 DELETE OVERLAP_UNKNOWN 0200000000548cb3ac4d76e7580a708a2823a4d7a4f06090 0", "[394-394] 0 DELETE OVERLAP_UNKNOWN 0200000000549a25ac4d76e7580a708a2823a4d7a4f06090 0", "[395-395] 0 DELETE OVERLAP_UNKNOWN 020000000054b359ac4d76e7580a708a2823a4d7a4f06090 0", "[396-396] 0 DELETE OVERLAP_UNKNOWN 020000000054c19dac4d76e7580a708a2823a4d7a4f06090 0", "[397-397] 0 DELETE OVERLAP_UNKNOWN 020000000054d757ac4d76e7580a708a2823a4d7a4f06090 0", ... "[1085-1085] 0 DELETE OVERLAP_UNKNOWN 02000000002a0b20bd4798638f237008ff42fbca276b52a2 0", "[1087-1506] 1 DATA NONOVERLAPPING 020000000000047e3b452de14ceaad2e78a87526026d2290 326.10 KB", "[1087-1506] 1 DATA NONOVERLAPPING 020000000000047e3b452de14ceaad2e78a87526026d2290 326.10 KB", ... "[2800-2800] 0 DELETE OVERLAP_UNKNOWN 02000000002f12d6bd4798638f237008ff42fbca276b52a2 0"

Reason: The duplicate table will go through ordered data compaction. Due to the special distribution of the table, the input rowset will be cut by the ordered data compaction, resulting in the full compaction only being performed on a part of the rowsets.

Solution: For full compaction, prohibit ordered data compaction.

Release note

None

Check List (For Author)

Check List (For Reviewer who merge this PR)

doris-robot commented 1 day ago

Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?
Yukang-Lian commented 1 day ago

run buildall

Yukang-Lian commented 1 day ago

regression test will be added later.

github-actions[bot] commented 1 day ago

clang-tidy review says "All clean, LGTM! :+1:"

doris-robot commented 1 day ago
TPC-H: Total hot run time: 40098 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 085de8b2f3b2136fc6a4b9745c25da01db2ef5cb, data reload: false ------ Round 1 ---------------------------------- q1 17619 7526 7390 7390 q2 2037 172 172 172 q3 10548 1148 1114 1114 q4 10217 812 782 782 q5 7586 2726 2710 2710 q6 241 153 150 150 q7 983 625 597 597 q8 9255 1848 1939 1848 q9 6492 6398 6419 6398 q10 6971 2283 2353 2283 q11 462 254 280 254 q12 412 225 214 214 q13 17758 3000 3040 3000 q14 245 215 224 215 q15 576 523 542 523 q16 648 585 591 585 q17 982 559 602 559 q18 7491 6674 6739 6674 q19 1326 959 992 959 q20 476 181 190 181 q21 3993 3287 3175 3175 q22 375 315 315 315 Total cold run time: 106693 ms Total hot run time: 40098 ms ----- Round 2, with runtime_filter_mode=off ----- q1 7384 7351 7334 7334 q2 336 234 229 229 q3 2904 2786 2789 2786 q4 1987 1762 1729 1729 q5 5415 5413 5422 5413 q6 223 143 143 143 q7 2109 1719 1686 1686 q8 3254 3421 3424 3421 q9 8535 8530 8555 8530 q10 3535 3530 3432 3432 q11 604 502 507 502 q12 812 573 604 573 q13 11608 3043 3061 3043 q14 306 273 263 263 q15 560 515 512 512 q16 684 636 631 631 q17 1807 1591 1577 1577 q18 7787 7570 7451 7451 q19 1671 1492 1516 1492 q20 2047 1839 1807 1807 q21 5566 5305 5189 5189 q22 627 544 571 544 Total cold run time: 69761 ms Total hot run time: 58287 ms ```
doris-robot commented 1 day ago
TPC-DS: Total hot run time: 192019 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 085de8b2f3b2136fc6a4b9745c25da01db2ef5cb, data reload: false query1 966 409 390 390 query2 6536 2111 2011 2011 query3 6714 217 217 217 query4 34014 23575 23547 23547 query5 4369 460 442 442 query6 284 191 186 186 query7 4641 307 309 307 query8 305 238 228 228 query9 9313 2659 2647 2647 query10 468 258 247 247 query11 18094 15337 15267 15267 query12 159 106 109 106 query13 1661 415 422 415 query14 10270 7330 7037 7037 query15 301 175 178 175 query16 8087 470 466 466 query17 1800 590 571 571 query18 2111 283 295 283 query19 372 156 145 145 query20 118 110 142 110 query21 207 103 100 100 query22 4655 4373 4414 4373 query23 35306 34268 34365 34268 query24 11516 2505 2609 2505 query25 673 383 373 373 query26 1833 151 151 151 query27 2855 278 281 278 query28 8058 2425 2422 2422 query29 1025 422 417 417 query30 314 149 153 149 query31 1043 816 814 814 query32 98 54 55 54 query33 780 296 283 283 query34 1019 503 531 503 query35 851 710 717 710 query36 1081 958 979 958 query37 271 76 73 73 query38 4346 4296 4209 4209 query39 1471 1431 1460 1431 query40 281 107 101 101 query41 46 44 46 44 query42 109 98 98 98 query43 537 484 491 484 query44 1301 813 813 813 query45 187 164 165 164 query46 1154 704 713 704 query47 1982 1836 1822 1822 query48 411 309 324 309 query49 1279 394 389 389 query50 816 406 384 384 query51 7361 7115 7071 7071 query52 97 89 87 87 query53 251 180 182 180 query54 1261 399 420 399 query55 98 76 80 76 query56 257 257 234 234 query57 1301 1197 1193 1193 query58 235 214 216 214 query59 3316 3097 3156 3097 query60 274 254 241 241 query61 110 105 110 105 query62 873 671 695 671 query63 213 188 186 186 query64 5053 651 635 635 query65 3337 3254 3262 3254 query66 1424 321 312 312 query67 16213 15928 15548 15548 query68 4485 564 559 559 query69 401 255 310 255 query70 1180 1155 1103 1103 query71 337 262 251 251 query72 6435 4038 4021 4021 query73 756 348 356 348 query74 10477 9026 9108 9026 query75 3527 2694 2695 2694 query76 2834 1101 1092 1092 query77 507 266 291 266 query78 10623 9511 9493 9493 query79 1629 583 598 583 query80 1226 442 432 432 query81 551 230 243 230 query82 981 124 117 117 query83 253 151 148 148 query84 235 73 78 73 query85 1348 303 301 301 query86 367 300 303 300 query87 4712 4695 4658 4658 query88 3543 2223 2191 2191 query89 408 301 289 289 query90 2117 190 185 185 query91 135 108 104 104 query92 61 50 51 50 query93 1293 539 556 539 query94 1133 295 301 295 query95 353 245 249 245 query96 624 284 278 278 query97 2901 2742 2718 2718 query98 216 205 197 197 query99 1553 1339 1295 1295 Total cold run time: 304963 ms Total hot run time: 192019 ms ```
doris-robot commented 1 day ago

TeamCity be ut coverage result: Function Coverage: 38.03% (9902/26039) Line Coverage: 29.22% (82841/283546) Region Coverage: 28.35% (42550/150085) Branch Coverage: 24.91% (21566/86592) Coverage Report: http://coverage.selectdb-in.cc/coverage/085de8b2f3b2136fc6a4b9745c25da01db2ef5cb_085de8b2f3b2136fc6a4b9745c25da01db2ef5cb/report/index.html

doris-robot commented 1 day ago
ClickBench: Total hot run time: 32.99 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 085de8b2f3b2136fc6a4b9745c25da01db2ef5cb, data reload: false query1 0.04 0.03 0.03 query2 0.08 0.03 0.06 query3 0.23 0.09 0.07 query4 1.62 0.11 0.11 query5 0.43 0.42 0.42 query6 1.16 0.68 0.66 query7 0.01 0.02 0.01 query8 0.04 0.03 0.02 query9 0.56 0.52 0.50 query10 0.56 0.57 0.56 query11 0.14 0.11 0.11 query12 0.14 0.11 0.11 query13 0.61 0.61 0.58 query14 2.71 2.72 2.75 query15 0.91 0.83 0.83 query16 0.40 0.39 0.38 query17 1.07 1.08 1.08 query18 0.22 0.22 0.22 query19 1.91 1.86 2.06 query20 0.01 0.01 0.01 query21 15.37 0.60 0.58 query22 2.66 2.01 2.09 query23 16.87 0.94 0.89 query24 2.78 0.91 1.77 query25 0.34 0.13 0.08 query26 0.42 0.14 0.15 query27 0.03 0.04 0.04 query28 10.59 1.10 1.07 query29 12.58 3.21 3.23 query30 0.25 0.06 0.06 query31 2.86 0.39 0.37 query32 3.28 0.47 0.47 query33 2.98 3.10 3.02 query34 17.10 4.50 4.56 query35 4.55 4.55 4.55 query36 0.64 0.49 0.48 query37 0.09 0.06 0.06 query38 0.05 0.03 0.03 query39 0.04 0.02 0.03 query40 0.16 0.12 0.12 query41 0.07 0.02 0.03 query42 0.04 0.02 0.02 query43 0.04 0.03 0.03 Total cold run time: 106.64 s Total hot run time: 32.99 s ```
zhannngchen commented 21 hours ago

please update "Problem Summary"

Yukang-Lian commented 16 hours ago

run buildall

github-actions[bot] commented 16 hours ago

clang-tidy review says "All clean, LGTM! :+1:"

doris-robot commented 15 hours ago

TeamCity be ut coverage result: Function Coverage: 38.04% (9902/26033) Line Coverage: 29.22% (82847/283517) Region Coverage: 28.35% (42546/150089) Branch Coverage: 24.91% (21570/86594) Coverage Report: http://coverage.selectdb-in.cc/coverage/02ba679f13d66c6b0b5374eb9e0a379c254d9cc6_02ba679f13d66c6b0b5374eb9e0a379c254d9cc6/report/index.html

doris-robot commented 15 hours ago
TPC-H: Total hot run time: 39731 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 02ba679f13d66c6b0b5374eb9e0a379c254d9cc6, data reload: false ------ Round 1 ---------------------------------- q1 17579 7477 7294 7294 q2 2065 182 173 173 q3 10558 1111 1155 1111 q4 10561 742 791 742 q5 7589 2684 2681 2681 q6 239 147 147 147 q7 971 620 619 619 q8 9232 1835 1941 1835 q9 6532 6466 6327 6327 q10 6997 2355 2287 2287 q11 479 261 254 254 q12 422 220 213 213 q13 17781 3013 3013 3013 q14 247 214 221 214 q15 581 540 511 511 q16 669 577 606 577 q17 973 561 564 561 q18 7307 6661 6674 6661 q19 1340 1026 915 915 q20 480 184 178 178 q21 4328 3112 3167 3112 q22 380 312 306 306 Total cold run time: 107310 ms Total hot run time: 39731 ms ----- Round 2, with runtime_filter_mode=off ----- q1 7275 7244 7276 7244 q2 331 230 229 229 q3 2876 2743 2919 2743 q4 2125 1868 1900 1868 q5 5717 5696 5619 5619 q6 231 143 149 143 q7 2237 1830 1821 1821 q8 3518 3568 3512 3512 q9 8797 8963 8960 8960 q10 3609 3543 3571 3543 q11 593 506 511 506 q12 846 644 677 644 q13 12153 3315 3251 3251 q14 311 274 265 265 q15 582 537 534 534 q16 692 637 633 633 q17 1859 1665 1631 1631 q18 8396 7857 7598 7598 q19 1710 1633 1569 1569 q20 2109 1875 1919 1875 q21 5546 5443 5470 5443 q22 649 530 547 530 Total cold run time: 72162 ms Total hot run time: 60161 ms ```
doris-robot commented 15 hours ago
TPC-DS: Total hot run time: 196709 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 02ba679f13d66c6b0b5374eb9e0a379c254d9cc6, data reload: false query1 1239 958 906 906 query2 6218 2047 2095 2047 query3 10945 4046 4124 4046 query4 67284 27918 23612 23612 query5 4935 461 430 430 query6 397 181 173 173 query7 5517 309 292 292 query8 318 227 225 225 query9 8450 2669 2663 2663 query10 453 253 256 253 query11 17128 15196 15908 15196 query12 159 101 101 101 query13 1489 409 411 409 query14 10986 7203 6777 6777 query15 228 197 182 182 query16 7083 476 501 476 query17 1101 593 570 570 query18 1890 293 299 293 query19 196 149 149 149 query20 120 109 113 109 query21 198 98 104 98 query22 4901 4490 4622 4490 query23 34729 34231 34812 34231 query24 5458 2577 2624 2577 query25 494 393 407 393 query26 642 150 144 144 query27 1750 282 325 282 query28 4460 2475 2463 2463 query29 682 430 427 427 query30 223 160 153 153 query31 1054 829 830 829 query32 63 52 53 52 query33 397 277 316 277 query34 937 524 513 513 query35 893 749 736 736 query36 1125 997 953 953 query37 122 74 84 74 query38 4525 4424 4432 4424 query39 1499 1504 1496 1496 query40 215 98 109 98 query41 46 43 42 42 query42 108 103 100 100 query43 545 494 495 494 query44 1221 835 873 835 query45 187 170 173 170 query46 1165 713 703 703 query47 2097 1970 1982 1970 query48 420 320 316 316 query49 720 381 403 381 query50 846 412 403 403 query51 7502 7242 7141 7141 query52 99 88 86 86 query53 258 177 176 176 query54 505 391 393 391 query55 82 74 86 74 query56 257 230 236 230 query57 1339 1212 1159 1159 query58 234 222 226 222 query59 3248 3001 3213 3001 query60 290 264 260 260 query61 135 129 126 126 query62 797 671 683 671 query63 218 191 201 191 query64 1501 778 633 633 query65 3269 3223 3207 3207 query66 702 286 304 286 query67 16042 15793 15864 15793 query68 3677 584 587 584 query69 411 248 264 248 query70 1144 1153 1143 1143 query71 335 248 253 248 query72 6386 3991 4094 3991 query73 777 359 359 359 query74 10059 8968 9095 8968 query75 3496 2669 2722 2669 query76 1761 1140 1128 1128 query77 499 278 268 268 query78 10614 9529 9343 9343 query79 1861 587 593 587 query80 1362 422 426 422 query81 521 243 232 232 query82 1331 117 118 117 query83 275 155 145 145 query84 275 77 70 70 query85 1016 295 296 295 query86 413 304 313 304 query87 4972 4645 4519 4519 query88 3682 2260 2216 2216 query89 418 293 315 293 query90 1895 185 184 184 query91 139 102 100 100 query92 67 51 48 48 query93 2497 546 541 541 query94 761 308 300 300 query95 345 244 241 241 query96 635 281 276 276 query97 2899 2686 2723 2686 query98 219 207 200 200 query99 2004 1304 1307 1304 Total cold run time: 321082 ms Total hot run time: 196709 ms ```
doris-robot commented 15 hours ago
ClickBench: Total hot run time: 32.85 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 02ba679f13d66c6b0b5374eb9e0a379c254d9cc6, data reload: false query1 0.03 0.03 0.04 query2 0.07 0.04 0.03 query3 0.23 0.07 0.08 query4 1.62 0.10 0.11 query5 0.43 0.42 0.43 query6 1.17 0.65 0.65 query7 0.02 0.01 0.02 query8 0.04 0.03 0.03 query9 0.59 0.51 0.51 query10 0.54 0.56 0.56 query11 0.14 0.10 0.10 query12 0.14 0.12 0.11 query13 0.62 0.60 0.60 query14 2.82 2.82 2.69 query15 0.89 0.82 0.83 query16 0.38 0.37 0.40 query17 0.99 1.04 1.05 query18 0.22 0.23 0.20 query19 1.95 1.78 2.02 query20 0.02 0.01 0.01 query21 15.35 0.58 0.58 query22 2.87 2.13 1.97 query23 16.89 0.90 0.82 query24 2.79 1.17 1.64 query25 0.26 0.21 0.06 query26 0.53 0.14 0.14 query27 0.04 0.04 0.05 query28 10.19 1.10 1.07 query29 12.57 3.27 3.26 query30 0.26 0.06 0.06 query31 2.84 0.38 0.36 query32 3.31 0.47 0.47 query33 3.09 2.98 3.04 query34 17.07 4.46 4.49 query35 4.51 4.63 4.50 query36 0.65 0.48 0.47 query37 0.09 0.06 0.06 query38 0.05 0.04 0.03 query39 0.03 0.02 0.02 query40 0.16 0.13 0.12 query41 0.08 0.03 0.03 query42 0.04 0.02 0.02 query43 0.03 0.04 0.03 Total cold run time: 106.61 s Total hot run time: 32.85 s ```
github-actions[bot] commented 13 hours ago

PR approved by at least one committer and no changes requested.

github-actions[bot] commented 13 hours ago

PR approved by anyone and no changes requested.