apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
11.8k stars 3.11k forks source link

[opt](hive) save hive table schema in transaction #37008

Closed wuwenchi closed 15 hours ago

wuwenchi commented 3 days ago

Proposed changes

Save the table schema, reduce the number of HMS calls, and improve write performance.

doris-robot commented 3 days ago

Thank you for your contribution to Apache Doris. Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website. See Doris Document.

wuwenchi commented 3 days ago

run buildall

doris-robot commented 3 days ago
TPC-H: Total hot run time: 39880 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 78ad69360e2907b6d81935117515b68ff5e1c83b, data reload: false ------ Round 1 ---------------------------------- q1 17612 4397 4253 4253 q2 2010 188 190 188 q3 10462 1197 1129 1129 q4 10182 814 708 708 q5 7480 2670 2613 2613 q6 220 140 141 140 q7 943 600 608 600 q8 9221 2070 2038 2038 q9 8871 6489 6515 6489 q10 8961 3667 3681 3667 q11 451 246 242 242 q12 457 241 232 232 q13 17778 3014 2998 2998 q14 265 227 212 212 q15 551 479 479 479 q16 492 378 380 378 q17 965 674 647 647 q18 7951 7465 7484 7465 q19 4442 1526 1516 1516 q20 649 330 330 330 q21 4890 3214 3213 3213 q22 383 343 345 343 Total cold run time: 115236 ms Total hot run time: 39880 ms ----- Round 2, with runtime_filter_mode=off ----- q1 4362 4243 4238 4238 q2 382 265 259 259 q3 2998 2744 2944 2744 q4 1968 1721 1706 1706 q5 5559 5575 5498 5498 q6 220 139 147 139 q7 2240 1839 1867 1839 q8 3281 3390 3406 3390 q9 8626 8660 8797 8660 q10 4099 3903 3763 3763 q11 579 482 486 482 q12 800 654 629 629 q13 17150 3214 3169 3169 q14 304 271 278 271 q15 525 477 495 477 q16 502 438 443 438 q17 1817 1510 1496 1496 q18 8136 7826 7887 7826 q19 1788 1742 1631 1631 q20 3079 1900 1857 1857 q21 5106 5062 4810 4810 q22 740 543 551 543 Total cold run time: 74261 ms Total hot run time: 55865 ms ```
doris-robot commented 3 days ago
TPC-DS: Total hot run time: 173666 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 78ad69360e2907b6d81935117515b68ff5e1c83b, data reload: false query1 926 379 382 379 query2 6467 2523 2321 2321 query3 6634 210 212 210 query4 18714 17491 17399 17399 query5 3600 465 481 465 query6 265 204 163 163 query7 4601 300 302 300 query8 299 291 292 291 query9 8521 2446 2441 2441 query10 543 283 273 273 query11 10526 10043 10152 10043 query12 111 84 89 84 query13 1636 376 366 366 query14 10159 7463 6456 6456 query15 224 186 189 186 query16 7756 262 273 262 query17 1900 548 514 514 query18 1943 273 269 269 query19 193 151 153 151 query20 91 87 80 80 query21 211 124 124 124 query22 4621 4247 4273 4247 query23 33958 33751 33476 33476 query24 11166 2907 2936 2907 query25 596 381 403 381 query26 707 171 156 156 query27 2324 337 334 334 query28 6292 2199 2179 2179 query29 912 662 676 662 query30 265 166 153 153 query31 972 765 762 762 query32 95 55 51 51 query33 766 311 291 291 query34 1012 477 489 477 query35 768 644 629 629 query36 1114 1010 972 972 query37 149 78 87 78 query38 3011 2817 2852 2817 query39 894 830 841 830 query40 206 139 129 129 query41 53 55 54 54 query42 106 101 103 101 query43 595 553 558 553 query44 1123 744 758 744 query45 193 165 164 164 query46 1091 750 724 724 query47 1900 1802 1793 1793 query48 359 301 294 294 query49 836 406 408 406 query50 775 387 402 387 query51 6826 6821 6655 6655 query52 101 100 98 98 query53 363 294 297 294 query54 858 451 440 440 query55 74 78 74 74 query56 276 262 257 257 query57 1124 1066 1063 1063 query58 235 244 243 243 query59 3365 3051 3108 3051 query60 293 278 284 278 query61 95 95 89 89 query62 596 442 437 437 query63 322 293 297 293 query64 8571 2216 1731 1731 query65 3231 3137 3105 3105 query66 755 323 326 323 query67 15612 14951 14861 14861 query68 4536 551 557 551 query69 626 443 383 383 query70 1194 1075 1129 1075 query71 418 278 274 274 query72 8121 5558 5484 5484 query73 758 326 325 325 query74 5841 5504 5487 5487 query75 3520 2701 2686 2686 query76 2779 926 920 920 query77 628 303 296 296 query78 10574 9980 9643 9643 query79 2714 526 534 526 query80 2044 538 490 490 query81 565 224 220 220 query82 797 106 104 104 query83 333 176 172 172 query84 270 91 85 85 query85 1511 279 268 268 query86 455 335 319 319 query87 3353 3100 3117 3100 query88 4140 2371 2371 2371 query89 487 380 393 380 query90 1679 189 189 189 query91 130 104 101 101 query92 58 49 53 49 query93 3820 530 519 519 query94 1018 191 190 190 query95 407 317 306 306 query96 594 270 276 270 query97 3241 3055 3058 3055 query98 213 197 200 197 query99 1245 865 811 811 Total cold run time: 272945 ms Total hot run time: 173666 ms ```
doris-robot commented 3 days ago
ClickBench: Total hot run time: 31.13 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 78ad69360e2907b6d81935117515b68ff5e1c83b, data reload: false query1 0.04 0.03 0.03 query2 0.08 0.04 0.04 query3 0.23 0.06 0.05 query4 1.68 0.08 0.09 query5 0.50 0.49 0.50 query6 1.13 0.72 0.73 query7 0.02 0.02 0.01 query8 0.04 0.04 0.04 query9 0.54 0.49 0.49 query10 0.54 0.53 0.54 query11 0.15 0.11 0.12 query12 0.15 0.12 0.12 query13 0.60 0.59 0.60 query14 0.77 0.77 0.79 query15 0.84 0.82 0.82 query16 0.37 0.37 0.37 query17 1.02 1.07 1.03 query18 0.21 0.23 0.26 query19 1.78 1.79 1.81 query20 0.01 0.01 0.01 query21 15.44 0.73 0.65 query22 4.70 5.80 2.42 query23 18.28 1.40 1.19 query24 2.11 0.23 0.23 query25 0.16 0.08 0.08 query26 0.27 0.17 0.18 query27 0.08 0.08 0.08 query28 13.16 1.01 1.00 query29 12.63 3.27 3.23 query30 0.25 0.06 0.05 query31 2.87 0.39 0.39 query32 3.28 0.48 0.47 query33 2.91 2.90 2.90 query34 17.24 4.49 4.39 query35 4.54 4.49 4.53 query36 0.65 0.46 0.46 query37 0.17 0.15 0.16 query38 0.15 0.15 0.14 query39 0.05 0.03 0.03 query40 0.17 0.15 0.15 query41 0.10 0.06 0.05 query42 0.06 0.05 0.05 query43 0.04 0.04 0.04 Total cold run time: 110.01 s Total hot run time: 31.13 s ```
github-actions[bot] commented 1 day ago

PR approved by at least one committer and no changes requested.

github-actions[bot] commented 1 day ago

PR approved by anyone and no changes requested.