apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.

https://doris.apache.org

Apache License 2.0

12.7k stars 3.28k forks source link

Release Note 2.1.7 #43319

Open yiguolei opened 1 week ago

yiguolei commented 1 week ago

Behavior changes

The following global variables will be forcibly set to the following default values:
- enable_nereids_dml: true
- enable_nereids_dml_with_pipeline: true
- enable_nereids_planner: true
- enable_fallback_to_original_planner: true
- enable_pipeline_x_engine: true
New columns have been added to the audit log. https://github.com/apache/doris/pull/42262
New features

Lakehouse

Async Materialized View
An asynchronous materialized view has added a property called use_for_rewrite to control whether it participates in transparent rewriting. https://github.com/apache/doris/pull/40332
Query Execution
The list of changed session variables is now output in the Profile. Link to the pull request
Support for trim_in, ltrim_in, and rtrim_in functions has been added. Link to the pull request (Note: This is a duplicate mention, but I'm including it as per your original list.)
Support for several URL functions (top_level_domain, first_significant_subdomain, cut_to_first_significant_subdomain) has been added. Link to the pull request
The bit_set function has been added. Link to the pull request
The count_substrings function has been added. Link to the pull request
The translate and url_encode functions have been added. Link to the pull request
The normal_cdf, to_iso8601, and from_iso8601_date functions have been added. Link to the pull request
(Duplicate entry, but for completeness) Support for trim_in, ltrim_in, and rtrim_in functions has been added. Link to the pull request
Storage Management
The information_schema.table_options and table_properties system tables have been added, supporting the querying of attributes set during table creation. https://github.com/apache/doris/pull/34384
Support for bitmap_empty as a default value has been implemented. https://github.com/apache/doris/pull/40364
A new session variable require_sequence_in_insert has been introduced to control whether a sequence column must be provided when performing INSERT INTO SELECT writes to a unique key table. https://github.com/apache/doris/pull/41655/files
Others
Allow for generating flame graphs on the BE WebUI page.https://github.com/apache/doris/pull/41044

Improvements

Lakehouse

Support for writing data to Hive Text format tables: https://github.com/apache/doris/pull/40537
- Documentation: https://doris.apache.org/docs/lakehouse/datalake-building/hive-build
Access MaxCompute data using MaxCompute Open Storage API: https://github.com/apache/doris/pull/41610
- Documentation: https://doris.apache.org/docs/lakehouse/database/max-compute
Support for Paimon DLF Catalog: https://github.com/apache/doris/pull/41694
- Documentation: https://doris.apache.org/docs/lakehouse/datalake-analytics/paimon
Added table$partitions syntax to directly query Hive partition information: https://github.com/apache/doris/pull/41230
- Documentation: https://doris.apache.org/docs/lakehouse/datalake-analytics/hive
Support for reading Parquet files in brotli compression format: https://github.com/apache/doris/pull/42162
Support for reading decimal 256 types in Parquet files: https://github.com/apache/doris/pull/42241
Support for reading Hive tables in OpenCsvSerde format: https://github.com/apache/doris/pull/42939
Async Materialized View
Refined the granularity of lock holding during the build process for asynchronous materialized views. https://github.com/apache/doris/pull/40402 and https://github.com/apache/doris/pull/41010.
Query optimizer
Improved the accuracy of statistic information collection and usage in extreme cases to enhance planning stability. https://github.com/apache/doris/pull/40457
Runtime filters can now be generated in more scenarios to improve query performance. https://github.com/apache/doris/pull/40815
Enhanced constant folding capabilities for numerical, date, and string functions to boost query performance. https://github.com/apache/doris/pull/40820
Optimized the column pruning algorithm to enhance query performance. https://github.com/apache/doris/pull/41548
Query Execution
Supported parallel preparation to reduce the time consumed by short queries. https://github.com/apache/doris/pull/40270
Corrected the names of some counters in the profile to match the audit logs. https://github.com/apache/doris/pull/41993
Added new local shuffle rules to speed up certain queries. https://github.com/apache/doris/pull/40637
Storage Management
The SHOW PARTITIONS command now supports displaying the commit version. https://github.com/apache/doris/pull/28274
Checked for unreasonable partition expressions when creating tables. https://github.com/apache/doris/pull/40158
Optimized the scheduling logic when encountering EOF in Routine Load. https://github.com/apache/doris/pull/40509
Made Routine Load aware of schema changes. https://github.com/apache/doris/pull/40508
Improved the timeout logic for Routine Load tasks. https://github.com/apache/doris/pull/41135
Others
Allowed closing the built-in service port of BRPC via BE configuration. https://github.com/apache/doris/pull/41047
Fixed issues with missing fields and duplicate records in audit logs. https://github.com/apache/doris/pull/43015
Bug fixes

Lakehouse
Fixed the inconsistency in the behavior of INSERT OVERWRITE with Hive. https://github.com/apache/doris/pull/39840
Cleaned up temporarily created folders to address the issue of too many empty folders on HDFS. https://github.com/apache/doris/pull/40424
Resolved memory leaks in FE caused by using the JDBC Catalog in some cases. https://github.com/apache/doris/pull/40923
Resolved memory leaks in BE caused by using the JDBC Catalog in some cases. https://github.com/apache/doris/pull/41266
Fixed errors in reading Snappy compressed formats in certain scenarios. https://github.com/apache/doris/pull/40862
Addressed potential FileSystem leaks on the FE side in certain scenarios. https://github.com/apache/doris/pull/41108
Resolved issues where using EXPLAIN VERBOSE to view external table execution plans could cause null pointer exceptions in some cases. https://github.com/apache/doris/pull/41231
Fixed the inability to read tables in Paimon parquet format. https://github.com/apache/doris/pull/41487
Addressed performance issues introduced by compatibility changes in the JDBC Oracle Catalog. https://github.com/apache/doris/pull/41407
Disabled predicate pushing down after implicit conversion to resolve incorrect query results in some cases with JDBC Catalog. https://github.com/apache/doris/pull/42242
Fixed issues with case-sensitive access to table names in the External Catalog. https://github.com/apache/doris/pull/42261
Async Materialized View
Fixed the issue where user-specified start times were not effective. https://github.com/apache/doris/pull/39573
Resolved the issue of nested materialized views not refreshing. https://github.com/apache/doris/pull/40433
Fixed the issue where materialized views might not refresh after the base table was deleted and recreated. https://github.com/apache/doris/pull/41762
Addressed issues where partition compensation rewrites could lead to incorrect results. https://github.com/apache/doris/pull/40803
Fixed potential errors in rewrite results when sql_select_limit was set. https://github.com/apache/doris/pull/40106 Semi-Structured Data Management
Fixed the issue of index file handle leaks. https://github.com/apache/doris/pull/41915
Addressed inaccuracies in the count() function of inverted indexes in special cases. https://github.com/apache/doris/pull/41127
Fixed exceptions with variant when light schema change was not enabled. https://github.com/apache/doris/pull/40908
Resolved memory leaks when variant returns arrays. https://github.com/apache/doris/pull/41339
Query optimizer
Corrected potential errors in nullable calculations for filter conditions during external table queries, leading to execution exceptions. https://github.com/apache/doris/pull/41014
Fixed potential errors in optimizing range comparison expressions. https://github.com/apache/doris/pull/41356
Query Execution
The match_regexp function could not correctly handle empty strings. https://github.com/apache/doris/pull/39503
Resolved issues where the scanner thread pool could become stuck in high-concurrency scenarios. https://github.com/apache/doris/pull/40495
Fixed errors in the results of the data_floor function. https://github.com/apache/doris/pull/41948
Addressed incorrect cancel messages in some scenarios. https://github.com/apache/doris/pull/41798
Fixed issues with excessive warning logs printed by arrow flight. https://github.com/apache/doris/pull/41770
Resolved issues where runtime filters failed to send in some scenarios. https://github.com/apache/doris/pull/41698
Fixed problems where some system table queries could not end normally or became stuck. https://github.com/apache/doris/pull/41592
Addressed incorrect results from window functions. https://github.com/apache/doris/pull/40761
Fixed issues where the encrypt and decrypt functions caused BE cores. https://github.com/apache/doris/pull/40726
Resolved errors in the results of the conv function. https://github.com/apache/doris/pull/40530
Storage Management
Fixed import failures when Memtable migration was used in multi-replica scenarios with machine crashes. https://github.com/apache/doris/pull/38003
Addressed inaccurate memory statistics during the Memtable flush phase during imports. https://github.com/apache/doris/pull/39536
Fixed fault tolerance issues with Memtable migration in multi-replica scenarios. https://github.com/apache/doris/pull/40477
Resolved inaccurate bvar statistics with Memtable migration. https://github.com/apache/doris/pull/40985
Fixed inaccurate progress reporting for S3 loads. https://github.com/apache/doris/pull/40987
Permissions
Fixed permission issues related to show columns, show sync, and show data from db.table. https://github.com/apache/doris/pull/39726
Others
Fixed the issue where the audit log plugin for version 2.0 could not be used in version 2.1. https://github.com/apache/doris/pull/41400

medivh511 commented 1 week ago

any update for support paimon 0.9?