Release plan 3.2 - Githubissues

bearmmx commented 1 year ago

3.2.0 release plan

Supports creating and dropping databases and managed tables in Hive catalogs, and supports exporting data to Hive's managed tables.
Supports Unified Catalog, which provides unified access when multiple types of catalogs share a common metastore like Hive metastore or AWS Glue.
Supports collecting statistics of tables in Hive, Hudi, and Iceberg catalogs using ANALYZE TABLE, facilitating optimization and accelerating subsequent queries.
Optimized query performance on ORC files, which can result in nearly a two-fold performance improvement for VARCHAR and CHAR data reading and enhance the decompression performance of ORC files in zlib compression format.
Optimized query performance on Parquet files by the follow improvements: adaptive I/O merging, Dict Filter with the faster predicate rewriting and on-demand dictionary column decoding, performace improvment of Dict Decode, performace improvment of late materialization, file footers caching.
Optimized the performance of the CSV Reader.
Supports Information Schema for external tables, allowing connection to StarRocks through Metabase to query tables in external catalogs.
Optimized the performance of the count calculation.
Iceberg Catalog supports collecting column statistics from Manifest files, collecting ndv (number of distinct values) and partition pruning, and reduces Iceberg metadata memory consumption to enhance stability in scenarios with large metadata volumes or high query concurrency.

Table function FILES() supports loading Parquet and ORC format data from Azure or GCP, extracting the value of a key/value pair from the file path as the value of a column, automatic table schema detection and unionization of the same batch of data files, and loading complex data types including ARRAY, JSON, MAP, and STRUCT.
Supports unloading data from StarRocks to Parquet-formatted files stored in AWS S3 or HDFS by using INSERT INTO FILES.
Support continuous data loading from AWS S3 or HDFS using the PIPE command.
While loading data, PIPE can detect new files or modifications in a remote storage directory, and then automatically load the new or modified data into the destination table in StarRocks. At the same time, PIPE can automatically splits a large loading task into smaller, serialized tasks to enhance stability in large-scale data ingestion scenarios and reduce the cost of error retries.

Supports manual optimization of data distribution strategy used in an existing table. Users can set a new partition key, bucket key, bucket number, or sort key for an existing table to cater to evolving business demands.
Supports the fast schema evolution, which is controlled by the table property
Optimized the persistent index for Primary Key tables by improving memory usage logic while reducing I/O read and write amplification.
Supports data re-distribution across disks for Primary Key tables.
Partitioned tables support automatic cooldown based on the partition time range and cooldown time.
The Publish phase of a load job that writes data into a Primary Key table is changed from asynchronous mode to synchronous mode. As such, the data loaded can be queried immediately after the load job finishes.
Optimized the automatic bucketing logic for tables with a random bucketing strategy.
In cases where the number of buckets is not specified, StarRocks dynamically creates tablets according to cluster information and the size of the data.

Supports HTTP SQL API, enabling users to access StarRocks data via HTTP and execute SELECT, SHOW, EXPLAIN, or KILL operations.
Supports the prepared statement for Primary Key tables with row storage. It allows better performance for processing high-concurrency queries. It also prevents SQL injection effectively.
Supports manual optimization of data distribution strategy used in an existing table. Users can set a new partition key, bucket key, bucket number, or sort key for an existing table to cater to evolving business demands.
Supports Runtime Profile and text-based Profile analysis commands (SHOW PROFILELIST, ANALYZE PROFILE, EXPLAIN ANALYZE) to allow users to directly analyze profiles via MySQL clients, facilitating bottleneck identification and discovery of optimization opportunities.
Optimized StarRocks' compatibility with Metabase and Superset. Supports integrating them with external catalogs.
Supports the prepared statement. It allows better performance for processing high-concurrency queries. It also prevents SQL injection effectively.

Asynchronous materialized view

Supports automatic refresh for asynchronous materialized views created upon a view when schema changes occur on the view or the base table of the view.
Supports the TRACE REWRITE clause, which can be used to analyze the transparent rewrite of asynchronous materialized views.
Supports Query Dump for asynchronous materialized views.
The Spill to Disk feature is enabled by default for the refresh tasks of asynchronous materialized views, reducing memory consumption.
Added a consistency check for materialized views' partitioning key.
Supports creating asynchronous materialized views based on tables in a Paimon catalog with partition-level refresh.
Supports backing up and restoring asynchronous materialized views.
Supports partition-level incremental refresh for asynchronous materialized views created upon Iceberg catalogs.

Synchronous materialized view

Added the following functions:

String functions: substring_index, url_extract_parameter, url_encode, url_decode, and translate
Date functions: dayofweek_iso, week_iso, quarters_add, quarters_sub, milliseconds_add, milliseconds_sub, date_diff, jodatime_format, str_to_jodatime, to_iso8601, to_tera_date, and to_tera_timestamp
Pattern matching function: regexp_extract_all
hash function: xx_hash3_64
Aggregate functions: approx_top_k
Window functions: cume_dist, percent_rank and session_number
Utility functions: dict_mapping and get_query_profile
array_agg supports the keyword DISTINCT.