StarRocks Roadmap 2023

[Dshadowzh]

Refer to roadmap 2022

From OLAP to Lakehouse

Data lake analytics

• [ ] Unified cache for data lake and StarRocks table
• [x] CBO optimization for no statistics.
• [ ] Incremental query and snapshot query from data lake
• [x] CTAS on data lake
• [ ] Indexes on data lake
• [x] Presto compatibility
• [ ] Kerberos integration with external catalogs
• [ ] More connectors(Kudu / Mongodb ... )
• Better table format support
  • [x] Better Hive/Iceberg/Hudi/Delta lake support
  • [ ] Hive/Iceberg/Hudi/Deltalake table sink Iceberg
  • [x] Parquet/Orc file writer
  • [ ] Open format: combine lake and local into one table
• More cloud integrations.
  • [x] Blob store integration
  • [ ] Authentication and privileges.

Decoupled storage and computing

• StarOS
  • [ ] LogStore: for low latency application (including Primary key index, MV, FE metadata and row store)
  • [x] FileStore: High performance local storage
  • [x] Resource manager: Support K8s and multi-cloud
  • [x] Multi blob storage support
• StarRocks based on StarOS
  • [x] Fully function support for StarRocks (All data model, all DDL, DML and compaction service)
  • [x] Multi-warehouse and multi-cluster
  • [ ] Decouple storage and computing for FE
  • [x] Cloud native architecture for primary key
  • [ ] Time travel query
  • [ ] Global cache and serverless cluster

Core system

Query

• [x] Operator spilling Docs
• [ ] New computing scheduling framework
• [ ] Enhance Resource group ability for resource isolation
• [ ] Adaptive Query Processing
• [ ] Expression Codegen
• [ ] Low cardinality optimization for nested type
• [x] map/struct native support, column reuse Map Docs Struct Docs
• [x] Generated column (former called materialized column) Docs
• [x] Query cache for join Docs
• [x] Lambda for all data types Docs
• [ ] GEO type and functions

Storage and data management

• [ ] Support larger scale of metadata, including decoupled storage and computing for FE, Fine-grained lock for metadata, multi-tenant management.
• [ ] Unify metadata access through information schema.
• [x] Wise number of buckets detection and distributed column selection.
• [x] Auto partition creation and partition by expression.
• [ ] Unified table models.
• [ ] Richer support for SQL: full support for update and merge into syntax
• [x] Better performance for partial update
• [ ] Z-order index
• [x] Row-oriented storage engine
• [ ] Full text / inverted index
• [ ] Index for json / map
• [ ] Key-value api query, bypass SQL layer to shortcut.

Real-time

Ingestion

• [ ] Unified loading and unloading
• [x] Schema registry integration
• [ ] WAL support for high concurrency ingestion and millisecond level data freshness
• [ ] Support external stream
• [ ] Kinesis support
• [x] Pb/avro data format support #16662

Materialized view

• [x] Query rewrite optimization Docs
• [x] External table materialized view Docs
• [x] Spilling and resource group for materalized view
• [ ] Streaming materialized view for single-table operator.
• [ ] Streaming materialized view for multi-table join.
• [ ] Streaming materialized view for deleting and updating.
• [ ] StarRocks internal stream and binlog.
• [ ] Materialized view for external stream.
• [ ] Incremental refresh for external table materialized view.

Ecosystem

BI connector

• [ ] Tableau connector / better compatibility
• [x] Metabase connector / better compatibility
• [x] Superset connector / better compatibility
• [x] Redash connector / better compatibility

SQL editor / notebook

• [ ] DBeaver integration
• [ ] Datagrip integration
• [ ] Hex integration
• [ ] QueryBook integration

ETL tools

• [x] Airbyte connector https://github.com/StarRocks/starrocks/tree/main/contrib/destination-starrocks
• [ ] Fivetran connector
• [x] Optimize the DBT connector

Compute engine

• [x] Optimize flink connector
• [x] Optimize spark connector

Monitor

• [x] Datadog integration https://github.com/StarRocks/starrocks/tree/main/contrib/datadog-connector

Security

• [x] Fully support for RBAC. Docs
• [x] Plugin framework.
• [x] Ranger integration.

Release

• [x] https://github.com/StarRocks/starrocks/issues/18125
• [x] 3.1 release notes
• [x] https://github.com/StarRocks/starrocks/issues/33079

[vkingnew]

This issue should stick on the top of all issue. Please help put it to the Begin.

with recursive syntax support : https://github.com/StarRocks/starrocks/issues/34616

[alberttwong]

@Dshadowzh Can we update this?

[Leowuqunqun]

@Dshadowzh MongoDB connect Is anyone following up? If not, I'd love to try

[wangsimo0]

@Dshadowzh MongoDB connect Is anyone following up? If not, I'd love to try

not yet, of course, more than welcome

[LonwoLonwo]

https://github.com/dbeaver/dbeaver/issues/21583 ;)