Doris Roadmap 2024

[morningman]

Roadmap 2023 Roadmap 2022

Separation of Storage and Computation

• Flexibility & Stateless
  • [x] Stateless BE node
  • [ ] Stateless FE node
• Load Isolation
  • [x] Multi cluster support
  • [x] Read & write isolation
• More storage support
  • [x] AWS S3
  • [x] Aliyun OSS
  • [x] Tencent Cloud COS
  • [x] Huawei Cloud OBS
  • [x] Baidu Cloud BOS
  • [x] GCP
  • [ ] Azure
  • [x] HDFS
• Performance
  • [x] Optimized cache policy
  • [ ] Optimization for cold data querying
• [x] Support data deletion
• SLA
  • [x] Upgrade BE with no impact
  • [ ] Upgrade FE with no impact
• Reliability
  • [ ] Snapshot & Time travel
  • [ ] Enhanced backup & restore
• [ ] Data sharing

Async Materialized View

• Build materialized view

  • [x] Support full refresh
  • [x] Support partition level refresh
  • [x] Support building mv from olap table
  • [x] Support building mv from hive table
  • [ ] Support building mv from iceberg table
  • [ ] Support building mv from hudi table
  • [ ] Nested materialized view with DAG
  • [ ] Incremental building for external table with partition granularity
  • [ ] Support partition rollup
  • [ ] Support partition TTL
  • [ ] Support REPLACE operation
  • [ ] Support refresh materialized view by time range

• Transparent Rewriting

  • [x] Support aggregation and rollup
  • [x] Support join
  • [ ] Query Partial rewriting
  • [ ] Rewriting supports nested materialized view

• Materialized view management

  • [ ] Materialized view recommendation

Semi Structure Data Analysis

• Inverted Index

  • [x] Support Inverted Index
  • [ ] Merging index files
  • [ ] Working with separation of storage and computation
  • [ ] Speed up the data loading with inverted index

• VARIANT data type

  • [x] Support VARIANT data type
  • [x] Working with inverted index

Query Optimizer

• Basic framework

  • [ ] Fully supports DQL, DML and DDL
  • [ ] Optimized memory consumption
  • [ ] Optimized apply order of RBO rules
  • [ ] Improved efficiency of Cascades enumeration

• Planning quality

  • Statistics
    • [ ] Support statistical for synced materialized views
    • [ ] Support partition level statistics collection
    • [ ] Supports histogram statistics collection
  • New distributed cost model
    • [ ] Optimized distributed cost model framework
    • [ ] Support runtime cost revaluation
    • [ ] Supports more accurate operator cost fitting models
  • Rules and enumerations
    • [ ] Expand RBO rules
    • [ ] Improve the quality of Cascades enumeration plan
    • [ ] Enhanced dphyper enumeration framework function, supports outer join enumeration and CDC
  • Enhance runtime filter adaptive capability
    • [ ] Adaptive runtime filter size
    • [ ] Adaptive runtime filter type
    • [ ] Adaptive runtime filter waiting time
  • [ ] Supports histogram-based data skew adaptive processing framework

DataLake Analysis

• Support more file format

  • [ ] RCFile
  • [ ] SequenceFile

• Support more lake format

  • [x] Support Iceberg with ORC
  • [x] Support Iceberg Equality Delete
  • [ ] Support more systable on Hudi
  • [ ] Support CDC scan on Hudi
  • [ ] Support more systable on Paimon

• Trino Connector compatibility

  • [x] Trino Connector compatibility framework
  • [ ] Support Trino DeltaLake Connector
  • [x] Support Trino Bigquery Connector
  • [ ] Support Trino Cassandra Connector

• Datalake write back

  • Hive
    • [x] Support unpartitioned table
    • [x] Support partitioned table
    • [x] Support INSERT OVERWRITE
    • [x] Support INSERT
  • Iceberg
    • [ ] Support unpartitioned table
    • [ ] Support partitioned table
    • [ ] Support update and delete
  • Hudi
  • Paimon

• Enhanced JDBC Catalog

  • [x] Support DB2
  • [ ] Support sharded database
  • [ ] Support query concurrency

• Enhanced file analysis

  • [ ] Support insert into table value function

• Enhanced file cache

  • [ ] Support memory-level file cache
  • [ ] Enhanced cache statistic and hits analysis

• Integrate with Apache Ranger

  • [x] Support Catalog/Database/Table/Resource/WorkloadGroup auth
  • [x] Support row policy
  • [x] Support data mask
  • [x] Support column level privilege

• SQL dialect support

  • [x] Presto/Trino
  • [x] Spark
  • [ ] Hive
  • [x] Clickhouse
  • [x] Oracle
  • [x] Postgres

Query Processing

• Resource Isolation
  • [x] Support hard/soft resource isolation for Query & Load
  • [ ] Enhance the visibility of resource usage
  • [ ] Automatically workload management at runtime
• [ ] Support store procedure
• Support Spill to disk
  • [ ] Sort Operator
  • [ ] Aggregate Operator
  • [ ] Join Operator
• [ ] Working with shuffle service
• [ ] Stage by stage query processing

Storage Engine

• Data Loading
  • [x] Support auto partiton when loading
  • [ ] Zero-ETL: Built-in data integration from OLTP CDC to Doris
  • [ ] Support transactional multi table INSERT INTO
  • [ ] Support MERGE INTO
• Data Modeling
  • [x] Auto incremental column
  • [ ] Support CLUSTER BY
  • [ ] Support KEY column in arbitrary order
• Cross cluster replication
  • [ ] Support Master/Slave switch
  • [ ] Support cross region deployment
  • [ ] Work with separation of storage and computation
• [ ] Support data binlog
• [ ] Enhanced Z-order index
• [ ] Optimized high-concurrency point query

Ecosystem & Tools

• Cluster Manager for Apache Doris
  • [ ] Support agent mode
  • [ ] Support k8s
  • [ ] Enhanced monitor and alert management
  • [ ] Visualized profile analysis
  • [ ] Support Notebook
  • [ ] Built-in visualized BI reports
• [x] Doris StreamLoader tool
• Doris Operator
• X2Doris
  • [x] Support Hive to Doris
  • [x] Support Doris to Doris
  • [x] Support Kudu to Doris
  • [x] Support StarRocks to Doris
  • [ ] Support Clickhouse to Doris
• BI tools compatibility
  • [x] Superset
  • [ ] Metabase
  • [ ] Navicat
  • [ ] Datagrip
  • [ ] Dbeaver
  • [ ] SmartBI
  • [ ] FineBI
• Data Integration
  • [ ] Kettle

[vinlee19]

Currently, I have completed the development and testing of the JDBC catalog for Apache Druid. If possible, I would like to contribute this feature. PR:https://github.com/apache/doris/pull/27270

[vinlee19]

Flink-connector-doris will use FlinkCDC to synchronize multiple tables or the entire database from MongoDB and DB2 to Doris.

[michael1991]

typo "Mutlt cluster support" => "Multi cluster support"

[Hanchers]

Looking forward to version 2.1

[longzmkm]

Looking forward to version 2.1

me too

[vonwind]

Walking with innovators

[zhbdesign]

Support generating columns

[morningman]

Currently, I have completed the development and testing of the JDBC catalog for Apache Druid. If possible, I would like to contribute this feature. PR:#27270

HI @vinlee19 , thanks for your contribution. I'm not sure if it is suitable for Druid to using JDBC as data connector? I'm concerning the performance issue. But indeed Trino is using JDBC connect Druid. I will take a look at this feature, and could you please also provide test cases (eg. druid docker compose)?

[liugddx]

https://github.com/dbeaver/dbeaver/pull/22836

[liugddx]

https://github.com/langchain-ai/langchain/pull/17527

[cs3163077]

Looking forward to version 2.1

me too

[sdhzwc]

Looking forward to version 2.1

me too

[dragonkid]

why there is no 'Support building mv from Paimon table'

[qianmoQ]

BI tools compatibility Can it be adapted to https://github.com/devlive-community/datacap?

[morningman]

why there is no 'Support building mv from Paimon table'

It will be supported

[morningman]

BI tools compatibility Can it be adapted to https://github.com/devlive-community/datacap? Hi @qianmoQ, I am not familiar with datacap, but you are very welcome to helping Doris adapt to it. I saw the Doris is on the log wall, maybe you can post a blog on Doris website about how to connect to Doris using datacap?

[ShawshankLin]

Support transactional multi table DELETE INSERT for adapting aggregate tables in dbt's Incremental models https://docs.getdbt.com/docs/build/incremental-models

[JiangJamm]

Look forward to supporting DataGrip and kettle!

[zhangm365]

The correct url for async-materialized-view item is as follows: https://doris.apache.org/docs/query/view-materialized-view/async-materialized-view

[shiliming]

binlog，binlog，binlog!

[mohuaiyuan]

Which version of Doris is preparing to support DB2？