Gluten Roadmap in 2024

[weiting-chen]

Description

Here listed Gluten roadmap in 2024. We will use this list to have an overview for Gluten, an easy way to track feature support plan, and to avoid duplicate work before starting the implementation.

• [x] Migrate to Apache Gluten: https://issues.apache.org/jira/browse/INFRA-25506
• [ ] Installation Method: pip/conda/docker
• [x] Spark version support: Spark3.4 UT(#3559 )(done), Spark 3.5 API/UT(#4425 )
• [ ] Data Type Support: complex, decimal32, timestamp, timestampNTZ, Calendar, Interval, UDT
• [ ] fuzz tests support decimal/agg, Spark query runner in Velox
• [ ] OS support: move to use static link via vcpkg to avoid any OS support issue
• [ ] File Format Support: Parquet(gap on some datatypes), ORC, Text, CSV, Json
• [ ] Data Source Support: ABFS, S3, GCS, Alluxio, HDFS viewfs
• [ ] Spark Functions Support, please check (#4039 )
• [ ] Spark Operator Support: SMJ, PyArrow
• [ ] Spill Support including Partial Agg with single agg, Join Probe side Spill, Window + Spill, Sort + Spill, Spill performance improvement, Issue fixing(#3030 )
• [ ] Shuffle Support including Sort based Shuffle, Merge before Compress, Reducer Optimization
• [ ] Data Lake Support(#3378 ): DeltaLake, Iceberg, Hudi, Paimon
• [x] RSS Support: Apache Uniffle(#3767 )
• [ ] Pyspark Support including Python UDF/Arrow UDF
• [ ] Stage level resource management(#4392 )
• [ ] Other Big Data Framework Support
• [ ] Performance Optimizations: Sort, Join, Agg., Parquet Write, Scan
• [ ] Other Accelerator such as GPU, FPGA Support
• [ ] Document and Website Update
• [ ] Planner: optimization for multiple fallback cases

[felipepessoto]

@weiting-chen, for item

• Data Lake Support(https://github.com/oap-project/gluten/issues/3378 ): DeltaLake, Iceberg, Hudi, Paimon

Do you know what changes to expect for Delta?

thanks

[weiting-chen]

@weiting-chen, for item

• Data Lake Support([VL] Unified design for data lake read support in Gluten + Velox #3378 ): DeltaLake, Iceberg, Hudi, Paimon

Do you know what changes to expect for Delta?

thanks

The plan is to support both COW table and MOR table this year, which means DeltaLake with and without Deletion Vector support.

[felipepessoto]

Thanks @weiting-chen

By Delta Deletion Vector support you mean for reads and writes?

[zemin-piao]

Hello @weiting-chen,

Thanks a lot for concluding the roadmap.

Regarding supporting Spark 3.5.1 in general, I saw there are two PRs merged.

https://github.com/apache/incubator-gluten/pull/4822 https://github.com/apache/incubator-gluten/pull/4845

May I ask there are any other works to be done in order to support Spark 3.5.1 on the plan, or these MRs concludes the work to support Spark 3.5.1?

[yjshen]

Thanks @weiting-chen for sharing the roadmap.

Regarding the item Other Big Data Framework Support, are there any designs or action items that could be shared here? Additionally, I saw an existing but archived effort at https://github.com/oap-project/Gluten-Trino, could you please provide more details or share your thoughts about it?