Support Apache Spark 3.3.0 - Githubissues

eto-ai / rikai

Parquet-based ML data format optimized for working with unstructured data

https://rikai.readthedocs.io/en/latest/

Apache License 2.0

136 stars 19 forks source link

Support Apache Spark 3.3.0 #682

Closed da-liii closed 1 year ago

da-liii commented 2 years ago

https://spark.apache.org/releases/spark-release-3-3-0.html

Three Highlights for Rikai

Support complex types for Parquet vectorized reader (SPARK-34863)
Provide a profiler for Python/Pandas UDFs (SPARK-37443)
More comprehensive DS V2 push down capabilities (SPARK-38788)