Update to Spark 3.5 and Delta 3.0 [Spark] [Delta]

Qbeast-io / qbeast-spark

Qbeast-spark: DataSource enabling multi-dimensional indexing and efficient data sampling. Big Data, free from the unnecessary!

https://qbeast.io/qbeast-our-tech/

Apache License 2.0

215 stars 19 forks source link

Closed osopardo1 closed 8 months ago

osopardo1 commented 1 year ago

Following the news on Delta and Spark releases we should update theqbeast-spark libraries to keep up to date with the latest features.

This is the summary of the new developments included in both releases.

Read the full notes here.

Delta Universal Format: allow you to read Delta tables with Hudi and Iceberg clients.
Delta Kernel: decoupling reading and writing for building Delta Connectors.
Delta Spark: change on package name (before was delta-core)
Support for Spark 3.5
Better performance for Deletion Vectors

Read the full notes here.

osopardo1 commented 11 months ago

For the record: this update would be for main-1.0.0

osopardo1 commented 11 months ago

Some things I am experiencing with the versions upgrade:

withNewTransaction method from Delta is deprecated. Now they enforce to pass a series of new arguments as an Option to commit to a table:
- CatalogTable (if exists)
- Snapshot (if exists)
- Still need to read the code and understand it's role in the Metadata Writer.
Properties/options are not passed as usual to the saveAsTable / Catalog code. A.k.a:columnsToIndex is not propagated properly and throws an AnalysisException when writing to Qbeast.
Delta has a new trait called TableChanges that can be confused for our own TableChanges interface in package core. Be careful when importing.

osopardo1 commented 10 months ago

This feature is merged in 1.0.0-main!

osopardo1 commented 8 months ago