search

apache / gravitino

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.
https://gravitino.apache.org
Apache License 2.0
762 stars 235 forks source link

[Improvement] test more Iceberg versions in spark connector #3778

Open FANNG1 opened 1 month ago

FANNG1 commented 1 month ago

What would you like to be improved?

For now, Iceberg versions are compiled and test in 1.4.3, we'd better test 1.3, and 1.5 versions.

How should we improve?

It's expensive to add another -PicebergVersion to run test, I suggest to test Iceberg 1.3 in Spark3.3, 1.4 in Spark3.4, 1.5 in spark3.5

FANNG1 commented 1 month ago

@jerryshao @caican00 WDYT?

caican00 commented 1 month ago

@jerryshao @caican00 WDYT?

FYI: The spark connector does not contain the iceberg runtime jar. Users generally use the iceberg runtime jar maintained by themselves. It seems unnecessary, and since iceberg1.4 and 1.5 both support spark3.3, spark3.4, and spark3.5, just testing Iceberg 1.3 in Spark3.3, 1.4 in Spark3.4, 1.5 in spark3.5 seems not comprehensive enough.