Adding RESTCatalog based Spark Integ Test

haizhou-zhao commented 2 weeks ago

Feature Request / Improvement

Current Spark smoke test runs on Hive Catalog, it would be nice if we run the same test cases on top of REST Catalog as well. Example: https://github.com/apache/iceberg/blob/main/spark/v3.5/spark-runtime/src/integration/java/org/apache/iceberg/spark/SmokeTest.java

Query engine

None

Willingness to contribute

[X] I can contribute this improvement/feature independently
[ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community
[ ] I cannot contribute this improvement/feature at this time

haizhou-zhao commented 1 week ago

cc: @danielcweeks

As discussed offline, I think the we can build on top of RCK classes from https://github.com/apache/iceberg/pull/10908 for REST Catalog based spark integration tests.

haizhou-zhao commented 1 week ago

Found several issues while attempting to enable integration test based on REST Catalog for Spark. Listing them below as I troubleshoot them one by one.

haizhou-zhao commented 1 week ago

https://github.com/apache/iceberg/issues/11103

haizhou-zhao commented 1 week ago

https://github.com/apache/iceberg/issues/11109

haizhou-zhao commented 2 days ago

https://github.com/apache/iceberg/issues/11154

apache / iceberg