search

datastrato / gravitino

World's most powerful data catalog service with providing a high-performance, geo-distributed and federated metadata lake.
https://datastrato.ai/docs/
Apache License 2.0
347 stars 150 forks source link

[#3186] feat(spark-connector): Support Iceberg Spark Procedure #3258

Closed caican00 closed 1 day ago

caican00 commented 2 weeks ago

What changes were proposed in this pull request?

Support Iceberg Spark Procedure.

Why are the changes needed?

Support manage Iceberg metadata using Spark SQL.

Fix: https://github.com/datastrato/gravitino/issues/3186

Does this PR introduce any user-facing change?

No.

How was this patch tested?

New ITs.

caican00 commented 4 days ago

Hi @FANNG1 could you help review this pr when you are free?

FANNG1 commented 3 days ago

Hi @FANNG1 could you help review this pr when you are free?

ok

caican00 commented 3 days ago

comments have been addressed. cc @FANNG1

caican00 commented 3 days ago

comments have been addressed. cc @FANNG1

@FANNG1 could you help review this again?

FANNG1 commented 1 day ago

LGTM except two comments

FANNG1 commented 1 day ago

@caican00 could you rebase with lattest code?

caican00 commented 1 day ago

building failed as NameIdentifier cannot be converted to String in CatalogKafkaIT.

/home/runner/work/gravitino/gravitino/catalogs/catalog-kafka/src/test/java/com/datastrato/gravitino/catalog/kafka/integration/test/CatalogKafkaIT.java:181: error: no suitable method found for createCatalog(NameIdentifier,Type,String,String,ImmutableMap<String,String>)
        metalake.createCatalog(
                ^
    method SupportsCatalogs.createCatalog(String,Type,String,String,Map<String,String>) is not applicable
      (argument mismatch; NameIdentifier cannot be converted to String)
    method GravitinoMetalake.createCatalog(String,Type,String,String,Map<String,String>) is not applicable

> Task :catalogs:catalog-kafka:compileTestJava
      (argument mismatch; NameIdentifier cannot be converted to String)
caican00 commented 1 day ago

@caican00 could you rebase with lattest code?

It is already the latest code

caican00 commented 1 day ago

building failed as NameIdentifier cannot be converted to String in CatalogKafkaIT.

/home/runner/work/gravitino/gravitino/catalogs/catalog-kafka/src/test/java/com/datastrato/gravitino/catalog/kafka/integration/test/CatalogKafkaIT.java:181: error: no suitable method found for createCatalog(NameIdentifier,Type,String,String,ImmutableMap<String,String>)
        metalake.createCatalog(
                ^
    method SupportsCatalogs.createCatalog(String,Type,String,String,Map<String,String>) is not applicable
      (argument mismatch; NameIdentifier cannot be converted to String)
    method GravitinoMetalake.createCatalog(String,Type,String,String,Map<String,String>) is not applicable

> Task :catalogs:catalog-kafka:compileTestJava
      (argument mismatch; NameIdentifier cannot be converted to String)

let's wait https://github.com/datastrato/gravitino/pull/3424 to be merged.

caican00 commented 1 day ago

@FANNG1 could you please help re-trigger the pipeline? thank you.

FANNG1 commented 1 day ago

@caican00 ,merged in main, thanks for your work