[#2543] feat(spark-connector): support row-level operations to iceberg Table - Githubissues

datastrato / gravitino

World's most powerful data catalog service with providing a high-performance, geo-distributed and federated metadata lake.

https://datastrato.ai/docs/

Apache License 2.0

401 stars 166 forks source link

[#2543] feat(spark-connector): support row-level operations to iceberg Table #3366

Closed jerryshao closed 2 weeks ago

jerryshao commented 2 weeks ago

What changes were proposed in this pull request?

refactor table implementation, make SparkIcebergTable extend Iceberg SparkTable, and SparkHiveTable extend Kyuubi HiveTable.
support row-level operations to iceberg Table

1. update tableName set c1=v1, c2=v2, ...

2. merge into targetTable t
   using sourceTable s
   on s.key=t.key
   when matched then ...
   when not matched then ...

3. delete from table where xxx

Why are the changes needed?

For spark-connector in Iceberg, it explicitly uses SparkTable to identify whether it is an Iceberg table, so the SparkIcebergTable must extend SparkTable.
support row-level operations to iceberg Table.

Fix: https://github.com/datastrato/gravitino/issues/2543

Does this PR introduce any user-facing change?

Yes, support update ... , merge into ..., delete from ...

How was this patch tested?

New ITs.

FANNG1 commented 2 weeks ago

@caican00 could analyse the erorr and fix it?

FANNG1 commented 2 weeks ago

this PR depends on #3358

FANNG1 commented 2 weeks ago

@caican00 could you fix the conflict?

jerryshao commented 2 weeks ago

Why there is a conflict in this part? AFAIK this part should be quite close in branch-0.5 and main.

FANNG1 commented 2 weeks ago

spark-connector/spark-connector/src/main/java/com/datastrato/gravitino/spark/connector/hive/GravitinoHiveCatalog.java

Maybe a github bug, I could patch the PR to branch-0.5 manually without modification.

jerryshao commented 2 weeks ago

Anyway, @caican00 would you please cherry-pick this commit to branch-0.5 and create a new PR? I'm going to close this PR.

caican00 commented 2 weeks ago

Anyway, @caican00 would you please cherry-pick this commit to branch-0.5 and create a new PR? I'm going to close this PR.

Sorry for the late reply. i have created a new pr for this commit to branch-0.5 cc @jerryshao @FANNG1