datastrato / gravitino

World's most powerful data catalog service with providing a high-performance, geo-distributed and federated metadata lake.
https://datastrato.ai/docs/
Apache License 2.0
347 stars 150 forks source link

[#2543] feat(spark-connector): support row-level operations to iceberg Table #3243

Closed caican00 closed 5 days ago

caican00 commented 2 weeks ago

What changes were proposed in this pull request?

1. update tableName set c1=v1, c2=v2, ...

2. merge into targetTable t
   using sourceTable s
   on s.key=t.key
   when matched then ...
   when not matched then ...

3. delete from table where xxx

Why are the changes needed?

  1. For spark-connector in Iceberg, it explicitly uses SparkTable to identify whether it is an Iceberg table, so the SparkIcebergTable must extend SparkTable.

  2. support row-level operations to iceberg Table.

Fix: https://github.com/datastrato/gravitino/issues/2543

Does this PR introduce any user-facing change?

Yes, support update ... , merge into ..., delete from ...

How was this patch tested?

New ITs.

caican00 commented 1 week ago

Hi @FANNG1 could you help review this pr? thank you.

caican00 commented 1 week ago

all comments have been addressed. cc @FANNG1

caican00 commented 1 week ago

all comments have been addressed except one in discussion. cc @FANNG1

FANNG1 commented 1 week ago

@jerryshao @qqqttt123 do you have time to review? Overall LGTM except minor comment.

caican00 commented 6 days ago

@FANNG1 all comments have been addressed.

qqqttt123 commented 6 days ago

If this is a user-facing change, you should add the documents.

caican00 commented 5 days ago

If this is a user-facing change, you should add the documents.

done.

caican00 commented 5 days ago

@FANNG1 could you help review again?

FANNG1 commented 5 days ago

@caican00 could you fix the conflict?

FANNG1 commented 5 days ago

@qqqttt123 do you have other comments?

caican00 commented 5 days ago

@caican00 could you fix the conflict?

done

caican00 commented 5 days ago

Hi @FANNG1 @qqqttt123 are there any else comments to fix?

FANNG1 commented 5 days ago

Hi @FANNG1 @qqqttt123 are there any else comments to fix?

LGTM,will merge the PR if no other comments this afternoon

FANNG1 commented 5 days ago

@caican00 great process for supporting the Iceberg catalog, thanks for your work.