search

datastrato / gravitino

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.
https://datastrato.ai/docs/
Apache License 2.0
616 stars 193 forks source link

[#2447] feat(spark-connector): support alter table AddColumn and DropColumn for spark-connector #2458

Closed caican00 closed 3 months ago

caican00 commented 3 months ago

What changes were proposed in this pull request?

support alterTable addColumn and dropColumn for spark-connector in AlterTableCommand.

Why are the changes needed?

Implement addColumn and dropColumn ops for Spark AlterTableCommand.

Fix: #2447

Does this PR introduce any user-facing change?

Yes, users can run the following commands to modify the table columns using spark sql.

ALTER TABLE StudentInfo ADD columns (col1 string);

ALTER TABLE StudentInfo DROP columns (col1);

How was this patch tested?

New uts.

caican00 commented 3 months ago

cc @FANNG1

caican00 commented 3 months ago

Hi @FANNG1 all comments have been addressed.

caican00 commented 3 months ago

all comments have been addressed in addition to the sperate function described above. cc @FANNG1

FANNG1 commented 3 months ago

all comments have been addressed in addition to the sperate function described above. cc @FANNG1

thanks @caican00 , could you extend the test as commented? Overall LGTM

caican00 commented 3 months ago

all comments have been addressed in addition to the sperate function described above. cc @FANNG1

thanks @caican00 , could you extend the test as commented? Overall LGTM

updated. cc @FANNG1

FANNG1 commented 3 months ago

IT failed, could you fix it?


SparkIT > testAlterTableAddAndDeleteColumn() FAILED
    java.lang.UnsupportedOperationException
        at java.base/java.util.AbstractList.add(AbstractList.java:153)
        at java.base/java.util.AbstractList.add(AbstractList.java:111)
        at com.datastrato.gravitino.integration.test.spark.SparkIT.testAlterTableAddAndDeleteColumn(SparkIT.java:293)
caican00 commented 3 months ago

IT failed, could you fix it?

SparkIT > testAlterTableAddAndDeleteColumn() FAILED
    java.lang.UnsupportedOperationException
        at java.base/java.util.AbstractList.add(AbstractList.java:153)
        at java.base/java.util.AbstractList.add(AbstractList.java:111)
        at com.datastrato.gravitino.integration.test.spark.SparkIT.testAlterTableAddAndDeleteColumn(SparkIT.java:293)

i will fix it. Thanks for informing me

FANNG1 commented 3 months ago

@caican00 thanks for your work

caican00 commented 3 months ago

@caican00 thanks for your work

@FANNG1 thanks for reviewing

caican00 commented 3 months ago

Hi @FANNG1 I would like to do other work of spark-connector. May I ask where should I start?

FANNG1 commented 3 months ago

UpdateColumnPosition and UpdateColumnType is not supported yet, but I'm not sure whether @hiirrxnn is working on it. it's a good start to support primitive spark types like #2451 , and challenging to support List Map and Struct type.

FANNG1 commented 3 months ago

please create corresponding issues before working on it :)

caican00 commented 3 months ago

please create corresponding issues before working on it :)

i got it and thanks for your reply.