search

apache / iceberg

Apache Iceberg
https://iceberg.apache.org/
Apache License 2.0
6.44k stars 2.22k forks source link

alter table drop partition can't use [if exists] keyword, why reason? #3554

Closed felixYyu closed 2 years ago

felixYyu commented 2 years ago

ALTER TABLE prod.db.sample DROP [if exists] PARTITION FIELD catalog image

felixYyu commented 2 years ago

ALTER TABLE ... ADD if exsits PARTITION FIELD col also

felixYyu commented 2 years ago

cc @rdblue @aokolnychyi @RussellSpitzer can you help? thanks base on iceberg-spark3-runtime version 0.12.1

RussellSpitzer commented 2 years ago

Those particles are not supported in the Spark SQL extension we use to support partition field modifications

https://github.com/apache/iceberg/blob/master/spark/v3.2/spark-extensions/src/main/antlr/org.apache.spark.sql.catalyst.parser.extensions/IcebergSqlExtensions.g4#L70-L71

When we switch to using the DataSourceV2 "alterPartition" capability which was recently added to Spark, we will be able to use the native grammar.

Supports Alter Partition Interface in Spark https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/SupportsPartitionManagement.java

Spark Grammar https://github.com/apache/spark/blob/master/sql/catalyst/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBase.g4#L178-L179

felixYyu commented 2 years ago

Will the spark SQL extension be supported in the future？

RussellSpitzer commented 2 years ago

Pull requests are welcome :) I think I actually saw someone working on this with "Show Partitions" support which uses the same extension point

felixYyu commented 2 years ago

I'm well, this PR assigne to me please. thanks Russell.

RussellSpitzer commented 2 years ago

Sounds good, let's close out this issue and i'll make a new one for the support drop partition.