Discussion: S3 soft deletion attributes

apache / iceberg-rust

Apache Iceberg

https://rust.iceberg.apache.org/

Apache License 2.0

592 stars 134 forks source link

Discussion: S3 soft deletion attributes #507

Open c-thiel opened 1 month ago

c-thiel commented 1 month ago

Currently our S3 FileIO does not support s3.delete.* attributes that are used to implement soft-deletion with a custom lifecycle policy.

Follow-up of https://github.com/apache/iceberg-rust/pull/505

Xuanwo commented 1 month ago

Copied my comment below:

The Iceberg Java FileIO implements soft deletes using PutObjectTags and a user-configured lifecycle to support it. Deleted objects can still be scanned or listed because S3 does not recognize the concept of soft deletes. While it's possible for OpenDAL to support this, I doubt its usefulness.

I personally think adopting the object version to implement soft-delete is a better idea for that.

liurenjie1024 commented 1 month ago

I'm not an s3 expert, but s3 docs shows that both custom tags and versioned object are two ways to allow custom lifecycle management, right? I'm not sure about the motivation behind the choice of using tags rather than versions. Maybe we could ask for help in dev list for the importance of supporting this feature?