Iceberg TTL setting

[smileyboy2019]

Feature Request / Improvement

iceberg The expired table of the table can be automatically archived, and TTL can be set for the table. At the same time, the archiving path can be set, and it can be automatically archived. When querying archived data, it has no impact on the user.

Query engine

None

[jbonofre]

By TTL, you mean on a table snapshot ?

Table snapshot has a timestamp, so, we could have a TTL based on snapshots timestamp and remove "expired" snapshot. Is it what you mean ?

Do you have an use case where this feature would make sense ? Is it just a way to "reduce" the manifest file ?

Thanks !

[BsoBird]

Do you mean that iceberg can support writing data to different paths in multi-level storage? Example: Tiered Storage on hdfs

I'm not sure I understand you mean correctly. For example, the iceberg table currently has three entries, last year's data, last month's data, and today's data. Suppose you set the TTL to three months, and you want to dump the data beyond three months into cold storage? @smileyboy2019

[smileyboy2019]

是支持生命周期的TTL，同时设置归档目录，到期的数据自动归档历史数据路径里面

//TTL时间设置，超过7天数据自动删除 CREATE TABLE my_table ( id INT, name STRING ) PARTITIONED BY (dt STRING, city STRING) TBLPROPERTIES ("hive.table.parameters.ttl" = "7 days"); //对这张表超过7天数据进行归档，自动存放到对应目录 set hive.archive.enabled=true; hive> set hive.archive.har.parentdir.settable=true; hive> set har.partfile.size=1099511627776;

 

志阳 @.***

 

------------------ 原始邮件 ------------------ 发件人: "PLASH @.>; 发送时间: 2024年5月27日(星期一) 下午5:30 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [apache/iceberg] iceberg TTL set (Issue #10372)

你是指iceberg可以支持将数据写入到多级存储的不同路径中么?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>