polarsignals / frostdb

❄️ Coolest database around 🧊 Embeddable column database written in Go.
Apache License 2.0
1.31k stars 66 forks source link

Iceberg maintenance #853

Closed thorfour closed 5 months ago

thorfour commented 5 months ago

This addresses a feature request from #838 to be able to have the storage engine expire data files that exceed a certain age.

It does this in two parts; one it will delete the data files from the table by performing a normal table writer where it simply removes the entries from the manifests and then commits the change.

Then once those data files are stranded (i.e no snapshot references them anymore) the maintenance code will deem them to be orphaned files and delete them from the bucket.