regression : delta.logRetentionDuration don't seems to be respected

[djouallah]

Environment

Delta-rs version: 17.1

Binding:

Environment: Python

---

Bug

What happened:

write_deltalake(delta_path, df,configuration = {"delta.logRetentionDuration": "interval 1 day"} ,mode="append",storage_options=storage_options)
dt = DeltaTable(delta_path,storage_options=storage_options)
dt.vacuum(retention_hours=0,dry_run=False,  enforce_retention_duration=False)
dt.create_checkpoint()
dt.cleanup_metadata()

don't seems to be working ?

[echai58]

I think the configuration key should be delta.logRetentionDuration.

[djouallah]

same issue

[ion-elgreco]

I think the configuration key should be delta.logRetentionDuration.

Correct

[djouallah]

@ion-elgreco delta.logRetentionDuration does not work either ?

[ion-elgreco]

@djouallah it does work, you provided interval 1 day, so you can't expect the logs to be deleted immediately : P, change it to 1 seconds interval and you can see they get removed

import polars as pl
from deltalake import DeltaTable, write_deltalake

df = pl.DataFrame({"foo": [1]})
delta_path = "test_Table"

write_deltalake(
    delta_path,
    df.to_arrow(),
    configuration={"delta.logRetentionDuration": "interval 1 seconds"},
    mode="overwrite",
)
dt = DeltaTable(delta_path)
dt.vacuum(retention_hours=0, dry_run=False, enforce_retention_duration=False)
dt.create_checkpoint()
dt.cleanup_metadata()

[djouallah]

no luck, I am writing to gcp fwiw

[ion-elgreco]

@djouallah please share the table configuration in the delta log

[djouallah]

Metadata(id: 62adbf63-1e61-479e-8187-8fd7ef308b5c, name: None, description: None, partition_columns: [], created_time: 1713594980946, configuration: {})

[ion-elgreco]

Metadata(id: 62adbf63-1e61-479e-8187-8fd7ef308b5c, name: None, description: None, partition_columns: [], created_time: 1713594980946, configuration: {})

Yeah, you didn't pass a configuration during creating so it's using the default of 30 days.

[djouallah]

ah, I see it has to be in the first time it was created, adding the option later using append or overwrite does not works, thanks !!!