delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python
https://delta-io.github.io/delta-rs/
Apache License 2.0
1.99k stars 362 forks source link

Merge on predicate throw error on date colum: Unable to convert expression to string #2420

Closed ibabeo closed 2 months ago

ibabeo commented 2 months ago

Environment

Delta-rs version: 0.16.4

Binding: python

Environment:


Bug

What happened: Merge on predicate throw error when the merge key is a date column, which table is partitioned by.

How to reproduce it:

import pandas as pd
from datetime import date
from deltalake import DeltaTable, write_deltalake
import tempfile 

f  = tempfile.TemporaryDirectory().name
df = pd.DataFrame(data={"Date": [date(2023,1,1)] *10 + [date(2024,1,1)]*10, "B":range(20), "C":"constant"})
write_deltalake(f, df, mode='append', partition_by=["Date"])
dt = DeltaTable(f)
df_new = pd.DataFrame(data={"Date":[date(2023,1,1)], "B":[1], "C":"new value"})
dt.merge(source=df_new, predicate='t.Date = s.Date AND t.B = s.B', target_alias='t', source_alias='s').when_matched_update_all().execute()

DeltaError Traceback (most recent call last) ... \lib\site-packages\deltalake\table.py] in execute(self) 1776 Dict: metrics 1777 """ -> 1778 metrics = self.table._table.merge_execute( 1779 source=self.source, 1780 predicate=self.predicate,

DeltaError: Generic DeltaTable error: Unable to convert expression to string

More details: The same code works in deltalake 0.16.1

ion-elgreco commented 2 months ago

This is already fixed here https://github.com/delta-io/delta-rs/pull/2357

And will be available in the next release