Open echai58 opened 7 months ago
It looks like we are building directly a physical plan here so that's why there is no type coercion.
We probably need to refactor this.
@ion-elgreco are there any updates on this?
@echai58 this PR got merged: https://github.com/delta-io/delta-rs/commit/f56d8c98ce5656ff4aee93aa3861efcdb322eda3, so it should be doable to refactor the delete operation now.
@ion-elgreco when will the predicate problem for delete/update operations be done ? I also met the problem.
@ion-elgreco when will the predicate problem for delete/update operations be done ? I also met the problem.
I dont have any plans to work on this. But feel free to pick it up
Environment
Delta-rs version: 0.13.0
Binding: python bindings
Environment:
Bug
What happened: It seems that when a string predicate is passed into
DeltaTable.delete
, when it gets parsed as a Datafusion Expression, it is not taking into account the schema of the table. For example, if there is a column with typepa.int32()
, and you try to use a predicate likeprice = 100
, it raises an errorValueError: Invalid comparison operation: Int32 <= Int64
, which I assume is coming from100
being parsed as a int64. This is supported by if I pass inprice = CAST(100 as INT)
instead, it works as expected.What you expected to happen: The parser should be schema-aware when converting the string predicate to a Datafusion Expression.
How to reproduce it: This is a minimal reproduction: