sparkutils / quality

A Quality Spark DQ Library
https://sparkutils.github.io/quality/
Apache License 2.0
4 stars 2 forks source link

add async filter optimisation for cast removal #41

Open chris-twiner opened 1 year ago

chris-twiner commented 1 year ago

given

create view table_view select *. cast( longfield as string) longfield_string from thetable;
select * from table_view where longfield_string = '12353463434'

the query will be treated as:

select * form table_view where cast (longfield as string) = '12353463434'

it should instead be treated as:

select * form table_view where longfield = cast ( '12353463434' as long) 

which will constant fold and allow longfield to be pushed down

chris-twiner commented 1 year ago

low prio due to demand change and potentially not sufficient configuration concept (i.e. given how much cast does it's possible a case is not possible to unpack - non-deterministic expressions etc.)