search

apache / iceberg-rust

Apache Iceberg
https://rust.iceberg.apache.org/
Apache License 2.0
675 stars 159 forks source link

feat(datafusion): Support pushdown more datafusion exprs to Iceberg #649

Closed FANNG1 closed 1 month ago

FANNG1 commented 2 months ago

fixes: #644

DataFusion expressions and Iceberg predicates do not have a one-to-one correspondence.

  1. Columns and literals are represented as Expr in DataFusion, whereas they are merely components of predicates in Iceberg.
  2. And and Or are operators in DataFusion, but they are treated as predicates in Iceberg.

This disparity complicates the transformation process when using TreeNode. To address this, I also refactored the code to simplify it.

FANNG1 commented 1 month ago

@liurenjie1024 @Xuanwo @sdd @a-agmon please help to review when you are free, thanks

FANNG1 commented 1 month ago

thanks @sdd @a-agmon @Xuanwo for the reviewing.