search

apache / arrow-rs

Official Rust implementation of Apache Arrow
https://arrow.apache.org/
Apache License 2.0
2.57k stars 779 forks source link

Support constant array. #5701

Open liurenjie1024 opened 6 months ago

liurenjie1024 commented 6 months ago

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

A constant array would be useful for improving performance, see https://github.com/apache/iceberg-rust/pull/295#discussion_r1582166733

Describe the solution you'd like

Allow constructing constant array.

Describe alternatives you've considered

Additional context

tustvold commented 6 months ago

Is https://docs.rs/arrow-array/latest/arrow_array/struct.Scalar.html perhaps what you are looking for?

liurenjie1024 commented 6 months ago

Is https://docs.rs/arrow-array/latest/arrow_array/struct.Scalar.html perhaps what you are looking for?

Not quite same. Here is the case, when we want to convert iceberg predicate to ArrowPredicateFn, say a < 1, for some reason here a is a constant, e.g. NULL, we can always return true. But ArrowPredicateFn requires to return an BooleanArray.

mapleFU commented 6 months ago

https://arrow.apache.org/docs/format/Columnar.html

Seems arrow doesn't have an constant array ( But array with type "Null" is similiar to that), maybe send this to maillist is prefered?

liurenjie1024 commented 6 months ago

https://arrow.apache.org/docs/format/Columnar.html

Seems arrow doesn't have an constant array ( But array with type "Null" is similiar to that), maybe send this to maillist is prefered?

Cool, I will send a mail to dev list.

tustvold commented 6 months ago

Constant propogation is something I'd expect to be handled by a query optimizer, perhaps in conjunction with range analysis. FWIW Datafusion is already capable of this

liurenjie1024 commented 6 months ago

Yes, it possible to be handled by expression rewrite.