Implement conversion from ColumnStatistics to NullableInterval - Githubissues

apache / datafusion

Apache DataFusion SQL Query Engine

https://datafusion.apache.org/

Apache License 2.0

5.29k stars 974 forks source link

Implement conversion from ColumnStatistics to NullableInterval #10510

Closed dmitrybugakov closed 2 weeks ago

dmitrybugakov commented 2 weeks ago

Which issue does this PR close?

This change addresses part of #10456.

Rationale for this change

What changes are included in this PR?

Added conversion from ColumnStatistics to NullableInterval to handle different nullability scenarios.
Defined Interval struct to represent the range of values in a column.
Implemented From for NullableInterval, handling Precision cases and determining nullability.
Added Unknown variant to NullableInterval to manage cases with insufficient statistics.

Are these changes tested?

Are there any user-facing changes?

dmitrybugakov commented 2 weeks ago

@alamb Do we want to introduce a Type in ColumnStats in this PR? I found that the changes will affect Proto and may require other changes that are not clear for me at the moment.

https://github.com/apache/datafusion/blob/842f3933e3496a022984c2a37254475a3bcde1bf/datafusion/core/src/datasource/statistics.rs#L141

https://github.com/apache/datafusion/blob/842f3933e3496a022984c2a37254475a3bcde1bf/datafusion/core/src/datasource/physical_plan/file_scan_config.rs#L124

Or should I close the PR and create a new issue instead?