search

apache / incubator-xtable

Apache XTable (incubating) is a cross-table converter for lakehouse table formats that facilitates interoperability across data processing systems and query engines.
https://xtable.apache.org/
Apache License 2.0
919 stars 147 forks source link

Wrongly handling non-numeric values of double type during the conversion from Delta #524

Closed emilie-wang closed 2 months ago

emilie-wang commented 3 months ago

Search before asking

Please describe the bug 🐞

This error happened during the translation of column stats from Delta → internal table →Iceberg. There is an edge case, when the double type field’s value is non-numeric like "NaN" or "Infinity", the value type become String when processing the snapshot from Delta source.

...INFO org.apache.xtable.iceberg.IcebergColumnStatsConverter:76 - name=weight, fieldType=double, valueType=class java.lang.String, getMinValue=-Infinity 
...INFO org.apache.xtable.iceberg.IcebergColumnStatsConverter:80 - name=weight, fieldType=double, valueType=class java.lang.String, getMaxValue=NaN

Are you willing to submit PR?

Code of Conduct