Problem
Previously, we were writing unbounded numerics, that does not specify precision and scale (i.e. numeric), as text since they can be too large to represent as parquet decimal. Most of the time users ignores the precision for numeric columns, so they were written as text. That prevents pushing down some operators on the numeric type by execution engines.
Improvement
We start to read/write unbounded numerics as numeric(38, 16) to parquet file. We throw a runtime error if an unbounded numeric value exceeds 22 digits before the decimal point or 16 digits after the decimal point.
For the users that bump into the error, we give hint to change the column type to a numeric(p,s) with precision and scale specified, to get rid of the error.
Fixes
Arrow to pg conversions were not correct for some cases
when there is no decimal point e.g. 1234 (fixed by relying on pg compatible Decimal128Type::format_decimal by arrow)
when the scale is negative e.g. numeric(5,-2) (arrow does not allow negative scale, fixed by adding abs(scale) to precision and setting the scale to 0, which means numeric(5,-2) => numeric(7,0) at parquet file. copy from can convert it back to numeric(5,-2))
Problem Previously, we were writing unbounded numerics, that does not specify precision and scale (i.e.
numeric), as text since they can be too large to represent as parquet decimal. Most of the time users ignores the precision for numeric columns, so they were written as text. That prevents pushing down some operators on the numeric type by execution engines.Improvement We start to read/write unbounded numerics as
numeric(38, 16)to parquet file. We throw a runtime error if an unbounded numeric value exceeds22 digitsbefore the decimal point or16 digitsafter the decimal point.For the users that bump into the error, we give hint to change the column type to a numeric(p,s) with precision and scale specified, to get rid of the error.
Fixes Arrow to pg conversions were not correct for some cases
Decimal128Type::format_decimalby arrow)abs(scale)toprecisionand setting thescaleto 0, which meansnumeric(5,-2) => numeric(7,0)at parquet file. copy from can convert it back to numeric(5,-2))These cases are fixed and covered by tests.