This PR attempts to introduce Decimal again. This time changes include:
New value type Decimal in basic.rs alongside ByteArray and Int96. Since decimal can be created from i32, i64 or ByteArray, we have enum with 3 variations of underlying storage (either array of clone of ByteArray).
Added conversion from INT32, INT64 and BYTE_ARRAY, and FIXED_BYTE_ARRAY.
Added method to convert decimal to a string representation, similar to timestamp and date.
Had to add dependency on num-bigint because of that.
Added tests.
As it was previously noted, Parquet-cpp/mr, Arrow explicitly support decimal as int128 value. Spark uses Java BigDecimal, which supports arbitrary precision values, but Spark itself limits Decimal to precision 38, which is int128.
We do not store decimal as a value, but rather 3 parts: unscaled value as bytes, precision and scale, which is different from Spark, for example, where they try storing it as long. When converting to a string we use num-bigint, which, similar to Java BigDecimal, should support arbitrary precision.
Coverage increased (+6.0e-05%) to 94.96% when pulling f0cc2f03222c8a98441c9524129988dd6f4d948f on sadikovi:add-decimal-2 into faf33c8cda25b3478d8816ffc9c0765dd390c385 on sunchao:master.
This PR attempts to introduce
Decimal
again. This time changes include:Decimal
inbasic.rs
alongside ByteArray and Int96. Since decimal can be created fromi32
,i64
orByteArray
, we have enum with 3 variations of underlying storage (either array of clone of ByteArray).num-bigint
because of that.As it was previously noted, Parquet-cpp/mr, Arrow explicitly support decimal as int128 value. Spark uses Java BigDecimal, which supports arbitrary precision values, but Spark itself limits Decimal to precision 38, which is int128.
We do not store decimal as a value, but rather 3 parts: unscaled value as bytes, precision and scale, which is different from Spark, for example, where they try storing it as long. When converting to a string we use
num-bigint
, which, similar to JavaBigDecimal
, should support arbitrary precision.