sunchao / parquet-rs

Apache Parquet implementation in Rust
Apache License 2.0
149 stars 20 forks source link

Add Decimal support 2 #103

Closed sadikovi closed 6 years ago

sadikovi commented 6 years ago

This PR attempts to introduce Decimal again. This time changes include:

As it was previously noted, Parquet-cpp/mr, Arrow explicitly support decimal as int128 value. Spark uses Java BigDecimal, which supports arbitrary precision values, but Spark itself limits Decimal to precision 38, which is int128.

We do not store decimal as a value, but rather 3 parts: unscaled value as bytes, precision and scale, which is different from Spark, for example, where they try storing it as long. When converting to a string we use num-bigint, which, similar to Java BigDecimal, should support arbitrary precision.

coveralls commented 6 years ago

Coverage Status

Coverage increased (+6.0e-05%) to 94.96% when pulling f0cc2f03222c8a98441c9524129988dd6f4d948f on sadikovi:add-decimal-2 into faf33c8cda25b3478d8816ffc9c0765dd390c385 on sunchao:master.

sunchao commented 6 years ago

Merged. Thanks @sadikovi !

sadikovi commented 6 years ago

Thank you very much!