search

vincenzobaz / spark-scala3

Apache License 2.0
89 stars 15 forks source link

Fix and test BigDecimal and BigInteger handling #57

Closed dickwall closed 10 months ago

dickwall commented 10 months ago

While using scala3encoders on some external data represented as BigDecimal in parquet, I discovered a few issues with the big decimal and big int(eger) handling and corrected for them, along with tests.

The choice of BigDecimal(38,18) for precision is an arbitrary choice but seems to be something of a standard for Parquet from what I can see.

dickwall commented 10 months ago

by the way, thanks for this library and for the excellent tutorials, loving that I can use Scala 3 for spark usage now.