vincenzobaz / spark-scala3

Apache License 2.0
89 stars 15 forks source link

Add support for `scala.math.BigInt` #22

Open DmytroMitin opened 2 years ago

DmytroMitin commented 2 years ago

https://stackoverflow.com/questions/74249859/spark-df-astype-fails-to-compile

vincenzobaz commented 2 years ago

Hello @DmytroMitin, thank you very much for this PR! Also amazing StackOverflow answer, it's great to spread knowledge about Spark and Scala 3

I am not very familiar with the spark type system unfortunately, so I have a question: why do you need to cases bigint and long? Are both Long and BigInt converted to the same underlying Spark datatype?

DmytroMitin commented 2 years ago

@vincenzobaz Well, I'm not an expert in Spark types either :) Unfortunately, it seems DecimalType codecs for BigInt are not always enough.

vincenzobaz commented 2 years ago

Thank you for the exhaustive explanation with examples, I hope it was not too painful to go through this

vincenzobaz commented 2 years ago

Since you seem to understand this topic well, If you are interested, I would be very interested in advice for #15