search

vincenzobaz / spark-scala3

Apache License 2.0
89 stars 15 forks source link

add support for parsing java.lang.Double #32

Closed Anghammar closed 1 year ago

Anghammar commented 1 year ago

This was just something I used personally so I thought I might as well do a pull request. I'm not sure if you are happy with the names I had to give to the givens though (as to not conflict with the generated name of the serializer/deserializer of Double).

vincenzobaz commented 1 year ago

Could we use the implicit conversions in the scala standard library to automatically derive java.lang types from scala ones?

adpi2 commented 1 year ago

Could we use the implicit conversions in the scala standard library to automatically derive java.lang types from scala ones?

Indeed, something like this could work:

given [T] (using Conversion[T, Double]): Serializer[T] with
    def inputType: DataType = DoubleType
    def serialize(inputObject: Expression): Expression = inputObject

But we should be cautious about custom conversion to double. If I define my own Conversion[Foo, Double] it will create a Serializer[Foo] which would probably fail at runtime because Foo cannot be cast to Double.

Anghammar commented 1 year ago

LGTM, thanks!

Don't you want to add the support for the other Java boxed types (java.lang.Long, java.lang.Float...)?

Yeah sure. I just added what I needed personally for parsing (at least one kind of) floats with null values.

Could we use the implicit conversions in the scala standard library to automatically derive java.lang types from scala ones?

Indeed, something like this could work:

given [T] (using Conversion[T, Double]): Serializer[T] with
    def inputType: DataType = DoubleType
    def serialize(inputObject: Expression): Expression = inputObject

But we should be cautious about custom conversion to double. If I define my own Conversion[Foo, Double] it will create a Serializer[Foo] which would probably fail at runtime because Foo cannot be cast to Double.

I'm sacred of implicit conversions of javas boxed types ever since I had a bug where the argument to the constructor of the companion object to Option where implicitly converted from null to zero.

adpi2 commented 1 year ago

I'm sacred of implicit conversions of javas boxed types ever since I had a bug where the argument to the constructor of the companion object to Option where implicitly converted from null to zero.

It's probably safer to avoid implicit conversions.

vincenzobaz commented 1 year ago

I meant more https://www.scala-lang.org/api/3.3.0/scala/math/ScalaNumericAnyConversions.html# or something similar: the scala standard library already knows how to obtain a scala Double from a java Double