search

AbsaOSS / spark-commons

Apache License 2.0
7 stars 0 forks source link

#80 Add enforceTypeOnNullTypeFields function to DataFrameImplicits #81

Closed jakipatryk closed 1 year ago

jakipatryk commented 1 year ago

Closes #80

lsulak commented 1 year ago

LGTM (just read the code)

jakipatryk commented 1 year ago

I actually wonder about the universality of the function. 🤔 Why are NullType so special, that they have to be cast but other misaligned types are kept as they were?

  1. NullType can be casted to any type, so the function is expected to not fail in any case.
  2. NullType isn't cool, because it doesn't work when transform is called on it, and it breaks for example alignSchema :D
benedeki commented 1 year ago

I actually wonder about the universality of the function. 🤔 Why are NullType so special, that they have to be cast but other misaligned types are kept as they were?

1. NullType can be casted to any type, so the function is expected to not fail in any case.

2. NullType isn't cool, because it doesn't work when `transform` is called on it, and it breaks for example `alignSchema` :D

Agreed offline to keep the function in the library. 👍