Open Jolanrensen opened 2 years ago
While Dataset<Option<*>> works, whereever a schema needs to be created, such as Dataset<Pair<*, Option<*>>, then it fails. Option is a Product but is handled as if it doesn't exist in Scala. Meaning Option<T> is encoded as T.
Dataset<Option<*>>
Dataset<Pair<*, Option<*>>
Option
Product
Option<T>
T
This needs to be looked into
While
Dataset<Option<*>>
works, whereever a schema needs to be created, such asDataset<Pair<*, Option<*>>
, then it fails.Option
is aProduct
but is handled as if it doesn't exist in Scala. MeaningOption<T>
is encoded asT
.This needs to be looked into