Open Jolanrensen opened 2 years ago
While Dataset<Option<*>> works, whereever a schema needs to be created, such as Dataset<Pair<*, Option<*>>, then it fails. Option is a Product but is handled as if it doesn't exist in Scala. Meaning Option<T> is encoded as T.
Dataset<Option<*>>
Dataset<Pair<*, Option<*>>
Option
Product
Option<T>
T
This needs to be looked into
While
Dataset<Option<*>>works, whereever a schema needs to be created, such asDataset<Pair<*, Option<*>>, then it fails.Optionis aProductbut is handled as if it doesn't exist in Scala. MeaningOption<T>is encoded asT.This needs to be looked into