Open palmerlao opened 7 years ago
I have an alternative proposal here.
What if we have a way to flatten schema's with Optional fields.
Example:
case class O(x: Option[Int], y: Option[Int])
val df: TypedDataset[O] = ...
val dfFlat = df.flatten : TypedDataset[(Int,Int)]
What happens here is that df
may have any of its fields null
(hence the Option
type). When we do the flatten, it essentially filter outs any null values and goes from Option[A]
to A
. It will do that for any optional fields.
@palmerlao Do you think this would solve your use case?
Let me know if this is what you mean. In your example, say that df
was made by
TypedDataset.create(Seq((Some(1), Some(2)), (None, Some(4)), (None, None)))
.
Then your proposed flatten
would give me dfFlat.collect.run
something along the lines of
Seq((1, 2))
?
It would be tough to make that work for my use case. Unfortunately, we work with highly denormalized data and nulls are very common.
However, I think there is some interest at my company for somehow building over frameless with Monocle. Do you think that would be something that other people find useful?
I see what you mean
This is also an issue the we got bitten by lately. Unfortunately we cannot adapt our model and thus, for now, need to use some hacky non-typechecked workarounds, which makes me sad.
I am way to new to shapeless and frameless to make a valuable contribution here, but I really hope that this is in general solvable.
I was hopping to get this working in https://github.com/typelevel/frameless/issues/204, but I hit a wall with UDFs. The idea is to be able to do a map on an optional column TypedColumn[T, Option[X]]
and then get back an unwrapped TypedColumn[T,X]
so you can do anything you would if the Option was not there. I will probably try to do some work around this for the 0.6.0 release (0.5.0 is already out the door).
In the meanwhile you can probably work around this using a UDF, but you will have to serialize the entire column. If that is not an issue for you, then a UDF is a fairly ok typesafe work around.
t.makeUDF( (x: Option[Foo]) => x.bar + 1)
Got the same issue.
Ran into this pretty quick as soon as we tried to do joinLeft
and wondering if there's anything new to report. I asked in gitter as well; sorry for the spam!
@palmerlao https://github.com/typelevel/frameless/pull/479 helps with this issues.
My understanding is that
Option
should be used to represent columns that one might mark nullable in vanilla Spark. I tried something along the lines of the following:The last line resulted in
What I think is reasonable is to return something of type
TypedColumn[A, Option[Int]]
. For comparison, in regular Spark:which I would expect
as.select(as.colMany('ob, 'i).show().run
to be roughly equivalent to up to some decisions on whether to displaynull
orNone
. Perhaps a reasonable way to approach this problem is to integrate the column selection mechanism with some kind of optics.