MrPowers / bebe

Filling in the Spark function gaps across APIs
50 stars 5 forks source link

Add column extractors by type #12

Closed alfonsorr closed 3 years ago

alfonsorr commented 3 years ago

Also added the column common functions implicit class

The extractors will allow something that I find interesting, and right now its hard to do in spark, to diverge the logic by the type of the column, an example will clarify more:

    def transformDateOrTimestamp(colName: String)(df:DataFrame): IntegerColumn =
      df(colName) match {
        case DateColumn(dc) => dc.day_of_month
        case TimestampColumn(tc) => tc.day_of_month
        case _ => IntegerColumn(0)
      }

This combines the power of the pattern matching and custom extractors to describe easily.

The implicit class allows to add common logic to the typed columns, like the method "as" and when the Boolean type is added, type safety comparations of columns 😄

MrPowers commented 3 years ago

@alfonsorr - this is awesome, great work! Really excited about the possibilities this is opening up!

alfonsorr commented 3 years ago

Thanks! 🎉