AbsaOSS / spark-commons

Apache License 2.0
7 stars 0 forks source link

Add function casting all NullType fields to a target type #80

Closed jakipatryk closed 1 year ago

jakipatryk commented 1 year ago

Background

In Spark, a field can have NullType type if it contains only nulls (untyped). It is problematic especially if one expects this field to be an array of something, and calls transform on it - in that case, an exception is thrown.

Feature

Function that would convert all fields with NullTypes to their corresponding type in target schema.