search

sparkutils / quality

A Quality Spark DQ Library
https://sparkutils.github.io/quality/
Apache License 2.0
4 stars 2 forks source link

expressionRunner - needs better serialization options #42

Closed chris-twiner closed 1 year ago

chris-twiner commented 1 year ago

cast does a great job of writing out, but not of reading.

json functions don't support map's with more than string keys, so useful grouping etc. won't work.

chris-twiner commented 1 year ago

CalendarIntervalType is different across spark impls - needs compat layer and forward/backwards compat? Neither TimestampNTZType YearMonthIntervalType DayTimeIntervalType are only on 3.3 and above UDTs need sql package so the compat layer needs to be there.

chris-twiner commented 1 year ago

a1845c3 has issues as 1.33 is used by 13.1/2 but the others use 1.24 which is not interface compatible. c831a29 tested against 12.2, 13.1/2 (using .1), 11.3, 10.4, 9.1