search

sparkutils / quality

A Quality Spark DQ Library
https://sparkutils.github.io/quality/
Apache License 2.0
4 stars 2 forks source link

Spark 4 #60

Open chris-twiner opened 6 months ago

chris-twiner commented 6 months ago

will be kept open until 4 drops, periodically re-running as needed

chris-twiner commented 6 months ago

1 test fail (other than delta)

4 has ArraySeq$ofRef :

image

3.5 has GenericArrayData :

image

chris-twiner commented 6 months ago

ArraySeq$ofRef is no longer a scala.collection.immutable.Seq but a scala.collection.Seq (not the same in 2.13) so the encoder creates the latter, not the former. CatalystConverters then does case other => other and just returns the thing.

chris-twiner commented 6 months ago

Delta doesn't work reliably, it'd need a proper supporting release (the anttlr versions are too different)

chris-twiner commented 2 months ago

tested on current snaps (after preview) and rc1 delta 4 against 0.1.3-RC5