search

sparkutils / quality

A Quality Spark DQ Library
https://sparkutils.github.io/quality/
Apache License 2.0
4 stars 2 forks source link

3.5 compat issues #51

Closed chris-twiner closed 8 months ago

chris-twiner commented 11 months ago

as it's out.... abstraction layer for:

  1. RowEncoder.apply no longer exists (one place in code)
  2. ViewLoading error handling, parse exceptions are now raised for errors which are pure parser (which makes sense as you can't have a plan if you can't parse)
  3. frameless doesn't work

the viewloading exception types themselves also need to move out

re 3.:

java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.ScalaReflection$.isNativeType(Lorg/apache/spark/sql/types/DataType;)Z

    at org.apache.spark.sql.reflection.package$.isNativeType(package.scala:16)
    at frameless.TypedEncoder$$anon$17.toCatalyst(TypedEncoder.scala:343)
    at frameless.TypedEncoder$$anon$19.toCatalyst(TypedEncoder.scala:505)
    at frameless.RecordEncoder.$anonfun$toCatalyst$2(RecordEncoder.scala:155)
    at scala.collection.immutable.List.map(List.scala:297)
    at frameless.RecordEncoder.toCatalyst(RecordEncoder.scala:153)
    at frameless.TypedExpressionEncoder$.apply(TypedExpressionEncoder.scala:28)
    at com.sparkutils.quality.impl.EncodersImplicits.ruleEngineResultExpEnc(SparkHelpers.scala:94)
chris-twiner commented 11 months ago

https://github.com/typelevel/frameless/issues/755

chris-twiner commented 11 months ago

The fix isn't 100% binary compatible, if code is using the missing or view analysis exceptions they may have runtime issues.