search

sparkutils / quality

A Quality Spark DQ Library
https://sparkutils.github.io/quality/
Apache License 2.0
4 stars 2 forks source link

13.3 DBR target needed #59

Closed chris-twiner closed 6 months ago

chris-twiner commented 6 months ago

java.lang.NoSuchMethodError: org.apache.spark.sql.types.AtomicType.ordering()Lscala/math/Ordering; java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.analysis.Analyzer.ResolveUserSpecifiedColumns()Lorg/apache/spark/sql/catalyst/analysis/Analyzer$ResolveUserSpecifiedColumns$;

on 13.1 build (shim based).

chris-twiner commented 6 months ago 
4) testSimpleProductionRulesReplaceDebug(com.sparkutils.qualityTests.RuleFolderTest)
java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.ScalaReflection$.isNativeType(Lorg/apache/spark/sql/types/DataType;)Z
    at org.apache.spark.sql.reflection.package$.isNativeType(package.scala:16)
    at frameless.TypedEncoder$$anon$17.toCatalyst(TypedEncoder.scala:343)
    at frameless.TypedEncoder$$anon$19.toCatalyst(TypedEncoder.scala:505)
    at frameless.RecordEncoder.$anonfun$toCatalyst$2(RecordEncoder.scala:155)
    at scala.collection.immutable.List.map(List.scala:297)
    at frameless.RecordEncoder.toCatalyst(RecordEncoder.scala:153)
    at frameless.TypedExpressionEncoder$.apply(TypedExpressionEncoder.scala:28)
    at com.sparkutils.quality.impl.EncodersImplicits.ruleFolderResultExpEnc(SparkHelpers.scala:107)
    at com.sparkutils.quality.impl.EncodersImplicits.ruleFolderResultExpEnc$(SparkHelpers.scala:107)
    at com.sparkutils.quality.implicits.package$.ruleFolderResultExpEnc(package.scala:8)
    at com.sparkutils.qualityTests.RuleFolderTest.$anonfun$doTestSimpleProductionRulesReplaceDebug$1(RuleFolderTest.scala:226)

won't run without frameless being shimmed, they've backported that as well.