search

sparkutils / quality

A Quality Spark DQ Library
https://sparkutils.github.io/quality/
Apache License 2.0
4 stars 2 forks source link

update_field with resolveWith do not mix #38

Open chris-twiner opened 1 year ago

chris-twiner commented 1 year ago

low prio as its possibly not needed to be fixed with later spark versions resolving improvements.

swappiing: 

  @Test // RuleEngineTest: 60
  def testSimpleProductionRules(): Unit = evalCodeGensNoResolve {

to evalCodeGens causes:

java.lang.ClassCastException: org.apache.spark.sql.qualityFunctions.NamedLambdaVariableCodeGen cannot be cast to org.apache.spark.sql.catalyst.expressions.NamedLambdaVariable
    at org.apache.spark.sql.qualityFunctions.FunN.$anonfun$elementVars$1(ReferenceFunctions.scala:232)
    at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at scala.collection.TraversableLike.map(TraversableLike.scala:238)
    at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
    at scala.collection.AbstractTraversable.map(Traversable.scala:108)
    at org.apache.spark.sql.qualityFunctions.FunN.elementVars$lzycompute(ReferenceFunctions.scala:232)
    at org.apache.spark.sql.qualityFunctions.FunN.elementVars(ReferenceFunctions.scala:232)
    at org.apache.spark.sql.qualityFunctions.FunN.eval(ReferenceFunctions.scala:236)
    at org.apache.spark.sql.catalyst.expressions.IsNull.eval(nullExpressions.scala:325)