kwai / blaze

Blazing-fast query execution engine speaks Apache Spark language and has Arrow-DataFusion at its core.
Apache License 2.0
1.3k stars 121 forks source link

Hashaggregate convert job throw ScalaReflectionException #636

Closed leizhang5s closed 4 weeks ago

leizhang5s commented 4 weeks ago

Describe the bug Hashaggregate convert job throw the following exception: 24/10/28 11:39:44 WARN BlazeConverters: Error converting exec: HashAggregateExec: expected a member of class HashAggregateExec, you provided value org.apache.spark.sql.execution.aggregate.HashAggregateExec.requiredChildDistributionExpressions scala.ScalaReflectionException: expected a member of class HashAggregateExec, you provided value org.apache.spark.sql.execution.aggregate.HashAggregateExec.requiredChildDistributionExpressions at scala.reflect.runtime.JavaMirrors$JavaMirror.scala$reflect$runtime$JavaMirrors$JavaMirror$$abort(JavaMirrors.scala:155) at scala.reflect.runtime.JavaMirrors$JavaMirror.ErrorNotMember(JavaMirrors.scala:161) at scala.reflect.runtime.JavaMirrors$JavaMirror.$anonfun$checkMemberOf$1(JavaMirrors.scala:256) at scala.reflect.runtime.JavaMirrors$JavaMirror.scala$reflect$runtime$JavaMirrors$JavaMirror$$checkMemberOf(JavaMirrors.scala:246) at scala.reflect.runtime.JavaMirrors$JavaMirror$JavaInstanceMirror.reflectField(JavaMirrors.scala:278) at scala.reflect.runtime.JavaMirrors$JavaMirror$JavaInstanceMirror.reflectField(JavaMirrors.scala:275) at org.apache.spark.sql.blaze.BlazeConverters$.$anonfun$convertHashAggregateExec$1(BlazeConverters.scala:616) at scala.collection.immutable.List.map(List.scala:293) at org.apache.spark.sql.blaze.BlazeConverters$.convertHashAggregateExec(BlazeConverters.scala:611) at org.apache.spark.sql.blaze.BlazeConverters$.$anonfun$convertSparkPlan$15(BlazeConverters.scala:177) at org.apache.spark.sql.blaze.BlazeConverters$.tryConvert(BlazeConverters.scala:243) at org.apache.spark.sql.blaze.BlazeConverters$.convertSparkPlan(BlazeConverters.scala:177) at org.apache.spark.sql.blaze.BlazeConvertStrategy$.$anonfun$apply$3(BlazeConvertStrategy.scala:66) at org.apache.spark.sql.blaze.BlazeConvertStrategy$.$anonfun$apply$3$adapted(BlazeConvertStrategy.scala:62) at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:263)

To Reproduce Steps to reproduce the behavior:

  1. run sql like select count(distinct upper(xxx)) from xxx using spark3.2.1 with blaze enabled
  2. See error above

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

Smartphone (please complete the following information):

Additional context Add any other context about the problem here.