search

NVIDIA / spark-rapids

Spark RAPIDS plugin - accelerate Apache Spark with GPUs
https://nvidia.github.io/spark-rapids
Apache License 2.0
823 stars 236 forks source link

[BUG] NullPointerException in GpuRand #11646

Closed firestarman closed 4 weeks ago

firestarman commented 1 month ago

We met a NPE in GpuRand which failed some customer queries. The full stack is as below

Caused by: java.lang.NullPointerException 
        at org.apache.spark.sql.rapids.catalyst.expressions.GpuRand.columnarEval(GpuRandomExpressions.scala:80) 
        at com.nvidia.spark.rapids.GpuExpression.columnarEvalAny(GpuExpressions.scala:190) 
        at com.nvidia.spark.rapids.GpuExpression.columnarEvalAny$(GpuExpressions.scala:190) 
        at org.apache.spark.sql.rapids.catalyst.expressions.GpuRand.columnarEvalAny(GpuRandomExpressions.scala:34) 
        at com.nvidia.spark.rapids.RapidsPluginImplicits$ReallyAGpuExpression.columnarEvalAny(implicits.scala:38) 
        at com.nvidia.spark.rapids.GpuBinaryExpression.columnarEval(GpuExpressions.scala:356) 
        at com.nvidia.spark.rapids.GpuBinaryExpression.columnarEval$(GpuExpressions.scala:355) 
        at com.nvidia.spark.rapids.CudfBinaryOperator.columnarEval(GpuExpressions.scala:463) 
        at com.nvidia.spark.rapids.RapidsPluginImplicits$ReallyAGpuExpression.columnarEval(implicits.scala:34) 
        at com.nvidia.spark.rapids.GpuUnaryExpression.columnarEval(GpuExpressions.scala:306) 
        at com.nvidia.spark.rapids.RapidsPluginImplicits$ReallyAGpuExpression.columnarEval(implicits.scala:34) 
        at com.nvidia.spark.rapids.GpuAlias.columnarEval(namedExpressions.scala:110) 
        at com.nvidia.spark.rapids.RapidsPluginImplicits$ReallyAGpuExpression.columnarEval(implicits.scala:34) 
        at com.nvidia.spark.rapids.GpuProjectExec$.$anonfun$project$1(basicPhysicalOperators.scala:112) 
        at com.nvidia.spark.rapids.RapidsPluginImplicits$MapsSafely.$anonfun$safeMap$1(implicits.scala:220) 
        at com.nvidia.spark.rapids.RapidsPluginImplicits$MapsSafely.$anonfun$safeMap$1$adapted(implicits.scala:217) 
        at scala.collection.immutable.List.foreach(List.scala:431) 
        at com.nvidia.spark.rapids.RapidsPluginImplicits$MapsSafely.safeMap(implicits.scala:217) 
        at com.nvidia.spark.rapids.RapidsPluginImplicits$AutoCloseableProducingSeq.safeMap(implicits.scala:252) 
        at com.nvidia.spark.rapids.GpuProjectExec$.project(basicPhysicalOperators.scala:112) 
        at com.nvidia.spark.rapids.GpuProjectExec$.projectAndClose(basicPhysicalOperators.scala:79) 
        at com.nvidia.spark.rapids.GpuAggFinalPassIterator$.reorderFinalBatch(GpuAggregateExec.scala:830) 
        at com.nvidia.spark.rapids.GpuAggFinalPassIterator$.$anonfun$makeIter$2(GpuAggregateExec.scala:848) 
        at com.nvidia.spark.rapids.Arm$.withResource(Arm.scala:30) 
        at com.nvidia.spark.rapids.GpuAggFinalPassIterator$.$anonfun$makeIter$1(GpuAggregateExec.scala:844) 
        at scala.collection.Iterator$$anon$10.next(Iterator.scala:461) 
        at com.nvidia.spark.rapids.DynamicGpuPartialAggregateIterator.$anonfun$next$16(GpuAggregateExec.scala:2262)