search

NVIDIA / spark-rapids

Spark RAPIDS plugin - accelerate Apache Spark with GPUs
https://nvidia.github.io/spark-rapids
Apache License 2.0
772 stars 227 forks source link

[BUG] Unit tests failed: [INTERVAL_ARITHMETIC_OVERFLOW] integer overflow. Use 'try_add' to tolerate overflow and return NULL instead #10498

Open NvTimLiu opened 5 months ago

NvTimLiu commented 5 months ago

Describe the bug

Unit tests failed: [INTERVAL_ARITHMETIC_OVERFLOW] integer overflow. Use 'try_add' to tolerate overflow and return NULL instead, details as below:

IntervalArithmeticSuite:
  - test year month interval arithmetic: Add
 24/02/24 13:04:17.049 Executor task launch worker for task 0.0 in stage 5.0 (TID 5) ERROR Executor: Exception in task 0.0 in stage 5.0 (TID 5)
 org.apache.spark.SparkArithmeticException: [INTERVAL_ARITHMETIC_OVERFLOW] integer overflow. Use 'try_add' to tolerate overflow and return NULL 
    at org.apache.spark.sql.errors.QueryExecutionErrors$.intervalArithmeticOverflowError(QueryExecutionErrors.scala:679) ~[spark-r:3.4.1]
    at org.apache.spark.sql.catalyst.util.IntervalMathUtils$.withOverflow(IntervalMathUtils.scala:43) ~[spark-catalyst_2.12-3.4.1.jar:3.4.1]
    at org.apache.spark.sql.catalyst.util.IntervalMathUtils$.addExact(IntervalMathUtils.scala:26) ~[spark-catalyst_2.12-3.4.1.jar:3.4.1]
    at org.apache.spark.sql.catalyst.util.IntervalMathUtils.addExact(IntervalMathUtils.scala) ~[spark-catalyst_2.12-3.4.1.jar:3.4.1]
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source) ~[?:?] - test year month interval arithmetic: Add overflow ,cpu
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) ~[spark-sql_2.12-3.4.1.jar:3.4.1]
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:760) ~[spark-sql_2.12-3.4.1.jar:3.4.1]
    at org.apache.spark.sql.execution.SparkPlan.$anonfun$getByteArrayRdd$1(SparkPlan.scala:388) ~[spark-sql_2.12-3.4.1.jar:3.4.1]
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:888) ~[spark-core_2.12-3.4.1.jar:3.4.1]
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:888) ~[spark-core_2.12-3.4.1.jar:3.4.1]
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) ~[spark-core_2.12-3.4.1.jar:3.4.1]
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364) ~[spark-core_2.12-3.4.1.jar:3.4.1]
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:328) ~[spark-core_2.12-3.4.1.jar:3.4.1]
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92) ~[spark-core_2.12-3.4.1.jar:3.4.1]
    at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161) ~[spark-core_2.12-3.4.1.jar:3.4.1]
    at org.apache.spark.scheduler.Task.run(Task.scala:139) ~[spark-core_2.12-3.4.1.jar:3.4.1]
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:554) ~[spark-core_2.12-3.4.1.jar:3.4.1]
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1529) ~[spark-core_2.12-3.4.1.jar:3.4.1]
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:557) ~[spark-core_2.12-3.4.1.jar:3.4.1]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_392]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_392]
    at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_392]
revans2 commented 5 months ago

What version of Spark was this running against? It looks like something in Spark itself changed and not our code.