search

NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS
Apache License 2.0
50 stars 37 forks source link

[BUG] MatchError in getDataSourceInfo #1042

Closed amahussein closed 4 months ago

amahussein commented 4 months ago

Describe the bug

The Profiler V24.02.2 throws an error while processing the dataSourceInfo

24/05/24 15:55:14 WARN Profiler: Exception occurred processing file: application_11111111111_0001_1
scala.MatchError: scan time total (min, med, max) (of class java.lang.String)
        at com.nvidia.spark.rapids.tool.profiling.CollectInformation.$anonfun$getDataSourceInfo$2(CollectInformation.scala:108)
        at com.nvidia.spark.rapids.tool.profiling.CollectInformation.$anonfun$getDataSourceInfo$2$adapted(CollectInformation.scala:108)
        at scala.collection.immutable.List.map(List.scala:293)
        at com.nvidia.spark.rapids.tool.profiling.CollectInformation.getIoMetrics$1(CollectInformation.scala:108)
        at com.nvidia.spark.rapids.tool.profiling.CollectInformation.$anonfun$getDataSourceInfo$6(CollectInformation.scala:129)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
        at scala.collection.TraversableLike.map(TraversableLike.scala:286)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:279)
        at scala.collection.AbstractTraversable.map(Traversable.scala:108)
        at com.nvidia.spark.rapids.tool.profiling.CollectInformation.$anonfun$getDataSourceInfo$3(CollectInformation.scala:125)
        at scala.collection.immutable.List.flatMap(List.scala:366)
        at com.nvidia.spark.rapids.tool.profiling.CollectInformation.getDataSourceInfo(CollectInformation.scala:117)
        at com.nvidia.spark.rapids.tool.profiling.Profiler.com$nvidia$spark$rapids$tool$profiling$Profiler$$processApps(Profiler.scala:340)
        at com.nvidia.spark.rapids.tool.profiling.Profiler$ProfileProcessThread$1.run(Profiler.scala:269)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
amahussein commented 4 months ago

This looks to be caused by and old spark version eventlog 2.3.4. We may need to figure out how to handle the metrics for those old spark eventlogs.