Uncaught throwable from user code: scala.MatchError: (elapsedTime,null) (of class scala.Tuple2) (taskmetrics.scala:206)

[turtlemonvh]

I am seeing an occasional exception when using the .report method on a TaskMetrics to render the metrics data as a string.

The problem seems to be associated with this line: https://github.com/LucaCanali/sparkMeasure/blob/master/src/main/scala/ch/cern/sparkmeasure/taskmetrics.scala#L206

Here is the top of the stacktrace.

18/12/17 20:53:14 ERROR Uncaught throwable from user code: scala.MatchError: (elapsedTime,null) (of class scala.Tuple2)
    at ch.cern.sparkmeasure.TaskMetrics$$anonfun$report$1.apply(taskmetrics.scala:206)
    at ch.cern.sparkmeasure.TaskMetrics$$anonfun$report$1.apply(taskmetrics.scala:206)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
    at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
    at ch.cern.sparkmeasure.TaskMetrics.report(taskmetrics.scala:206)
    at com.ionic.helperfunctions.SparkMeasureHelpers$.save(SparkMeasure.scala:116)

It looks like this is probably likely caused by calling .report when there are no records in listenerTask.taskMetricsData (I'll try to confirm this).

If this is the case, there are a few options for fixing. The most obvious one is to change the match statement to something like:

      .map {
        case ((n: String, v: Long)) => Utils.prettyPrintValues(n, v)
        case ((n: String, null)) => n + " => null"
      }
     ).mkString("\n")

But there are, of course, other options.

[turtlemonvh]

I confirmed the mode of failure. Here is a simple script to reproduce the error.

import ch.cern.sparkmeasure.{ StageMetrics, TaskMetrics }

val taskMetrics = TaskMetrics(spark)
taskMetrics.begin
val report = taskMetrics.report

println("report", report)

This fails with the same error: scala.MatchError: (elapsedTime,null) (of class scala.Tuple2).

I am happy to submit a PR for the fix if you can confirm that fixing the match statement (as mentioned above) is ok with you.

[LucaCanali]

Thanks @turtlemonvh for reporting this. Please go ahead with the PR.

[turtlemonvh]

PR: https://github.com/LucaCanali/sparkMeasure/pull/21