printReport error on the complex query

[Jirapong]

I got an error after run following command

val stageMetrics = ch.cern.sparkmeasure.StageMetrics(spark) 

stageMetrics.begin()

spark.sql("SELECT 1=1")

stageMetrics.end()
stageMetrics.printReport()

got following error

at ch.cern.sparkmeasure.StageMetrics$$anonfun$report$1.apply(stagemetrics.scala:184)
    at ch.cern.sparkmeasure.StageMetrics$$anonfun$report$1.apply(stagemetrics.scala:184)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
    at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
    at ch.cern.sparkmeasure.StageMetrics.report(stagemetrics.scala:184)
    at ch.cern.sparkmeasure.StageMetrics.printReport(stagemetrics.scala:193)
    at linea8b963cf2ca141a2ac1e991add9914c629.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-652119476452024:3)
    at linea8b963cf2ca141a2ac1e991add9914c629.$read$$iw$$iw$$iw$$iw$$iw.<init>(command-652119476452024:48)
    at linea8b963cf2ca141a2ac1e991add9914c629.$read$$iw$$iw$$iw$$iw.<init>(command-652119476452024:50)
    at linea8b963cf2ca141a2ac1e991add9914c629.$read$$iw$$iw$$iw.<init>(command-652119476452024:52)
    at linea8b963cf2ca141a2ac1e991add9914c629.$read$$iw$$iw.<init>(command-652119476452024:54)
    at linea8b963cf2ca141a2ac1e991add9914c629.$read$$iw.<init>(command-652119476452024:56)
    at linea8b963cf2ca141a2ac1e991add9914c629.$read.<init>(command-652119476452024:58)
    at linea8b963cf2ca141a2ac1e991add9914c629.$read$.<init>(command-652119476452024:62)
    at linea8b963cf2ca141a2ac1e991add9914c629.$read$.<clinit>(command-652119476452024)
    at linea8b963cf2ca141a2ac1e991add9914c629.$eval$.$print$lzycompute(<notebook>:7)
    at linea8b963cf2ca141a2ac1e991add9914c629.$eval$.$print(<notebook>:6)

[LucaCanali]

Hi, The intended usage is:

val stageMetrics = ch.cern.sparkmeasure.StageMetrics(spark) 
stageMetrics.begin()

...execute one or more Spark jobs...

stageMetrics.end()
stageMetrics.printReport()

See also https://github.com/LucaCanali/sparkMeasure/blob/master/docs/Scala_shell_and_notebooks.md Please check again if this works in your case.

[Jirapong]

Sorry that is a wrong syntax. I believe the error caused by metric is not produce any number like

val stageMetrics = ch.cern.sparkmeasure.StageMetrics(spark) 

stageMetrics.begin()
spark.sql("SELECT 1=1")
stageMetrics.end()
stageMetrics.printReport()

scala> spark.sql("select * from PerfStageMetrics").show
+-----+--------+-------+----+--------------+--------------+-------------+--------+---------------+---------------+-----------------------+--------------------------+-----------------------+---------+----------+-----------------------+----------------+------------------+-------------------+-----------+---------+--------------+------------+--------------------+---------------------+-------------------------+-------------------------+--------------------------+----------------+-------------------+---------------------+
|jobId|jobGroup|stageId|name|submissionTime|completionTime|stageDuration|numTasks|executorRunTime|executorCpuTime|executorDeserializeTime|executorDeserializeCpuTime|resultSerializationTime|jvmGCTime|resultSize|numUpdatedBlockStatuses|diskBytesSpilled|memoryBytesSpilled|peakExecutionMemory|recordsRead|bytesRead|recordsWritten|bytesWritten|shuffleFetchWaitTime|shuffleTotalBytesRead|shuffleTotalBlocksFetched|shuffleLocalBlocksFetched|shuffleRemoteBlocksFetched|shuffleWriteTime|shuffleBytesWritten|shuffleRecordsWritten|
+-----+--------+-------+----+--------------+--------------+-------------+--------+---------------+---------------+-----------------------+--------------------------+-----------------------+---------+----------+-----------------------+----------------+------------------+-------------------+-----------+---------+--------------+------------+--------------------+---------------------+-------------------------+-------------------------+--------------------------+----------------+-------------------+---------------------+
+-----+--------+-------+----+--------------+--------------+-------------+--------+---------------+---------------+-----------------------+--------------------------+-----------------------+---------+----------+-----------------------+----------------+------------------+-------------------+-----------+---------+--------------+------------+--------------------+---------------------+-------------------------+-------------------------+--------------------------+----------------+-------------------+---------------------+

BTW, I'm working on PR to prevent this error.

[LucaCanali]

PR has been merged.