Closed Jirapong closed 5 years ago
Hi, The intended usage is:
val stageMetrics = ch.cern.sparkmeasure.StageMetrics(spark)
stageMetrics.begin()
...execute one or more Spark jobs...
stageMetrics.end()
stageMetrics.printReport()
See also https://github.com/LucaCanali/sparkMeasure/blob/master/docs/Scala_shell_and_notebooks.md Please check again if this works in your case.
Sorry that is a wrong syntax. I believe the error caused by metric is not produce any number like
val stageMetrics = ch.cern.sparkmeasure.StageMetrics(spark)
stageMetrics.begin()
spark.sql("SELECT 1=1")
stageMetrics.end()
stageMetrics.printReport()
scala> spark.sql("select * from PerfStageMetrics").show
+-----+--------+-------+----+--------------+--------------+-------------+--------+---------------+---------------+-----------------------+--------------------------+-----------------------+---------+----------+-----------------------+----------------+------------------+-------------------+-----------+---------+--------------+------------+--------------------+---------------------+-------------------------+-------------------------+--------------------------+----------------+-------------------+---------------------+
|jobId|jobGroup|stageId|name|submissionTime|completionTime|stageDuration|numTasks|executorRunTime|executorCpuTime|executorDeserializeTime|executorDeserializeCpuTime|resultSerializationTime|jvmGCTime|resultSize|numUpdatedBlockStatuses|diskBytesSpilled|memoryBytesSpilled|peakExecutionMemory|recordsRead|bytesRead|recordsWritten|bytesWritten|shuffleFetchWaitTime|shuffleTotalBytesRead|shuffleTotalBlocksFetched|shuffleLocalBlocksFetched|shuffleRemoteBlocksFetched|shuffleWriteTime|shuffleBytesWritten|shuffleRecordsWritten|
+-----+--------+-------+----+--------------+--------------+-------------+--------+---------------+---------------+-----------------------+--------------------------+-----------------------+---------+----------+-----------------------+----------------+------------------+-------------------+-----------+---------+--------------+------------+--------------------+---------------------+-------------------------+-------------------------+--------------------------+----------------+-------------------+---------------------+
+-----+--------+-------+----+--------------+--------------+-------------+--------+---------------+---------------+-----------------------+--------------------------+-----------------------+---------+----------+-----------------------+----------------+------------------+-------------------+-----------+---------+--------------+------------+--------------------+---------------------+-------------------------+-------------------------+--------------------------+----------------+-------------------+---------------------+
BTW, I'm working on PR to prevent this error.
PR has been merged.
I got an error after run following command
got following error