search

LucaCanali / sparkMeasure

This is the development repository for sparkMeasure, a tool and library designed for efficient analysis and troubleshooting of Apache Spark jobs. It focuses on easing the collection and examination of Spark metrics, making it a practical choice for both developers and data engineers.
Apache License 2.0
690 stars 144 forks source link

NoSuchMethodError: org.apache.spark.scheduler.SparkListenerExecutorMetricsUpdate.executorUpdates() #48

Closed mlcoop closed 1 year ago

mlcoop commented 1 year ago

Trying to use sparkMeasure in scala spark application. Get this error:


java.lang.NoSuchMethodError: org.apache.spark.scheduler.SparkListenerExecutorMetricsUpdate.executorUpdates()Lscala/collection/Map;```
scalaVersion:2.12.13
sparkVersion: 2.4.6.0
sparkMeasure: 0.22
LucaCanali commented 1 year ago

Hi, For Spark 2.4.6 you would need to use sparkMeasure v 0.19 (latest version of sparkMEasure that works with Spark 2.4). sparkMeasure 0.22 only works with Spark 3.x.

mlcoop commented 1 year ago

is there difference in sparkMeasure implementation while using in cluster ?

mlcoop commented 1 year ago

@LucaCanali I am also using hadoop-> "3.2.0", is it compatible with 0.19 version ? because i started getting errors like java.lang.IllegalAccessError: class org.apache.hadoop.hdfs.web.HftpFileSystem cannot access its superinterface org.apache.hadoop.hdfs.web.TokenAspect$TokenManagementDelegator

mlcoop commented 1 year ago

Apparently it does, so i changed to 0.17 which seems to be working

LucaCanali commented 1 year ago

Good to hear this is working for you.