search

LucaCanali / sparkMeasure

This is the development repository for sparkMeasure, a tool and library designed for efficient analysis and troubleshooting of Apache Spark jobs. It focuses on easing the collection and examination of Spark metrics, making it a practical choice for both developers and data engineers.
Apache License 2.0
690 stars 144 forks source link

TypeError: 'JavaPackage' object is not callable #41

Closed rogerrogeryau closed 2 years ago

rogerrogeryau commented 2 years ago

self.stagemetrics = self.sc._jvm.ch.cern.sparkmeasure.StageMetrics(self.sparksession._jsparkSession) TypeError: 'JavaPackage' object is not callable issue is hit when I spark-submit the .py script to emr.

LucaCanali commented 2 years ago

Hi, thanks for reporting this. Can you please provide more context on when/where you encounter the error message?

Does the Python example provided with sparkmesure work for you? You can run it as: bin/spark-submit --packages ch.cern.sparkmeasure:spark-measure_2.12:0.21 <path_to_examples>/test_sparkmeasure_python.py

LucaCanali commented 2 years ago

Closing for lack of activity.

razajafri commented 1 year ago

This error appears when the user fails to set --packages as a spark flag