search

swan-cern / sparkmonitor

An extension for Jupyter Lab & Jupyter Notebook to monitor Apache Spark (pyspark) from notebooks
https://pypi.org/project/sparkmonitor/
Apache License 2.0
46 stars 9 forks source link

fix spark3 version #9

Closed rccern closed 2 years ago

rahul26goyal commented 2 years ago

@krishnan-r why was the version reverted?

krishnan-r commented 2 years ago

Hi @rahul26goyal,

The sparklistener jar bundled with sparkmonitor for Spark 3.2 had a conflict with the version of json4s packaged in the jar and the one provided in the spark distribution.

To fix that we shaded json4s in the jar in #6, this allows supporting all the spark 3.x versions with a single JAR file.

When doing that we built the JAR using spark 3.2 as a dependency. However this broke earlier versions because the JAR built for 3.2 was not backward compatible, whereas the jar built for 3.0.1 works fine with 3.2 (if I remember correctly).

If this does not work with 3.2, we can perhaps have a separate JAR built and packaged with 3.2, similar to what we do between 2.x and 3.x. But this involves some more work.