search

qubole / sparklens

Qubole Sparklens tool for performance tuning Apache Spark
http://sparklens.qubole.com
Apache License 2.0
566 stars 138 forks source link

Release for Scala 2.12 #62

Open janekdb opened 3 years ago

janekdb commented 3 years ago

Hi,

Any plans to release for Scala 2.12?

Scala 2.13 and Spark 3 would be good to have support for.

Thanks, Janek

oleksiilopasov commented 3 years ago

Just change Scala and Spark versions in build.sbt and rebuild the package with sbt compile && sbt package Scala 2.12 and Spark 2.4.0 worked for me.

conrosebraugh commented 3 years ago

Does Qubole have any plans to release a build like that to spark-packages?

sergialonsaco commented 2 years ago

Hey, any update on this topic?

markballenger commented 2 years ago

Scala 2.12 support would be nice. For myself, i'm using GCP DataProc with an image of Apache Spark 3.1.3 and Scala 2.12.14. https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-release-2.0

Having sparklens as a supported listener solution in this environment with an emailed report would be really nice, because my organization tends to restrict FW rules aggressively that makes it very difficult to utilize many of the UI based diagnostic tools.

Just change Scala and Spark versions in build.sbt and rebuild the package with sbt compile && sbt package Scala 2.12 and Spark 2.4.0 worked for me. @oleksiilopasov thanks, but this did not work for me. Compilation failed, SparkHadoopUtil is not longer accessible, along with some other errors.

tonyye commented 4 months ago

Anyone tried building sparklens with Spark 3.5.1? First I had to build and publish sbt-spark-package 0.2.4 locally but then I get this error: [error] /Users/antonioye/code/sparklens/src/main/scala/com/qubole/sparklens/helper/HDFSConfigHelper.scala:28: object SparkHadoopUtil in package deploy cannot be accessed in package org.apache.spark.deploy