search

LucaCanali / sparkMeasure

This is the development repository for sparkMeasure, a tool and library designed for efficient analysis and troubleshooting of Apache Spark jobs. It focuses on easing the collection and examination of Spark metrics, making it a practical choice for both developers and data engineers.
Apache License 2.0
690 stars 144 forks source link

testSparkMeasureScala sbt problem #45

Closed DanielAronovich closed 1 year ago

DanielAronovich commented 1 year ago

Hi,

While trying to run the following example code: https://github.com/LucaCanali/sparkMeasure/tree/master/examples/testSparkMeasureScala

I get the following error when running sbt package.

What am I doing wrong?

C:\repos\sparkMeasure\examples\testSparkMeasureScala> sbt package [info] welcome to sbt 1.7.1 (Oracle Corporation Java 19.0.1) [info] loading global plugins from C:\Users\DanielAronovich\.sbt\1.0\plugins [info] loading project definition from C:\repos\sparkMeasure\examples\testSparkMeasureScala\project [info] loading settings for project testsparkmeasurescala from build.sbt ... [info] set current project to testSparkMeasureScala (in build file:/C:/repos/sparkMeasure/examples/testSparkMeasureScala/) [warn] [warn] Note: Unresolved dependencies path: [error] sbt.librarymanagement.ResolveException: Error downloading ch.cern.sparkmeasure:spark-measure_2.12:0.21 [error] Not found [error] Not found [error] not found: C:\Users\DanielAronovich\.ivy2\localch.cern.sparkmeasure\spark-measure_2.12\0.21\ivys\ivy.xml [error] not found: https://repo1.maven.org/maven2/ch/cern/sparkmeasure/spark-measure_2.12/0.21/spark-measure_2.12-0.21.pom [error] at lmcoursier.CoursierDependencyResolution.unresolvedWarningOrThrow(CoursierDependencyResolution.scala:345) [error] at lmcoursier.CoursierDependencyResolution.$anonfun$update$38(CoursierDependencyResolution.scala:314) [error] at scala.util.Either$LeftProjection.map(Either.scala:573) [error] at lmcoursier.CoursierDependencyResolution.update(CoursierDependencyResolution.scala:314) [error] at sbt.librarymanagement.DependencyResolution.update(DependencyResolution.scala:60) [error] at sbt.internal.LibraryManagement$.resolve$1(LibraryManagement.scala:59) [error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$12(LibraryManagement.scala:133) [error] at sbt.util.Tracked$.$anonfun$lastOutput$1(Tracked.scala:73) [error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$20(LibraryManagement.scala:146) [error] at scala.util.control.Exception$Catch.apply(Exception.scala:228) [error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$11(LibraryManagement.scala:146) [error] at sbt.internal.LibraryManagement$.$anonfun$cachedUpdate$11$adapted(LibraryManagement.scala:127) [error] at sbt.util.Tracked$.$anonfun$inputChangedW$1(Tracked.scala:219) [error] at sbt.internal.LibraryManagement$.cachedUpdate(LibraryManagement.scala:160) [error] at sbt.Classpaths$.$anonfun$updateTask0$1(Defaults.scala:3688) [error] at scala.Function1.$anonfun$compose$1(Function1.scala:49) [error] at sbt.internal.util.$tilde$greater.$anonfun$$u2219$1(TypeFunctions.scala:62) [error] at sbt.std.Transform$$anon$4.work(Transform.scala:68) [error] at sbt.Execute.$anonfun$submit$2(Execute.scala:282) [error] at sbt.internal.util.ErrorHandling$.wideConvert(ErrorHandling.scala:23) [error] at sbt.Execute.work(Execute.scala:291) [error] at sbt.Execute.$anonfun$submit$1(Execute.scala:282) [error] at sbt.ConcurrentRestrictions$$anon$4.$anonfun$submitValid$1(ConcurrentRestrictions.scala:265) [error] at sbt.CompletionService$$anon$2.call(CompletionService.scala:64) [error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) [error] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) [error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) [error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [error] at java.base/java.lang.Thread.run(Thread.java:1589) [error] (update) sbt.librarymanagement.ResolveException: Error downloading ch.cern.sparkmeasure:spark-measure_2.12:0.21 [error] Not found [error] Not found [error] not found: C:\Users\DanielAronovich\.ivy2\localch.cern.sparkmeasure\spark-measure_2.12\0.21\ivys\ivy.xml [error] not found: https://repo1.maven.org/maven2/ch/cern/sparkmeasure/spark-measure_2.12/0.21/spark-measure_2.12-0.21.pom [error] Total time: 4 s, completed Nov 15, 2022, 3:11:11 PM

LucaCanali commented 1 year ago

Hi @DanielAronovich thanks for reporting this issue. It is due to an issue with sparkMeasure 0.21 in maven central. The build should use v0.22 instead. I have just pushed the change https://github.com/LucaCanali/sparkMeasure/commit/d34975d42775354eba763d1f5fe7d62feda4af2b

Best, Luca

DanielAronovich commented 1 year ago

Thanks Luca, Much appreciated!