search

databricks / spark-deep-learning

Deep Learning Pipelines for Apache Spark
https://databricks.github.io/spark-deep-learning
Apache License 2.0
1.99k stars 494 forks source link

build/sbt assembly error #129

Open Rizwanabro opened 6 years ago

Rizwanabro commented 6 years ago

Hello everyone, i am facing this issue while running build/sbt command can anybody guide me what is error about and its solution please :)

build/sbt assembly Using /usr/lib/jvm/java-8-oracle as default JAVA_HOME. Note, this will be overridden by -java-home if it is set. Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0 [warn] Executing in batch mode. [warn] For better performance, hit [ENTER] to switch to interactive mode, or [warn] consider launching sbt without any commands, or explicitly passing 'shell' [info] Loading project definition from /home/rizwan/keras/spark-deep-learning-master/project [info] Set current project to spark-deep-learning-master (in build file:/home/rizwan/keras/spark-deep-learning-master/) Listing /home/rizwan/keras/spark-deep-learning-master/python ... Listing /home/rizwan/keras/spark-deep-learning-master/python/.pylint ... Listing /home/rizwan/keras/spark-deep-learning-master/python/docs ... Listing /home/rizwan/keras/spark-deep-learning-master/python/docs/_templates ... Listing /home/rizwan/keras/spark-deep-learning-master/python/docs/static ... Listing /home/rizwan/keras/spark-deep-learning-master/python/model_gen ... Listing /home/rizwan/keras/spark-deep-learning-master/python/sparkdl ... Listing /home/rizwan/keras/spark-deep-learning-master/python/sparkdl/pycache ... Listing /home/rizwan/keras/spark-deep-learning-master/python/sparkdl/estimators ... Listing /home/rizwan/keras/spark-deep-learning-master/python/sparkdl/graph ... Listing /home/rizwan/keras/spark-deep-learning-master/python/sparkdl/graph/pycache ... Listing /home/rizwan/keras/spark-deep-learning-master/python/sparkdl/image ... Listing /home/rizwan/keras/spark-deep-learning-master/python/sparkdl/param ... Listing /home/rizwan/keras/spark-deep-learning-master/python/sparkdl/param/pycache ... Listing /home/rizwan/keras/spark-deep-learning-master/python/sparkdl/transformers ... Listing /home/rizwan/keras/spark-deep-learning-master/python/sparkdl/transformers/pycache ... Listing /home/rizwan/keras/spark-deep-learning-master/python/sparkdl/udf ... Listing /home/rizwan/keras/spark-deep-learning-master/python/sparkdl/utils ... Listing /home/rizwan/keras/spark-deep-learning-master/python/sparkdl/utils/pycache ... Listing /home/rizwan/keras/spark-deep-learning-master/python/tests ... Listing /home/rizwan/keras/spark-deep-learning-master/python/tests/pycache ... Listing /home/rizwan/keras/spark-deep-learning-master/python/tests/estimators ... Listing /home/rizwan/keras/spark-deep-learning-master/python/tests/graph ... Listing /home/rizwan/keras/spark-deep-learning-master/python/tests/image ... Listing /home/rizwan/keras/spark-deep-learning-master/python/tests/param ... Listing /home/rizwan/keras/spark-deep-learning-master/python/tests/resources ... Listing /home/rizwan/keras/spark-deep-learning-master/python/tests/resources/images ... Listing /home/rizwan/keras/spark-deep-learning-master/python/tests/resources/images/1_channels ... Listing /home/rizwan/keras/spark-deep-learning-master/python/tests/resources/images/3_channels ... Listing /home/rizwan/keras/spark-deep-learning-master/python/tests/resources/images/4_channels ... Listing /home/rizwan/keras/spark-deep-learning-master/python/tests/transformers ... Listing /home/rizwan/keras/spark-deep-learning-master/python/tests/udf ... Listing /home/rizwan/keras/spark-deep-learning-master/python/tests/utils ... [info] Updating {file:/home/rizwan/keras/spark-deep-learning-master/}spark-deep-learning-master... [info] Resolving org.apache.directory.server#apacheds-kerberos-codec;2.0.0-M15 .[info] Resolving org.glassfish.jersey.containers#jersey-container-servlet;2.22.2[info] Resolving org.glassfish.jersey.containers#jersey-container-servlet-core;2[info] Resolving com.fasterxml.jackson.module#jackson-module-scala_2.11;2.6.7.1 [info] Resolving jline#jline;2.12.1 ... [warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] :: UNRESOLVED DEPENDENCIES :: [warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] :: org.apache.commons#commons-proxy;1.0: org.apache.commons#commons-parent;7!commons-parent.pom(pom.original) origin location must be absolute: file:/home/rizwan/.m2/repository/org/apache/commons/commons-parent/7/commons-parent-7.pom [warn] :: org.slf4j#slf4j-api;1.7.7: org.slf4j#slf4j-api;1.7.7!slf4j-api.pom(pom.original) origin location must be absolute: file:/home/rizwan/.m2/repository/org/slf4j/slf4j-api/1.7.7/slf4j-api-1.7.7.pom [warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] [warn] Note: Unresolved dependencies path: [warn] org.apache.commons:commons-proxy:1.0 [warn] +- databricks:tensorframes:0.3.0-s_2.11 ((sbtsparkpackage.SparkPackagePlugin) SparkPackagePlugin.scala#L307) [warn] +- default:spark-deep-learning-master_2.11:1.0.0-spark2.3 [warn] org.slf4j:slf4j-api:1.7.7 [warn] +- com.typesafe.scala-logging:scala-logging-slf4j_2.11:2.1.2 (/home/rizwan/keras/spark-deep-learning-master/build.sbt#L43-52) [warn] +- databricks:tensorframes:0.3.0-s_2.11 ((sbtsparkpackage.SparkPackagePlugin) SparkPackagePlugin.scala#L307) [warn] +- default:spark-deep-learning-master_2.11:1.0.0-spark2.3 sbt.ResolveException: unresolved dependency: org.apache.commons#commons-proxy;1.0: org.apache.commons#commons-parent;7!commons-parent.pom(pom.original) origin location must be absolute: file:/home/rizwan/.m2/repository/org/apache/commons/commons-parent/7/commons-parent-7.pom unresolved dependency: org.slf4j#slf4j-api;1.7.7: org.slf4j#slf4j-api;1.7.7!slf4j-api.pom(pom.original) origin location must be absolute: file:/home/rizwan/.m2/repository/org/slf4j/slf4j-api/1.7.7/slf4j-api-1.7.7.pom at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:313) at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:191) at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:168) at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:156) at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:156) at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:133) at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:57) at sbt.IvySbt$$anon$4.call(Ivy.scala:65) at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93) at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78) at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97) at xsbt.boot.Using$.withResource(Using.scala:10) at xsbt.boot.Using$.apply(Using.scala:9) at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58) at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48) at xsbt.boot.Locks$.apply0(Locks.scala:31) at xsbt.boot.Locks$.apply(Locks.scala:28) at sbt.IvySbt.withDefaultLogger(Ivy.scala:65) at sbt.IvySbt.withIvy(Ivy.scala:128) at sbt.IvySbt.withIvy(Ivy.scala:125) at sbt.IvySbt$Module.withModule(Ivy.scala:156) at sbt.IvyActions$.updateEither(IvyActions.scala:168) at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1488) at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1484) at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$121.apply(Defaults.scala:1519) at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$121.apply(Defaults.scala:1517) at sbt.Tracked$$anonfun$lastOutput$1.apply(Tracked.scala:37) at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1522) at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1516) at sbt.Tracked$$anonfun$inputChanged$1.apply(Tracked.scala:60) at sbt.Classpaths$.cachedUpdate(Defaults.scala:1539) at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1466) at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1418) at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47) at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40) at sbt.std.Transform$$anon$4.work(System.scala:63) at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228) at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:228) at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17) at sbt.Execute.work(Execute.scala:237) at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228) at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:228) at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159) at sbt.CompletionService$$anon$2.call(CompletionService.scala:28) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.commons#commons-proxy;1.0: org.apache.commons#commons-parent;7!commons-parent.pom(pom.original) origin location must be absolute: file:/home/rizwan/.m2/repository/org/apache/commons/commons-parent/7/commons-parent-7.pom [error] unresolved dependency: org.slf4j#slf4j-api;1.7.7: org.slf4j#slf4j-api;1.7.7!slf4j-api.pom(pom.original) origin location must be absolute: file:/home/rizwan/.m2/repository/org/slf4j/slf4j-api/1.7.7/slf4j-api-1.7.7.pom [error] Total time: 20 s, completed May 7, 2018 2:10:45 AM

spark-water commented 4 years ago

I ran into the same problem. Here is how I fixed it. Hopefully it would help others

  1. remove .ivy2/cashe rm -rf ~/.ivy2/cache

  2. redo git clone

  3. now go into project root and run build/sbt assembly

you should have a build success (at least for me). Good luck!