search

apache-spark-on-k8s / spark

Apache Spark enhanced with native Kubernetes scheduler back-end: NOTE this repository is being ARCHIVED as all new development for the kubernetes scheduler back-end is now on https://github.com/apache/spark/
https://spark.apache.org/
Apache License 2.0
612 stars 118 forks source link

Dependency failure on build #380

Closed luck02 closed 7 years ago

luck02 commented 7 years ago

I'm trying to build against branch-2.1-kubernetes due to discussion here: https://github.com/apache-spark-on-k8s/spark/issues/4

Getting a dependency failure:

[ERROR] Failed to execute goal on project spark-network-shuffle_2.11: Could not resolve dependencies for project org.apache.spark:spark-network-shuffle_2.11:jar:2.1.0-k8s-0.2.0-SNAPSHOT: Failure to find org.apache.spark:spark-network-common_2.11:jar:tests:2.1.0-k8s-0.2.0-SNAPSHOT in http://repository.apache.org/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots has elapsed or updates are forced -> [Help 1]

Full output:

garylucas@gary-lucas:~/dev/spark|branch-2.1-kubernetes
⇒  build/mvn compile -Pkubernetes -pl resource-managers/kubernetes/core -am -DskipTests
Using `mvn` from path: /usr/local/bin/mvn
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
[INFO] Scanning for projects...
[WARNING] The project org.apache.spark:spark-parent_2.11:pom:2.1.0-k8s-0.2.0-SNAPSHOT uses prerequisites which is only intended for maven-plugin projects but not for non maven-plugin projects. For such purposes you should use the maven-enforcer-plugin. See https://maven.apache.org/enforcer/enforcer-rules/requireMavenVersion.html
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] Spark Project Parent POM
[INFO] Spark Project Tags
[INFO] Spark Project Networking
[INFO] Spark Project Shuffle Streaming Service
[INFO] Spark Project Unsafe
[INFO] Spark Project Launcher
[INFO] Spark Project Core
[INFO] Spark Project Kubernetes
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Project Parent POM 2.1.0-k8s-0.2.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-versions) @ spark-parent_2.11 ---
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:add-source (eclipse-add-source) @ spark-parent_2.11 ---
[INFO] Add Source directory: /Users/garylucas/dev/spark/src/main/scala
[INFO] Add Test Source directory: /Users/garylucas/dev/spark/src/test/scala
[INFO]
[INFO] --- maven-dependency-plugin:2.10:build-classpath (default-cli) @ spark-parent_2.11 ---
[INFO] Dependencies classpath:
/Users/garylucas/.m2/repository/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ spark-parent_2.11 ---
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ spark-parent_2.11 ---
[INFO] No sources to compile
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Project Tags 2.1.0-k8s-0.2.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-versions) @ spark-tags_2.11 ---
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:add-source (eclipse-add-source) @ spark-tags_2.11 ---
[INFO] Add Source directory: /Users/garylucas/dev/spark/common/tags/src/main/scala
[INFO] Add Test Source directory: /Users/garylucas/dev/spark/common/tags/src/test/scala
[INFO]
[INFO] --- maven-dependency-plugin:2.10:build-classpath (default-cli) @ spark-tags_2.11 ---
[INFO] Dependencies classpath:
/Users/garylucas/.m2/repository/org/scala-lang/modules/scala-xml_2.11/1.0.2/scala-xml_2.11-1.0.2.jar:/Users/garylucas/.m2/repository/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar:/Users/garylucas/.m2/repository/org/scalatest/scalatest_2.11/2.2.6/scalatest_2.11-2.2.6.jar:/Users/garylucas/.m2/repository/org/scala-lang/scala-library/2.11.8/scala-library-2.11.8.jar:/Users/garylucas/.m2/repository/org/scala-lang/scala-reflect/2.11.8/scala-reflect-2.11.8.jar
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ spark-tags_2.11 ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ spark-tags_2.11 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/garylucas/dev/spark/common/tags/src/main/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ spark-tags_2.11 ---
[INFO] Using zinc server for incremental compilation
[warn] Pruning sources from previous analysis, due to incompatible CompileSetup.
[info] Compiling 2 Scala sources and 9 Java sources to /Users/garylucas/dev/spark/common/tags/target/scala-2.11/classes...
[warn] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[warn] 1 warning
[info] Compile success at Jul 18, 2017 9:56:36 PM [1.072s]
[INFO]
[INFO] --- maven-compiler-plugin:3.5.1:compile (default-compile) @ spark-tags_2.11 ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 9 source files to /Users/garylucas/dev/spark/common/tags/target/scala-2.11/classes
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Project Networking 2.1.0-k8s-0.2.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-versions) @ spark-network-common_2.11 ---
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:add-source (eclipse-add-source) @ spark-network-common_2.11 ---
[INFO] Add Source directory: /Users/garylucas/dev/spark/common/network-common/src/main/scala
[INFO] Add Test Source directory: /Users/garylucas/dev/spark/common/network-common/src/test/scala
[INFO]
[INFO] --- maven-dependency-plugin:2.10:build-classpath (default-cli) @ spark-network-common_2.11 ---
[INFO] Dependencies classpath:
/Users/garylucas/.m2/repository/io/netty/netty-all/4.0.42.Final/netty-all-4.0.42.Final.jar:/Users/garylucas/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.6.5/jackson-core-2.6.5.jar:/Users/garylucas/.m2/repository/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar:/Users/garylucas/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.6.5/jackson-annotations-2.6.5.jar:/Users/garylucas/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/Users/garylucas/dev/spark/common/tags/target/scala-2.11/classes:/Users/garylucas/.m2/repository/org/fusesource/leveldbjni/leveldbjni-all/1.8/leveldbjni-all-1.8.jar:/Users/garylucas/.m2/repository/org/apache/commons/commons-lang3/3.5/commons-lang3-3.5.jar:/Users/garylucas/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.6.5/jackson-databind-2.6.5.jar:/Users/garylucas/.m2/repository/com/google/guava/guava/14.0.1/guava-14.0.1.jar
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ spark-network-common_2.11 ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ spark-network-common_2.11 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /Users/garylucas/dev/spark/common/network-common/src/main/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ spark-network-common_2.11 ---
[INFO] Using zinc server for incremental compilation
[warn] Pruning sources from previous analysis, due to incompatible CompileSetup.
[info] Compiling 65 Java sources to /Users/garylucas/dev/spark/common/network-common/target/scala-2.11/classes...
[warn] warning: [options] bootstrap class path not set in conjunction with -source 1.7
[warn] 1 warning
[info] Compile success at Jul 18, 2017 9:56:38 PM [1.816s]
[INFO]
[INFO] --- maven-compiler-plugin:3.5.1:compile (default-compile) @ spark-network-common_2.11 ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 65 source files to /Users/garylucas/dev/spark/common/network-common/target/scala-2.11/classes
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Project Shuffle Streaming Service 2.1.0-k8s-0.2.0-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [  1.426 s]
[INFO] Spark Project Tags ................................. SUCCESS [  2.087 s]
[INFO] Spark Project Networking ........................... SUCCESS [  3.345 s]
[INFO] Spark Project Shuffle Streaming Service ............ FAILURE [  0.019 s]
[INFO] Spark Project Unsafe ............................... SKIPPED
[INFO] Spark Project Launcher ............................. SKIPPED
[INFO] Spark Project Core ................................. SKIPPED
[INFO] Spark Project Kubernetes ........................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 7.307 s
[INFO] Finished at: 2017-07-18T21:56:40-07:00
[INFO] Final Memory: 38M/412M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project spark-network-shuffle_2.11: Could not resolve dependencies for project org.apache.spark:spark-network-shuffle_2.11:jar:2.1.0-k8s-0.2.0-SNAPSHOT: Failure to find org.apache.spark:spark-network-common_2.11:jar:tests:2.1.0-k8s-0.2.0-SNAPSHOT in http://repository.apache.org/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of apache.snapshots has elapsed or updates are forced -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :spark-network-shuffle_2.11
garylucas@gary-lucas:~/dev/spark|branch-2.1-kubernetes

Based on output I'm thinking it's a transient problem or someone's definitely run into it previously.

Thanks!

liyinan926 commented 7 years ago

Try running the following commands:

build/mvn install -Pkubernetes -pl resource-managers/kubernetes/core -am -DskipTests
build/mvn compile -T 4C -Pkubernetes -pl resource-managers/kubernetes/core -am -DskipTests
luck02 commented 7 years ago

Interesting, well that got me past the initial problem. I'll do a PR to update the docs in the near future. Thanks!

mccheah commented 7 years ago

I don't think mvn install should be required here - I would have expected the -am flag to allow the reactor to detect upstream dependents accordingly.