yahoo / CaffeOnSpark

Distributed deep learning on Hadoop and Spark clusters.
Apache License 2.0
1.27k stars 358 forks source link

build caffe-grid failed for many times with spark1.6.1 during downloading from maven ,should I change any profile or it just because of the internet problem? #236

Closed dawuchen closed 7 years ago

dawuchen commented 7 years ago

[INFO] Downloading: http://maven.twttr.com/org/scala-lang/scala-compiler/2.10.0/scala-compiler-2.10.0.jar [INFO] Downloading: http://maven.twttr.com/org/scalanlp/breeze_2.10/0.11.2/breeze_2.10-0.11.2.jar [INFO] Downloading: http://maven.twttr.com/org/spire-math/spire_2.10/0.7.4/spire_2.10-0.7.4.jar Mar 09, 2017 11:23:28 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://maven.twttr.com:80: Network is unreachable Mar 09, 2017 11:23:28 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://maven.twttr.com:80: Network is unreachable Mar 09, 2017 11:23:28 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://maven.twttr.com:80: Network is unreachable Mar 09, 2017 11:23:28 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute INFO: Retrying request to {}->http://maven.twttr.com:80 Mar 09, 2017 11:23:28 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute INFO: Retrying request to {}->http://maven.twttr.com:80 Mar 09, 2017 11:23:28 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute INFO: Retrying request to {}->http://maven.twttr.com:80 Mar 09, 2017 11:25:35 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://maven.twttr.com:80: Network is unreachable Mar 09, 2017 11:25:35 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://maven.twttr.com:80: Network is unreachable Mar 09, 2017 11:25:35 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://maven.twttr.com:80: Network is unreachable Mar 09, 2017 11:25:35 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute INFO: Retrying request to {}->http://maven.twttr.com:80 Mar 09, 2017 11:25:35 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute INFO: Retrying request to {}->http://maven.twttr.com:80 Mar 09, 2017 11:25:35 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute INFO: Retrying request to {}->http://maven.twttr.com:80 Mar 09, 2017 11:27:43 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://maven.twttr.com:80: Network is unreachable Mar 09, 2017 11:27:43 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://maven.twttr.com:80: Network is unreachable Mar 09, 2017 11:27:43 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://maven.twttr.com:80: Network is unreachable Mar 09, 2017 11:27:43 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute INFO: Retrying request to {}->http://maven.twttr.com:80 Mar 09, 2017 11:27:43 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute INFO: Retrying request to {}->http://maven.twttr.com:80 Mar 09, 2017 11:27:43 PM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute INFO: Retrying request to {}->http://maven.twttr.com:80 [INFO] Downloading: https://oss.sonatype.org/content/repositories/snapshots/org/scala-lang/scala-compiler/2.10.0/scala-compiler-2.10.0.jar [INFO] Downloading: https://oss.sonatype.org/content/repositories/releases/org/scala-lang/scala-compiler/2.10.0/scala-compiler-2.10.0.jar [INFO] Downloading: https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-hdfs/2.5.0/hadoop-hdfs-2.5.0.jar [INFO] Downloading: https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-yarn-common/2.5.0/hadoop-yarn-common-2.5.0.jar [INFO] Downloading: https://repository.apache.org/content/repositories/snapshots/com/google/guava/guava/11.0.2/guava-11.0.2.jar [INFO] Downloading: https://repository.apache.org/content/repositories/snapshots/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar [INFO] Downloading: https://repository.apache.org/content/repositories/snapshots/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar [INFO] Downloading: https://repository.apache.org/content/repositories/snapshots/com/sun/jersey/jersey-json/1.9/jersey-json-1.9.jar [INFO] Downloading: https://repository.apache.org/content/repositories/snapshots/tomcat/jasper-compiler/5.5.23/jasper-compiler-5.5.23.jar [INFO] Downloading: https://repository.apache.org/content/repositories/snapshots/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar [INFO] Downloading: https://repository.apache.org/content/repositories/snapshots/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar [INFO] Downloading: https://repository.apache.org/content/repositories/snapshots/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar [INFO] Downloading: https://repository.apache.org/content/repositories/snapshots/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar [INFO] Downloading: http://repository.jboss.org/nexus/content/groups/public/org/apache/hadoop/hadoop-hdfs/2.5.0/hadoop-hdfs-2.5.0.jar [INFO] Downloading: http://repository.jboss.org/nexus/content/groups/public/org/apache/hadoop/hadoop-yarn-common/2.5.0/hadoop-yarn-common-2.5.0.jar [INFO] Downloading: http://repository.jboss.org/nexus/content/groups/public/com/google/guava/guava/11.0.2/guava-11.0.2.jar [INFO] Downloading: http://repository.jboss.org/nexus/content/groups/public/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar [INFO] Downloading: http://repository.jboss.org/nexus/content/groups/public/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar [INFO] Downloading: http://repository.jboss.org/nexus/content/groups/public/com/sun/jersey/jersey-json/1.9/jersey-json-1.9.jar [INFO] Downloading: http://repository.jboss.org/nexus/content/groups/public/tomcat/jasper-compiler/5.5.23/jasper-compiler-5.5.23.jar [INFO] Downloading: http://repository.jboss.org/nexus/content/groups/public/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar [INFO] Downloading: http://repository.jboss.org/nexus/content/groups/public/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar [INFO] Downloading: http://repository.jboss.org/nexus/content/groups/public/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar [INFO] Downloading: http://repository.jboss.org/nexus/content/groups/public/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] caffe .............................................. SUCCESS [ 0.002 s] [INFO] caffe-distri ....................................... SUCCESS [ 43.792 s] [INFO] caffe-grid ......................................... FAILURE [ 05:43 h] [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 05:44 h [INFO] Finished at: 2017-03-10T00:09:59+08:00 [INFO] Final Memory: 45M/1217M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal on project caffe-grid: Could not resolve dependencies for project com.yahoo.ml:caffe-grid:jar:0.1-SNAPSHOT: The following artifacts could not be resolved: org.apache.spark:spark-core_2.10:jar:1.6.0, org.scala-lang:scala-compiler:jar:2.10.0, org.scalanlp:breeze_2.10:jar:0.11.2, org.spire-math:spire_2.10:jar:0.7.4, org.apache.hadoop:hadoop-hdfs:jar:2.5.0, org.apache.hadoop:hadoop-yarn-common:jar:2.5.0, org.apache.hadoop:hadoop-common:jar:2.5.0, com.google.guava:guava:jar:11.0.2, org.mortbay.jetty:jetty:jar:6.1.26, org.mortbay.jetty:jetty-util:jar:6.1.26, com.sun.jersey:jersey-json:jar:1.9, tomcat:jasper-compiler:jar:5.5.23, commons-configuration:commons-configuration:jar:1.6, commons-beanutils:commons-beanutils:jar:1.7.0, org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13, org.apache.zookeeper:zookeeper:jar:3.4.6, org.scala-lang:scala-library:jar:2.10.4, org.deephacks.lmdbjni:lmdbjni-osx64:jar:0.4.6, org.scalatest:scalatest_2.10:jar:2.2.6, org.scala-lang:scala-reflect:jar:2.10.6: Could not transfer artifact org.apache.spark:spark-core_2.10:jar:1.6.0 from/to scala-tools.org (http://scala-tools.org/repo-releases): sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :caffe-grid make: *** [build] Error 1 cdw@cdw-All-Series:~/CaffeOnSpark$

dawuchen commented 7 years ago

I've just solved the network problem .Because the central maven repo is in abroad and the download speed can let me get crazy,I changed a mirror server to download and the speed is very fast!