junshi15
commented
8 years ago Some unit tests failed? We will know more if you can show more log.
Open guody5 opened 8 years ago
junshi15
commented
8 years ago Some unit tests failed? We will know more if you can show more log.
guody5
commented
8 years ago cd caffe-public; make proto; make -j4 -e distribute; cd ..
make[1]: Entering directory /home/hadoop/whx/CaffeOnSpark/caffe-public' make[1]: Nothing to be done forproto'.
make[1]: Leaving directory /home/hadoop/whx/CaffeOnSpark/caffe-public' make[1]: Entering directory/home/hadoop/whx/CaffeOnSpark/caffe-public'
cp -r src/caffe/proto distribute/
cp -r include distribute/ mkdir -p distribute/include/caffe/proto cp .build_release/src/caffe/proto/caffe.pb.h distribute/include/caffe/proto
cp .build_release/tools/upgrade_solver_proto_text.bin .build_release/tools/compute_image_mean.bin .build_release/tools/device_query.bin .build_release/tools/extract_features.bin .build_release/tools/test_net.bin .build_release/tools/caffe.bin .build_release/tools/upgrade_net_proto_text.bin .build_release/tools/convert_imageset.bin .build_release/tools/net_speed_benchmark.bin .build_release/tools/train_net.bin .build_release/tools/finetune_net.bin .build_release/tools/upgrade_net_proto_binary.bin distribute/bin cp .build_release/examples/mnist/convert_mnist_data.bin .build_release/examples/siamese/convert_mnist_siamese_data.bin .build_release/examples/cifar10/convert_cifar_data.bin .build_release/examples/cpp_classification/classification.bin distribute/bin
cp .build_release/lib/libcaffe.a distribute/lib install -m 644 .build_release/lib/libcaffe.so.1.0.0-rc3 distribute/lib cd distribute/lib; rm -f libcaffe.so; ln -s libcaffe.so.1.0.0-rc3 libcaffe.so
cp -r python distribute/python
make[1]: Leaving directory `/home/hadoop/whx/CaffeOnSpark/caffe-public'
export LD_LIBRARY_PATH="/usr/local/cuda/lib64:/usr/lib/nvidia-352:/usr/local/cuda/lib64::/home/hadoop/whx/CuDNN/lib64:/home/hadoop/whx/OpenBLAS/lib:/home/hadoop/whx/CaffeOnSpark/caffe-public/distribute/lib:/home/hadoop/whx/CaffeOnSpark/caffe-distri/distribute/lib:/usr/lib64:/lib64 "; GLOG_minloglevel=1 mvn -Dspark2 -B package -DskipTests
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for com.yahoo.ml:caffe-grid:jar:0.1-SNAPSHOT
[WARNING] The expression ${version} is deprecated. Please use ${project.version} instead.
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] caffe
[INFO] caffe-distri
[INFO] caffe-grid
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building caffe 0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building caffe-distri 0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (proto) @ caffe-distri ---
[INFO] Executing tasks
protoc:
[exec] make[1]: Entering directory /home/hadoop/whx/CaffeOnSpark/caffe-distri' [exec] PROTOC ../caffe-public/distribute/proto/caffe.proto [exec] find:target/surefire-reports/Command': No such file or directory
[exec] find: line': No such file or directory [exec] find:suite': No such file or directory
[exec] find: target/surefire-reports/old/Command': No such file or directory [exec] find:line': No such file or directory
[exec] find: suite': No such file or directory [exec] make[1]: Leaving directory/home/hadoop/whx/CaffeOnSpark/caffe-distri'
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.0:compile (compile-generator) @ caffe-distri ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 10 source files to /home/hadoop/whx/CaffeOnSpark/caffe-distri/target/classes
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ caffe-distri ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /home/hadoop/whx/CaffeOnSpark/caffe-distri/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ caffe-distri ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 10 source files to /home/hadoop/whx/CaffeOnSpark/caffe-distri/target/classes
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (make) @ caffe-distri ---
[INFO] Executing tasks
make:
[exec] make[1]: Entering directory /home/hadoop/whx/CaffeOnSpark/caffe-distri' [exec] javah -d include/jni -force -cp target/classes \ [exec] com.yahoo.ml.jcaffe.FloatDataTransformer [exec] find:target/surefire-reports/Command': No such file or directory
[exec] find: line': No such file or directory [exec] find:suite': No such file or directory
[exec] find: target/surefire-reports/old/Command': No such file or directory [exec] find:line': No such file or directory
[exec] find: suite': No such file or directory [exec] javah -d include/jni -force -cp target/classes \ [exec] com.yahoo.ml.jcaffe.FloatArray [exec] javah -d include/jni -force -cp target/classes \ [exec] com.yahoo.ml.jcaffe.MatVector [exec] javah -d include/jni -force -cp target/classes \ [exec] com.yahoo.ml.jcaffe.CaffeNet [exec] javah -d include/jni -force -cp target/classes \ [exec] com.yahoo.ml.jcaffe.FloatBlob [exec] javah -d include/jni -force -cp target/classes \ [exec] com.yahoo.ml.jcaffe.Mat [exec] CXX src/main/cpp/util/mini_cluster.cpp [exec] CXX src/main/cpp/util/MemoryInputAdapter.cpp [exec] CXX src/main/cpp/util/parallel_cpu.cpp [exec] CXX src/main/cpp/util/socket.cpp [exec] CXX src/main/cpp/util/socket_sync.cpp [exec] CXX src/main/cpp/util/blocking_queue_extra.cpp [exec] CXX src/main/cpp/util/InputAdapter.cpp [exec] CXX src/main/cpp/util/socket_sync_cpu.cpp [exec] CXX src/main/cpp/util/CoSInputAdapter.cpp [exec] CXX src/main/cpp/cos_data_layer.cpp [exec] CXX src/main/cpp/jni/JniFloatDataTransformer.cpp [exec] CXX src/main/cpp/jni/JniFloatArray.cpp [exec] CXX src/main/cpp/jni/JniMatVector.cpp [exec] CXX src/main/cpp/jni/JniCaffeNet.cpp [exec] CXX src/main/cpp/jni/JniFloatBlob.cpp [exec] CXX src/main/cpp/jni/JniMat.cpp [exec] CXX src/main/cpp/CaffeNet.cpp [exec] CXX src/main/cpp/common.cpp [exec] AR -o .build_release/lib/libcaffedistri.a [exec] LD -o .build_release/lib/libcaffedistri.so .build_release/src/main/cpp/util/mini_cluster.o .build_release/src/main/cpp/util/rdma.o .build_release/src/main/cpp/util/rdma_sync.o .build_release/src/main/cpp/util/MemoryInputAdapter.o .build_release/src/main/cpp/util/parallel_cpu.o .build_release/src/main/cpp/util/socket.o .build_release/src/main/cpp/util/socket_sync.o .build_release/src/main/cpp/util/blocking_queue_extra.o .build_release/src/main/cpp/util/InputAdapter.o .build_release/src/main/cpp/util/socket_sync_cpu.o .build_release/src/main/cpp/util/CoSInputAdapter.o .build_release/src/main/cpp/cos_data_layer.o .build_release/src/main/cpp/jni/JniFloatDataTransformer.o .build_release/src/main/cpp/jni/JniFloatArray.o .build_release/src/main/cpp/jni/JniMatVector.o .build_release/src/main/cpp/jni/JniCaffeNet.o .build_release/src/main/cpp/jni/JniFloatBlob.o .build_release/src/main/cpp/jni/JniMat.o .build_release/src/main/cpp/CaffeNet.o .build_release/src/main/cpp/common.o [exec] # add include [exec] cp -r include distribute/ [exec] # add libraries [exec] cp .build_release/lib/libcaffedistri.a distribute/lib [exec] cp .build_release/lib/libcaffedistri.so distribute/lib [exec] make[1]: Leaving directory/home/hadoop/whx/CaffeOnSpark/caffe-distri'
[INFO] Executed tasks
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ caffe-distri ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.0:testCompile (default-testCompile) @ caffe-distri ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 8 source files to /home/hadoop/whx/CaffeOnSpark/caffe-distri/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ caffe-distri ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ caffe-distri ---
[INFO] Building jar: /home/hadoop/whx/CaffeOnSpark/caffe-distri/target/caffe-distri-0.1-SNAPSHOT.jar
[INFO]
[INFO] --- maven-assembly-plugin:2.2-beta-5:single (make-assembly) @ caffe-distri ---
[INFO] META-INF/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] org/ already added, skipping
[INFO] org/slf4j/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/maven/org.slf4j/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] org/ already added, skipping
[INFO] Building jar: /home/hadoop/whx/CaffeOnSpark/caffe-distri/target/caffe-distri-0.1-SNAPSHOT-jar-with-dependencies.jar
[INFO] META-INF/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] org/ already added, skipping
[INFO] org/slf4j/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/maven/org.slf4j/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] org/ already added, skipping
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building caffe-grid 0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-compiler-plugin:3.0:compile (compile-generator) @ caffe-grid ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ caffe-grid ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-scala-plugin:2.14.2:add-source (scala-compile-first) @ caffe-grid ---
[INFO] Add Source directory: /home/hadoop/whx/CaffeOnSpark/caffe-grid/src/main/scala
[INFO] Add Test Source directory: /home/hadoop/whx/CaffeOnSpark/caffe-grid/src/test/scala
[INFO]
[INFO] --- maven-scala-plugin:2.14.2:compile (scala-compile-first) @ caffe-grid ---
[INFO] Checking for multiple versions of scala
[WARNING] Expected all dependencies to require Scala version: 2.11.7
[WARNING] com.twitter:chill_2.11:0.8.0 requires scala version: 2.11.7
[WARNING] org.apache.spark:spark-core_2.11:2.0.0 requires scala version: 2.11.8
[WARNING] Multiple versions of scala libraries detected!
[INFO] includes = [/*.java,*/.scala,]
[INFO] excludes = []
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ caffe-grid ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ caffe-grid ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 11 resources
[INFO]
[INFO] --- maven-scala-plugin:2.14.2:testCompile (scala-test-compile) @ caffe-grid ---
[INFO] Checking for multiple versions of scala
[WARNING] Expected all dependencies to require Scala version: 2.11.7
[WARNING] com.twitter:chill_2.11:0.8.0 requires scala version: 2.11.7
[WARNING] org.apache.spark:spark-core_2.11:2.0.0 requires scala version: 2.11.8
[WARNING] Multiple versions of scala libraries detected!
[INFO] includes = [/*.java,*/.scala,]
[INFO] excludes = []
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-compiler-plugin:3.0:testCompile (default-testCompile) @ caffe-grid ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ caffe-grid ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- scalatest-maven-plugin:1.0:test (test) @ caffe-grid ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ caffe-grid ---
[INFO]
[INFO] --- maven-assembly-plugin:2.2-beta-5:single (make-assembly) @ caffe-grid ---
[INFO] META-INF/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] org/ already added, skipping
[INFO] org/slf4j/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/maven/org.slf4j/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] org/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] org/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/maven/org.deephacks.lmdbjni/ already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/native/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/maven/org.deephacks.lmdbjni/ already added, skipping
[INFO] Building jar: /home/hadoop/whx/CaffeOnSpark/caffe-grid/target/caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar
[INFO] META-INF/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] org/ already added, skipping
[INFO] org/slf4j/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/maven/org.slf4j/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] org/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] org/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/maven/org.deephacks.lmdbjni/ already added, skipping
[INFO] META-INF/ already added, skipping
[INFO] META-INF/MANIFEST.MF already added, skipping
[INFO] META-INF/native/ already added, skipping
[INFO] META-INF/maven/ already added, skipping
[INFO] META-INF/maven/org.deephacks.lmdbjni/ already added, skipping
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] caffe .............................................. SUCCESS [ 0.002 s]
[INFO] caffe-distri ....................................... SUCCESS [ 40.814 s]
[INFO] caffe-grid ......................................... SUCCESS [ 2.748 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 43.736 s
[INFO] Finished at: 2016-11-25T19:14:16+08:00
[INFO] Final Memory: 47M/727M
[INFO] ------------------------------------------------------------------------
jar -xvf caffe-grid/target/caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar META-INF/native/linux64/liblmdbjni.so
inflated: META-INF/native/linux64/liblmdbjni.so
mv META-INF/native/linux64/liblmdbjni.so /home/hadoop/whx/CaffeOnSpark/caffe-distri/distribute/lib
/home/hadoop/whx/CaffeOnSpark/scripts/setup-mnist.sh
--2016-11-25 19:14:19-- https://raw.githubusercontent.com/BVLC/caffe/master/data/mnist/get_mnist.sh
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.100.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.100.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 408 [text/plain]
Saving to: ‘get_mnist.sh.5’
0K 100% 246M=0s2016-11-25 19:14:19 (246 MB/s) - ‘get_mnist.sh.5’ saved [408/408]
Downloading... --2016-11-25 19:14:19-- http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz Resolving yann.lecun.com (yann.lecun.com)... 128.122.47.89 Connecting to yann.lecun.com (yann.lecun.com)|128.122.47.89|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 9912422 (9.5M) [application/x-gzip] Saving to: ‘train-images-idx3-ubyte.gz’
0K .......... .......... .......... .......... .......... 0% 62.0K 2m35s
50K .......... .......... .......... .......... .......... 1% 62.4K 2m34s100K .......... .......... .......... .......... .......... 1% 63.0K 2m33s 150K .......... .......... .......... .......... .......... 2% 63.9K 2m31s 200K .......... .......... .......... .......... .......... 2% 187K 2m10s 250K .......... .......... .......... .......... .......... 3% 94.5K 2m4s 300K .......... .......... .......... .......... .......... 3% 189K 1m53s 350K .......... .......... .......... .......... .......... 4% 193K 1m45s 400K .......... .......... .......... .......... .......... 4% 192K 98s 450K .......... .......... .......... .......... .......... 5% 192K 92s 500K .......... .......... .......... .......... .......... 5% 7.16M 84s 550K .......... .......... .......... .......... .......... 6% 196K 80s 600K .......... .......... .......... .......... .......... 6% 193K 77s 650K .......... .......... .......... .......... .......... 7% 10.9M 71s 700K .......... .......... .......... .......... .......... 7% 191K 69s 750K .......... .......... .......... .......... .......... 8% 9.12M 65s 800K .......... .......... .......... .......... .......... 8% 197K 63s 850K .......... .......... .......... .......... .......... 9% 8.14M 59s 900K .......... .......... .......... .......... .......... 9% 198K 58s 950K .......... .......... .......... .......... .......... 10% 9.47M 55s 1000K .......... .......... .......... .......... .......... 10% 202K 54s 1050K .......... .......... .......... .......... .......... 11% 5.31M 51s 1100K .......... .......... .......... .......... .......... 11% 10.1M 49s 1150K .......... .......... .......... .......... .......... 12% 203K 48s 1200K .......... .......... .......... .......... .......... 12% 4.76M 46s 1250K .......... .......... .......... .......... .......... 13% 200K 46s 1300K .......... .......... .......... .......... .......... 13% 7.32M 44s 1350K .......... .......... .......... .......... .......... 14% 5.86M 42s 1400K .......... .......... .......... .......... .......... 14% 205K 42s 1450K .......... .......... .......... .......... .......... 15% 9.47M 40s 1500K .......... .......... .......... .......... .......... 16% 5.50M 39s 1550K .......... .......... .......... .......... .......... 16% 4.11M 37s 1600K .......... .......... .......... .......... .......... 17% 209K 37s 1650K .......... .......... .......... .......... .......... 17% 4.12M 36s 1700K .......... .......... .......... .......... .......... 18% 8.30M 35s 1750K .......... .......... .......... .......... .......... 18% 1.82M 34s 1800K .......... .......... .......... .......... .......... 19% 227K 33s 1850K .......... .......... .......... .......... .......... 19% 6.06M 32s 1900K .......... .......... .......... .......... .......... 20% 3.66M 31s 1950K .......... .......... .......... .......... .......... 20% 3.94M 30s 2000K .......... .......... .......... .......... .......... 21% 222K 30s 2050K .......... .......... .......... .......... .......... 21% 6.33M 29s 2100K .......... .......... .......... .......... .......... 22% 6.33M 29s 2150K .......... .......... .......... .......... .......... 22% 1.10M 28s 2200K .......... .......... .......... .......... .......... 23% 747K 27s 2250K .......... .......... .......... .......... .......... 23% 364K 27s 2300K .......... .......... .......... .......... .......... 24% 4.93M 26s 2350K .......... .......... .......... .......... .......... 24% 781K 26s 2400K .......... .......... .......... .......... .......... 25% 716K 25s 2450K .......... .......... .......... .......... .......... 25% 87.2K 26s 2500K .......... .......... .......... .......... .......... 26% 55.1K 28s 2550K .......... .......... .......... .......... .......... 26% 125M 27s 2600K .......... .......... .......... .......... .......... 27% 120M 27s 2650K .......... .......... .......... .......... .......... 27% 145M 26s 2700K .......... .......... .......... .......... .......... 28% 146M 25s 2750K .......... .......... .......... .......... .......... 28% 148M 25s 2800K .......... .......... .......... .......... .......... 29% 129M 24s 2850K .......... .......... .......... .......... .......... 29% 148M 23s 2900K .......... .......... .......... .......... .......... 30% 126M 23s 2950K .......... .......... .......... .......... .......... 30% 150M 22s 3000K .......... .......... .......... .......... .......... 31% 126M 22s 3050K .......... .......... .......... .......... .......... 32% 145M 21s 3100K .......... .......... .......... .......... .......... 32% 151M 21s 3150K .......... .......... .......... .......... .......... 33% 147M 20s 3200K .......... .......... .......... .......... .......... 33% 131M 20s 3250K .......... .......... .......... .......... .......... 34% 151M 19s 3300K .......... .......... .......... .......... .......... 34% 151M 19s 3350K .......... .......... .......... .......... .......... 35% 38.6K 21s 3400K .......... .......... .......... .......... .......... 35% 95.6K 21s 3450K .......... .......... .......... .......... .......... 36% 95.1K 22s 3500K .......... .......... .......... .......... .......... 36% 194K 22s 3550K .......... .......... .......... .......... .......... 37% 190K 22s 3600K .......... .......... .......... .......... .......... 37% 10.8M 21s 3650K .......... .......... .......... .......... .......... 38% 190K 21s 3700K .......... .......... .......... .......... .......... 38% 203K 21s 3750K .......... .......... .......... .......... .......... 39% 7.58M 21s 3800K .......... .......... .......... .......... .......... 39% 10.8M 20s 3850K .......... .......... .......... .......... .......... 40% 207K 20s 3900K .......... .......... .......... .......... .......... 40% 11.6M 20s 3950K .......... .......... .......... .......... .......... 41% 10.8M 19s 4000K .......... .......... .......... .......... .......... 41% 204K 19s 4050K .......... .......... .......... .......... .......... 42% 8.01M 19s 4100K .......... .......... .......... .......... .......... 42% 207K 19s 4150K .......... .......... .......... .......... .......... 43% 6.66M 19s 4200K .......... .......... .......... .......... .......... 43% 5.65M 18s 4250K .......... .......... .......... .......... .......... 44% 205K 18s 4300K .......... .......... .......... .......... .......... 44% 4.99M 18s 4350K .......... .......... .......... .......... .......... 45% 6.41M 17s 4400K .......... .......... .......... .......... .......... 45% 10.3M 17s 4450K .......... .......... .......... .......... .......... 46% 205K 17s 4500K .......... .......... .......... .......... .......... 47% 5.51M 17s 4550K .......... .......... .......... .......... .......... 47% 7.01M 16s 4600K .......... .......... .......... .......... .......... 48% 203K 16s 4650K .......... .......... .......... .......... .......... 48% 1.22M 16s 4700K .......... .......... .......... .......... .......... 49% 676K 16s 4750K .......... .......... .......... .......... .......... 49% 333K 15s 4800K .......... .......... .......... .......... .......... 50% 4.74M 15s 4850K .......... .......... .......... .......... .......... 50% 1.00M 15s 4900K .......... .......... .......... .......... .......... 51% 716K 15s 4950K .......... .......... .......... .......... .......... 51% 371K 15s 5000K .......... .......... .......... .......... .......... 52% 3.41M 14s 5050K .......... .......... .......... .......... .......... 52% 780K 14s 5100K .......... .......... .......... .......... .......... 53% 725K 14s 5150K .......... .......... .......... .......... .......... 53% 713K 14s 5200K .......... .......... .......... .......... .......... 54% 714K 13s 5250K .......... .......... .......... .......... .......... 54% 705K 13s 5300K .......... .......... .......... .......... .......... 55% 715K 13s 5350K .......... .......... .......... .......... .......... 55% 139K 13s 5400K .......... .......... .......... .......... .......... 56% 137M 13s 5450K .......... .......... .......... .......... .......... 56% 147M 12s 5500K .......... .......... .......... .......... .......... 57% 149M 12s 5550K .......... .......... .......... .......... .......... 57% 114M 12s 5600K .......... .......... .......... .......... .......... 58% 831K 12s 5650K .......... .......... .......... .......... .......... 58% 611K 11s 5700K .......... .......... .......... .......... .......... 59% 502K 11s 5750K .......... .......... .......... .......... .......... 59% 713K 11s 5800K .......... .......... .......... .......... .......... 60% 711K 11s 5850K .......... .......... .......... .......... .......... 60% 527K 11s 5900K .......... .......... .......... .......... .......... 61% 642K 11s 5950K .......... .......... .......... .......... .......... 61% 695K 10s 6000K .......... .......... .......... .......... .......... 62% 152K 10s 6050K .......... .......... .......... .......... .......... 63% 114M 10s 6100K .......... .......... .......... .......... .......... 63% 142M 10s 6150K .......... .......... .......... .......... .......... 64% 145M 10s 6200K .......... .......... .......... .......... .......... 64% 613K 10s 6250K .......... .......... .......... .......... .......... 65% 511K 9s 6300K .......... .......... .......... .......... .......... 65% 488K 9s 6350K .......... .......... .......... .......... .......... 66% 396K 9s 6400K .......... .......... .......... .......... .......... 66% 606K 9s 6450K .......... .......... .......... .......... .......... 67% 432K 9s 6500K .......... .......... .......... .......... .......... 67% 538K 9s 6550K .......... .......... .......... .......... .......... 68% 422K 8s 6600K .......... .......... .......... .......... .......... 68% 538K 8s 6650K .......... .......... .......... .......... .......... 69% 552K 8s 6700K .......... .......... .......... .......... .......... 69% 437K 8s 6750K .......... .......... .......... .......... .......... 70% 512K 8s 6800K .......... .......... .......... .......... .......... 70% 716K 8s 6850K .......... .......... .......... .......... .......... 71% 427K 8s 6900K .......... .......... .......... .......... .......... 71% 425K 7s 6950K .......... .......... .......... .......... .......... 72% 774K 7s 7000K .......... .......... .......... .......... .......... 72% 520K 7s 7050K .......... .......... .......... .......... .......... 73% 415K 7s 7100K .......... .......... .......... .......... .......... 73% 569K 7s 7150K .......... .......... .......... .......... .......... 74% 736K 7s 7200K .......... .......... .......... .......... .......... 74% 416K 6s 7250K .......... .......... .......... .......... .......... 75% 524K 6s 7300K .......... .......... .......... .......... .......... 75% 741K 6s 7350K .......... .......... .......... .......... .......... 76% 370K 6s 7400K .......... .......... .......... .......... .......... 76% 599K 6s 7450K .......... .......... .......... .......... .......... 77% 579K 6s 7500K .......... .......... .......... .......... .......... 77% 449K 6s 7550K .......... .......... .......... .......... .......... 78% 657K 5s 7600K .......... .......... .......... .......... .......... 79% 515K 5s 7650K .......... .......... .......... .......... .......... 79% 534K 5s 7700K .......... .......... .......... .......... .......... 80% 583K 5s 7750K .......... .......... .......... .......... .......... 80% 491K 5s 7800K .......... .......... .......... .......... .......... 81% 639K 5s 7850K .......... .......... .......... .......... .......... 81% 484K 5s 7900K .......... .......... .......... .......... .......... 82% 490K 4s 7950K .......... .......... .......... .......... .......... 82% 582K 4s 8000K .......... .......... .......... .......... .......... 83% 459K 4s 8050K .......... .......... .......... .......... .......... 83% 576K 4s 8100K .......... .......... .......... .......... .......... 84% 469K 4s 8150K .......... .......... .......... .......... .......... 84% 550K 4s 8200K .......... .......... .......... .......... .......... 85% 507K 4s 8250K .......... .......... .......... .......... .......... 85% 565K 4s 8300K .......... .......... .......... .......... .......... 86% 493K 3s 8350K .......... .......... .......... .......... .......... 86% 553K 3s 8400K .......... .......... .......... .......... .......... 87% 472K 3s 8450K .......... .......... .......... .......... .......... 87% 746K 3s 8500K .......... .......... .......... .......... .......... 88% 516K 3s 8550K .......... .......... .......... .......... .......... 88% 380K 3s 8600K .......... .......... .......... .......... .......... 89% 965K 3s 8650K .......... .......... .......... .......... .......... 89% 562K 2s 8700K .......... .......... .......... .......... .......... 90% 405K 2s 8750K .......... .......... .......... .......... .......... 90% 611K 2s 8800K .......... .......... .......... .......... .......... 91% 851K 2s 8850K .......... .......... .......... .......... .......... 91% 371K 2s 8900K .......... .......... .......... .......... .......... 92% 507K 2s 8950K .......... .......... .......... .......... .......... 92% 781K 2s 9000K .......... .......... .......... .......... .......... 93% 484K 2s 9050K .......... .......... .......... .......... .......... 94% 527K 1s 9100K .......... .......... .......... .......... .......... 94% 776K 1s 9150K .......... .......... .......... .......... .......... 95% 129K 1s 9200K .......... .......... .......... .......... .......... 95% 115M 1s 9250K .......... .......... .......... .......... .......... 96% 131M 1s 9300K .......... .......... .......... .......... .......... 96% 3.80M 1s 9350K .......... .......... .......... .......... .......... 97% 414K 1s 9400K .......... .......... .......... .......... .......... 97% 398K 1s 9450K .......... .......... .......... .......... .......... 98% 512K 0s 9500K .......... .......... .......... .......... .......... 98% 391K 0s 9550K .......... .......... .......... .......... .......... 99% 420K 0s 9600K .......... .......... .......... .......... .......... 99% 588K 0s 9650K .......... .......... .......... 100% 374K=24s
2016-11-25 19:14:43 (403 KB/s) - ‘train-images-idx3-ubyte.gz’ saved [9912422/9912422]
--2016-11-25 19:14:44-- http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz Resolving yann.lecun.com (yann.lecun.com)... 128.122.47.89 Connecting to yann.lecun.com (yann.lecun.com)|128.122.47.89|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 28881 (28K) [application/x-gzip] Saving to: ‘train-labels-idx1-ubyte.gz’
0K .......... .......... ........ 100% 123K=0.2s2016-11-25 19:14:44 (123 KB/s) - ‘train-labels-idx1-ubyte.gz’ saved [28881/28881]
--2016-11-25 19:14:44-- http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz Resolving yann.lecun.com (yann.lecun.com)... 128.122.47.89 Connecting to yann.lecun.com (yann.lecun.com)|128.122.47.89|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1648877 (1.6M) [application/x-gzip] Saving to: ‘t10k-images-idx3-ubyte.gz’
0K .......... .......... .......... .......... .......... 3% 110K 14s
50K .......... .......... .......... .......... .......... 6% 111K 14s100K .......... .......... .......... .......... .......... 9% 11.0M 9s 150K .......... .......... .......... .......... .......... 12% 223K 8s 200K .......... .......... .......... .......... .......... 15% 1.18M 6s 250K .......... .......... .......... .......... .......... 18% 268K 6s 300K .......... .......... .......... .......... .......... 21% 1.17M 5s 350K .......... .......... .......... .......... .......... 24% 240K 5s 400K .......... .......... .......... .......... .......... 27% 715K 4s 450K .......... .......... .......... .......... .......... 31% 348K 4s 500K .......... .......... .......... .......... .......... 34% 783K 4s 550K .......... .......... .......... .......... .......... 37% 313K 3s 600K .......... .......... .......... .......... .......... 40% 2.14M 3s 650K .......... .......... .......... .......... .......... 43% 780K 3s 700K .......... .......... .......... .......... .......... 46% 307K 3s 750K .......... .......... .......... .......... .......... 49% 1.49M 2s 800K .......... .......... .......... .......... .......... 52% 291K 2s 850K .......... .......... .......... .......... .......... 55% 2.03M 2s 900K .......... .......... .......... .......... .......... 58% 1.10M 2s 950K .......... .......... .......... .......... .......... 62% 298K 2s 1000K .......... .......... .......... .......... .......... 65% 2.18M 1s 1050K .......... .......... .......... .......... .......... 68% 1.30M 1s 1100K .......... .......... .......... .......... .......... 71% 303K 1s 1150K .......... .......... .......... .......... .......... 74% 2.06M 1s 1200K .......... .......... .......... .......... .......... 77% 1.28M 1s 1250K .......... .......... .......... .......... .......... 80% 1.03M 1s 1300K .......... .......... .......... .......... .......... 83% 357K 1s 1350K .......... .......... .......... .......... .......... 86% 1.18M 0s 1400K .......... .......... .......... .......... .......... 90% 1.53M 0s 1450K .......... .......... .......... .......... .......... 93% 330K 0s 1500K .......... .......... .......... .......... .......... 96% 1.10M 0s 1550K .......... .......... .......... .......... .......... 99% 5.13M 0s 1600K .......... 100% 454K=3.5s
2016-11-25 19:14:48 (458 KB/s) - ‘t10k-images-idx3-ubyte.gz’ saved [1648877/1648877]
--2016-11-25 19:14:48-- http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz Resolving yann.lecun.com (yann.lecun.com)... 128.122.47.89 Connecting to yann.lecun.com (yann.lecun.com)|128.122.47.89|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 4542 (4.4K) [application/x-gzip] Saving to: ‘t10k-labels-idx1-ubyte.gz’
0K .... 100% 212M=0s2016-11-25 19:14:50 (212 MB/s) - ‘t10k-labels-idx1-ubyte.gz’ saved [4542/4542]
Creating lmdb...
I1125 19:14:50.533814 30347 db_lmdb.cpp:35] Opened lmdb examples/mnist/mnist_train_lmdb
I1125 19:14:50.533916 30347 convert_mnist_data.cpp:88] A total of 60000 items.
I1125 19:14:50.533921 30347 convert_mnist_data.cpp:89] Rows: 28 Cols: 28
I1125 19:14:55.375974 30347 convert_mnist_data.cpp:108] Processed 60000 files.
I1125 19:14:55.420739 30349 db_lmdb.cpp:35] Opened lmdb examples/mnist/mnist_test_lmdb
I1125 19:14:55.420835 30349 convert_mnist_data.cpp:88] A total of 10000 items.
I1125 19:14:55.420840 30349 convert_mnist_data.cpp:89] Rows: 28 Cols: 28
I1125 19:14:56.167332 30349 convert_mnist_data.cpp:108] Processed 10000 files.
Done.
export LD_LIBRARY_PATH="/usr/local/cuda/lib64:/usr/lib/nvidia-352:/usr/local/cuda/lib64::/home/hadoop/whx/CuDNN/lib64:/home/hadoop/whx/OpenBLAS/lib:/home/hadoop/whx/CaffeOnSpark/caffe-public/distribute/lib:/home/hadoop/whx/CaffeOnSpark/caffe-distri/distribute/lib:/usr/lib64:/lib64 "; GLOG_minloglevel=1 mvn -Dspark2 -B test
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for com.yahoo.ml:caffe-grid:jar:0.1-SNAPSHOT
[WARNING] The expression ${version} is deprecated. Please use ${project.version} instead.
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] caffe
[INFO] caffe-distri
[INFO] caffe-grid
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building caffe 0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building caffe-distri 0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (proto) @ caffe-distri ---
[INFO] Executing tasks
protoc:
[exec] make[1]: Entering directory /home/hadoop/whx/CaffeOnSpark/caffe-distri' [exec] make[1]: Nothing to be done forproto'.
[exec] make[1]: Leavifind: target/surefire-reports/Command': No such file or directory [exec] find:line': No such file or directory
[exec] find: suite': No such file or directory [exec] find:target/surefire-reports/old/Command': No such file or directory
[exec] find: line': No such file or directory [exec] find:suite': No such file or directory
[exec] ng directory `/home/hadoop/whx/CaffeOnSpark/caffe-distri'
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.0:compile (compile-generator) @ caffe-distri ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ caffe-distri ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] skip non existing resourceDirectory /home/hadoop/whx/CaffeOnSpark/caffe-distri/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ caffe-distri ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (make) @ caffe-distri ---
[INFO] Executing tasks
make:
[exec] make[1]: Entering directory /home/hadoop/whx/CaffeOnSpark/caffe-distri' [exec] # add include [exec] cp -r include distribute/ [exec] # add libraries [exec] cp .build_release/lib/libcaffedistri.a distribute/lib [exec] cp .build_release/lib/libcaffedistri.so distribute/lib [exec] make[1]: Leaving dfind:target/surefire-reports/Command': No such file or directory
[exec] find: line': No such file or directory [exec] find:suite': No such file or directory
[exec] find: target/surefire-reports/old/Command': No such file or directory [exec] find:line': No such file or directory
[exec] find: suite': No such file or directory [exec] irectory/home/hadoop/whx/CaffeOnSpark/caffe-distri'
[INFO] Executed tasks
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ caffe-distri ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.0:testCompile (default-testCompile) @ caffe-distri ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ caffe-distri ---
[INFO] Surefire report directory: /home/hadoop/whx/CaffeOnSpark/caffe-distri/target/surefire-reports
Running TestSuite Configuring TestNG with: org.apache.maven.surefire.testng.conf.TestNG652Configurator@5d22bbb7 WARNING: Logging before InitGoogleLogging() is written to STDERR E1125 19:17:13.390496 30819 socket.cpp:315] ERROR: Unable to connect to socket server [211.139.136.73] E1125 19:19:21.646510 30819 socket.cpp:315] ERROR: Unable to connect to socket server [211.139.136.73] E1125 19:21:30.926529 30819 socket.cpp:315] ERROR: Unable to connect to socket server [211.139.136.73] E1125 19:23:42.254417 30819 socket.cpp:315] ERROR: Unable to connect to socket server [221.179.46.190] E1125 19:25:57.550531 30819 socket.cpp:315] ERROR: Unable to connect to socket server [221.179.46.194] E1125 19:26:26.898597 30819 JniCaffeNet.cpp:238] Solver index invalid E1125 19:26:33.576465 30819 JniCaffeNet.cpp:260] Solver index invalid E1125 19:26:40.333170 30819 JniCaffeNet.cpp:482] Solver index invalid E1125 19:26:47.072821 30819 JniCaffeNet.cpp:548] Solver index invalid E1125 19:26:53.914898 30819 JniCaffeNet.cpp:288] data is NULL CaffeNetTest training:....................CaffeNetTest test:....................E1125 19:27:57.419908 30819 JniCaffeNet.cpp:394] data is NULL E1125 19:27:57.425267 30819 JniFloatArray.cpp:35] negative index E1125 19:27:57.425463 30819 JniFloatArray.cpp:69] Invalid index for floatarray E1125 19:27:57.425674 30819 JniFloatBlob.cpp:116] source is NULL E1125 19:27:57.425961 30819 JniFloatBlob.cpp:85] Invalid val for reshape dimension E1125 19:27:57.426113 30819 JniFloatBlob.cpp:184] input array is NULL E1125 19:27:57.431785 30819 JniMatVector.cpp:110] Invalid Mat index in MatVector E1125 19:27:57.432226 30819 JniMatVector.cpp:211] Invalid Mat index in MatVector E1125 19:27:57.432659 30819 JniMatVector.cpp:153] Invalid Mat index in MatVector E1125 19:27:57.433079 30819 JniMatVector.cpp:182] Invalid Mat index in MatVector E1125 19:27:57.434484 30819 common.cpp:38] object is NULL E1125 19:27:57.434500 30819 JniMatVector.cpp:79] invalid native address of Mat E1125 19:27:57.434993 30819 JniMatVector.cpp:85] invalid index in MatVector E1125 19:27:57.435289 30819 JniMatVector.cpp:21] Negative MatVector size specified TRAINING:........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................ TEST:........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................ Tests run: 38, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 784.157 sec
Results :
Tests run: 38, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building caffe-grid 0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-compiler-plugin:3.0:compile (compile-generator) @ caffe-grid ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 1 source file to /home/hadoop/whx/CaffeOnSpark/caffe-grid/target/classes
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ caffe-grid ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-scala-plugin:2.14.2:add-source (scala-compile-first) @ caffe-grid ---
[INFO] Add Source directory: /home/hadoop/whx/CaffeOnSpark/caffe-grid/src/main/scala
[INFO] Add Test Source directory: /home/hadoop/whx/CaffeOnSpark/caffe-grid/src/test/scala
[INFO]
[INFO] --- maven-scala-plugin:2.14.2:compile (scala-compile-first) @ caffe-grid ---
[INFO] Checking for multiple versions of scala
[WARNING] Expected all dependencies to require Scala version: 2.11.7
[WARNING] com.twitter:chill_2.11:0.8.0 requires scala version: 2.11.7
[WARNING] org.apache.spark:spark-core_2.11:2.0.0 requires scala version: 2.11.8
[WARNING] Multiple versions of scala libraries detected!
[INFO] includes = [/*.java,*/.scala,]
[INFO] excludes = []
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ caffe-grid ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 1 source file to /home/hadoop/whx/CaffeOnSpark/caffe-grid/target/classes
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ caffe-grid ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 11 resources
[INFO]
[INFO] --- maven-scala-plugin:2.14.2:testCompile (scala-test-compile) @ caffe-grid ---
[INFO] Checking for multiple versions of scala
[WARNING] Expected all dependencies to require Scala version: 2.11.7
[WARNING] com.twitter:chill_2.11:0.8.0 requires scala version: 2.11.7
[WARNING] org.apache.spark:spark-core_2.11:2.0.0 requires scala version: 2.11.8
[WARNING] Multiple versions of scala libraries detected!
[INFO] includes = [/*.java,*/.scala,]
[INFO] excludes = []
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-compiler-plugin:3.0:testCompile (default-testCompile) @ caffe-grid ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ caffe-grid ---
[INFO] Surefire report directory: /home/hadoop/whx/CaffeOnSpark/caffe-grid/target/surefire-reports
Running TestSuite Configuring TestNG with: org.apache.maven.surefire.testng.conf.TestNG652Configurator@22f71333 Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.201 sec
Results :
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] --- scalatest-maven-plugin:1.0:test (test) @ caffe-grid ---
Discovery starting.
Discovery completed in 172 milliseconds.
Run starting. Expected test count is: 7
InterleaveTest:
2016-11-25 19:28:07,258 [ScalaTest-main-running-DiscoverySuite] INFO spark.SparkContext - Running Spark version 2.0.0
2016-11-25 19:28:07,479 [ScalaTest-main-running-DiscoverySuite] INFO spark.SecurityManager - Changing view acls to: hadoop
2016-11-25 19:28:07,479 [ScalaTest-main-running-DiscoverySuite] INFO spark.SecurityManager - Changing modify acls to: hadoop
2016-11-25 19:28:07,480 [ScalaTest-main-running-DiscoverySuite] INFO spark.SecurityManager - Changing view acls groups to:
2016-11-25 19:28:07,480 [ScalaTest-main-running-DiscoverySuite] INFO spark.SecurityManager - Changing modify acls groups to:
2016-11-25 19:28:07,480 [ScalaTest-main-running-DiscoverySuite] INFO spark.SecurityManager - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set()
2016-11-25 19:28:07,644 [ScalaTest-main-running-DiscoverySuite] INFO util.Utils - Successfully started service 'sparkDriver' on port 40616.
2016-11-25 19:28:07,653 [ScalaTest-main-running-DiscoverySuite] INFO spark.SparkEnv - Registering MapOutputTracker
2016-11-25 19:28:07,661 [ScalaTest-main-running-DiscoverySuite] INFO spark.SparkEnv - Registering BlockManagerMaster
2016-11-25 19:28:07,668 [ScalaTest-main-running-DiscoverySuite] INFO storage.DiskBlockManager - Created local directory at /tmp/blockmgr-f25a0950-121f-45f0-aacd-97bfa0627e61
2016-11-25 19:28:07,677 [ScalaTest-main-running-DiscoverySuite] INFO memory.MemoryStore - MemoryStore started with capacity 1941.6 MB
2016-11-25 19:28:07,702 [ScalaTest-main-running-DiscoverySuite] INFO spark.SparkEnv - Registering OutputCommitCoordinator
2016-11-25 19:28:07,744 [ScalaTest-main-running-DiscoverySuite] INFO util.log - Logging initialized @1015ms
2016-11-25 19:28:07,785 [ScalaTest-main-running-DiscoverySuite] INFO server.Server - jetty-9.2.z-SNAPSHOT
2016-11-25 19:28:07,793 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@67ef029{/jobs,null,AVAILABLE}
2016-11-25 19:28:07,793 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@7df587ef{/jobs/json,null,AVAILABLE}
2016-11-25 19:28:07,793 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@6e57e95e{/jobs/job,null,AVAILABLE}
2016-11-25 19:28:07,793 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@2755d705{/jobs/job/json,null,AVAILABLE}
2016-11-25 19:28:07,793 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@56db847e{/stages,null,AVAILABLE}
2016-11-25 19:28:07,793 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@740abb5{/stages/json,null,AVAILABLE}
2016-11-25 19:28:07,793 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@560cbf1a{/stages/stage,null,AVAILABLE}
2016-11-25 19:28:07,794 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@5fe8b721{/stages/stage/json,null,AVAILABLE}
2016-11-25 19:28:07,794 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@551a20d6{/stages/pool,null,AVAILABLE}
2016-11-25 19:28:07,794 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@578524c3{/stages/pool/json,null,AVAILABLE}
2016-11-25 19:28:07,794 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@64c2b546{/storage,null,AVAILABLE}
2016-11-25 19:28:07,794 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@7e094740{/storage/json,null,AVAILABLE}
2016-11-25 19:28:07,794 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@7a11c4c7{/storage/rdd,null,AVAILABLE}
2016-11-25 19:28:07,794 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@4cc547a{/storage/rdd/json,null,AVAILABLE}
2016-11-25 19:28:07,794 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@7555b920{/environment,null,AVAILABLE}
2016-11-25 19:28:07,794 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@4152d38d{/environment/json,null,AVAILABLE}
2016-11-25 19:28:07,794 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@3591009c{/executors,null,AVAILABLE}
2016-11-25 19:28:07,794 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@5398edd0{/executors/json,null,AVAILABLE}
2016-11-25 19:28:07,795 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@b5cc23a{/executors/threadDump,null,AVAILABLE}
2016-11-25 19:28:07,795 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@5cc5b667{/executors/threadDump/json,null,AVAILABLE}
2016-11-25 19:28:07,797 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@61edc883{/static,null,AVAILABLE}
2016-11-25 19:28:07,798 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@758f4f03{/,null,AVAILABLE}
2016-11-25 19:28:07,798 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@182f1e9a{/api,null,AVAILABLE}
2016-11-25 19:28:07,798 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@6928f576{/stages/stage/kill,null,AVAILABLE}
2016-11-25 19:28:07,801 [ScalaTest-main-running-DiscoverySuite] INFO server.ServerConnector - Started ServerConnector@3deb2326{HTTP/1.1}{0.0.0.0:4040}
2016-11-25 19:28:07,801 [ScalaTest-main-running-DiscoverySuite] INFO server.Server - Started @1072ms
2016-11-25 19:28:07,801 [ScalaTest-main-running-DiscoverySuite] INFO util.Utils - Successfully started service 'SparkUI' on port 4040.
2016-11-25 19:28:07,802 [ScalaTest-main-running-DiscoverySuite] INFO ui.SparkUI - Bound SparkUI to 0.0.0.0, and started at http://192.168.1.103:4040
2016-11-25 19:28:07,844 [ScalaTest-main-running-DiscoverySuite] INFO executor.Executor - Starting executor ID driver on host localhost
2016-11-25 19:28:07,857 [ScalaTest-main-running-DiscoverySuite] INFO util.Utils - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 46570.
2016-11-25 19:28:07,858 [ScalaTest-main-running-DiscoverySuite] INFO netty.NettyBlockTransferService - Server created on 192.168.1.103:46570
2016-11-25 19:28:07,858 [ScalaTest-main-running-DiscoverySuite] INFO storage.BlockManagerMaster - Registering BlockManager BlockManagerId(driver, 192.168.1.103, 46570)
2016-11-25 19:28:07,860 [dispatcher-event-loop-2] INFO storage.BlockManagerMasterEndpoint - Registering block manager 192.168.1.103:46570 with 1941.6 MB RAM, BlockManagerId(driver, 192.168.1.103, 46570)
2016-11-25 19:28:07,862 [ScalaTest-main-running-DiscoverySuite] INFO storage.BlockManagerMaster - Registered BlockManager BlockManagerId(driver, 192.168.1.103, 46570)
2016-11-25 19:28:07,941 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@383f3558{/metrics/json,null,AVAILABLE}
RUN ABORTED
java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaMirrors$JavaMirror;
at com.yahoo.ml.caffe.CaffeOnSpark.
Results :
Tests run: 38, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building caffe-grid 0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-compiler-plugin:3.0:compile (compile-generator) @ caffe-grid ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 1 source file to /home/hadoop/whx/CaffeOnSpark/caffe-grid/target/classes
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ caffe-grid ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 1 resource
[INFO]
[INFO] --- maven-scala-plugin:2.14.2:add-source (scala-compile-first) @ caffe-grid ---
[INFO] Add Source directory: /home/hadoop/whx/CaffeOnSpark/caffe-grid/src/main/scala
[INFO] Add Test Source directory: /home/hadoop/whx/CaffeOnSpark/caffe-grid/src/test/scala
[INFO]
[INFO] --- maven-scala-plugin:2.14.2:compile (scala-compile-first) @ caffe-grid ---
[INFO] Checking for multiple versions of scala
[WARNING] Expected all dependencies to require Scala version: 2.11.7
[WARNING] com.twitter:chill_2.11:0.8.0 requires scala version: 2.11.7
[WARNING] org.apache.spark:spark-core_2.11:2.0.0 requires scala version: 2.11.8
[WARNING] Multiple versions of scala libraries detected!
[INFO] includes = [/*.java,*/.scala,]
[INFO] excludes = []
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ caffe-grid ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO] Compiling 1 source file to /home/hadoop/whx/CaffeOnSpark/caffe-grid/target/classes
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ caffe-grid ---
[WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 11 resources
[INFO]
[INFO] --- maven-scala-plugin:2.14.2:testCompile (scala-test-compile) @ caffe-grid ---
[INFO] Checking for multiple versions of scala
[WARNING] Expected all dependencies to require Scala version: 2.11.7
[WARNING] com.twitter:chill_2.11:0.8.0 requires scala version: 2.11.7
[WARNING] org.apache.spark:spark-core_2.11:2.0.0 requires scala version: 2.11.8
[WARNING] Multiple versions of scala libraries detected!
[INFO] includes = [/*.java,*/.scala,]
[INFO] excludes = []
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-compiler-plugin:3.0:testCompile (default-testCompile) @ caffe-grid ---
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e. build is platform dependent!
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ caffe-grid ---
[INFO] Surefire report directory: /home/hadoop/whx/CaffeOnSpark/caffe-grid/target/surefire-reports
Running TestSuite Configuring TestNG with: org.apache.maven.surefire.testng.conf.TestNG652Configurator@22f71333 Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.239 sec
Results :
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] --- scalatest-maven-plugin:1.0:test (test) @ caffe-grid ---
Discovery starting.
Discovery completed in 189 milliseconds.
Run starting. Expected test count is: 7
InterleaveTest:
2016-11-25 19:24:59,831 [ScalaTest-main-running-DiscoverySuite] INFO spark.SparkContext - Running Spark version 2.0.0
2016-11-25 19:25:00,063 [ScalaTest-main-running-DiscoverySuite] INFO spark.SecurityManager - Changing view acls to: hadoop
2016-11-25 19:25:00,064 [ScalaTest-main-running-DiscoverySuite] INFO spark.SecurityManager - Changing modify acls to: hadoop
2016-11-25 19:25:00,064 [ScalaTest-main-running-DiscoverySuite] INFO spark.SecurityManager - Changing view acls groups to:
2016-11-25 19:25:00,064 [ScalaTest-main-running-DiscoverySuite] INFO spark.SecurityManager - Changing modify acls groups to:
2016-11-25 19:25:00,064 [ScalaTest-main-running-DiscoverySuite] INFO spark.SecurityManager - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set()
2016-11-25 19:25:00,233 [ScalaTest-main-running-DiscoverySuite] INFO util.Utils - Successfully started service 'sparkDriver' on port 49138.
2016-11-25 19:25:00,242 [ScalaTest-main-running-DiscoverySuite] INFO spark.SparkEnv - Registering MapOutputTracker
2016-11-25 19:25:00,251 [ScalaTest-main-running-DiscoverySuite] INFO spark.SparkEnv - Registering BlockManagerMaster
2016-11-25 19:25:00,257 [ScalaTest-main-running-DiscoverySuite] INFO storage.DiskBlockManager - Created local directory at /tmp/blockmgr-fd57c0fc-d23a-478e-9938-9b004884237a
2016-11-25 19:25:00,267 [ScalaTest-main-running-DiscoverySuite] INFO memory.MemoryStore - MemoryStore started with capacity 1941.6 MB
2016-11-25 19:25:00,293 [ScalaTest-main-running-DiscoverySuite] INFO spark.SparkEnv - Registering OutputCommitCoordinator
2016-11-25 19:25:00,339 [ScalaTest-main-running-DiscoverySuite] INFO util.log - Logging initialized @1135ms
2016-11-25 19:25:00,379 [ScalaTest-main-running-DiscoverySuite] INFO server.Server - jetty-9.2.z-SNAPSHOT
2016-11-25 19:25:00,387 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@2e77b8cf{/jobs,null,AVAILABLE}
2016-11-25 19:25:00,387 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@2c4ca0f9{/jobs/json,null,AVAILABLE}
2016-11-25 19:25:00,387 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@67ef029{/jobs/job,null,AVAILABLE}
2016-11-25 19:25:00,388 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@7df587ef{/jobs/job/json,null,AVAILABLE}
2016-11-25 19:25:00,388 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@6e57e95e{/stages,null,AVAILABLE}
2016-11-25 19:25:00,388 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@2755d705{/stages/json,null,AVAILABLE}
2016-11-25 19:25:00,388 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@56db847e{/stages/stage,null,AVAILABLE}
2016-11-25 19:25:00,388 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@740abb5{/stages/stage/json,null,AVAILABLE}
2016-11-25 19:25:00,388 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@560cbf1a{/stages/pool,null,AVAILABLE}
2016-11-25 19:25:00,388 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@5fe8b721{/stages/pool/json,null,AVAILABLE}
2016-11-25 19:25:00,388 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@551a20d6{/storage,null,AVAILABLE}
2016-11-25 19:25:00,388 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@578524c3{/storage/json,null,AVAILABLE}
2016-11-25 19:25:00,389 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@64c2b546{/storage/rdd,null,AVAILABLE}
2016-11-25 19:25:00,389 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@7e094740{/storage/rdd/json,null,AVAILABLE}
2016-11-25 19:25:00,389 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@7a11c4c7{/environment,null,AVAILABLE}
2016-11-25 19:25:00,389 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@4cc547a{/environment/json,null,AVAILABLE}
2016-11-25 19:25:00,389 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@7555b920{/executors,null,AVAILABLE}
2016-11-25 19:25:00,389 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@4152d38d{/executors/json,null,AVAILABLE}
2016-11-25 19:25:00,389 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@3591009c{/executors/threadDump,null,AVAILABLE}
2016-11-25 19:25:00,389 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@5398edd0{/executors/threadDump/json,null,AVAILABLE}
2016-11-25 19:25:00,392 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@b5cc23a{/static,null,AVAILABLE}
2016-11-25 19:25:00,392 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@5cc5b667{/,null,AVAILABLE}
2016-11-25 19:25:00,393 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@61edc883{/api,null,AVAILABLE}
2016-11-25 19:25:00,393 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@758f4f03{/stages/stage/kill,null,AVAILABLE}
2016-11-25 19:25:00,396 [ScalaTest-main-running-DiscoverySuite] INFO server.ServerConnector - Started ServerConnector@761cc6b3{HTTP/1.1}{0.0.0.0:4040}
2016-11-25 19:25:00,396 [ScalaTest-main-running-DiscoverySuite] INFO server.Server - Started @1193ms
2016-11-25 19:25:00,396 [ScalaTest-main-running-DiscoverySuite] INFO util.Utils - Successfully started service 'SparkUI' on port 4040.
2016-11-25 19:25:00,397 [ScalaTest-main-running-DiscoverySuite] INFO ui.SparkUI - Bound SparkUI to 0.0.0.0, and started at http://192.168.1.103:4040
2016-11-25 19:25:00,440 [ScalaTest-main-running-DiscoverySuite] INFO executor.Executor - Starting executor ID driver on host localhost
2016-11-25 19:25:00,452 [ScalaTest-main-running-DiscoverySuite] INFO util.Utils - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 33910.
2016-11-25 19:25:00,452 [ScalaTest-main-running-DiscoverySuite] INFO netty.NettyBlockTransferService - Server created on 192.168.1.103:33910
2016-11-25 19:25:00,453 [ScalaTest-main-running-DiscoverySuite] INFO storage.BlockManagerMaster - Registering BlockManager BlockManagerId(driver, 192.168.1.103, 33910)
2016-11-25 19:25:00,454 [dispatcher-event-loop-2] INFO storage.BlockManagerMasterEndpoint - Registering block manager 192.168.1.103:33910 with 1941.6 MB RAM, BlockManagerId(driver, 192.168.1.103, 33910)
2016-11-25 19:25:00,456 [ScalaTest-main-running-DiscoverySuite] INFO storage.BlockManagerMaster - Registered BlockManager BlockManagerId(driver, 192.168.1.103, 33910)
2016-11-25 19:25:00,545 [ScalaTest-main-running-DiscoverySuite] INFO handler.ContextHandler - Started o.s.j.s.ServletContextHandler@27ead29e{/metrics/json,null,AVAILABLE}
RUN ABORTED
java.lang.NoSuchMethodError: scala.reflect.api.JavaUniverse.runtimeMirror(Ljava/lang/ClassLoader;)Lscala/reflect/api/JavaMirrors$JavaMirror;
at com.yahoo.ml.caffe.CaffeOnSpark.
This is my log.Thanks for your help,Junshi15
junshi15
commented
8 years ago InterleaveTest failed. I am not sure why, though.
guody5
commented
8 years ago Junshi15: I have solve this problem. I remove some code from caffeonspark/caffe-grid/pom.xml. In the end,I make build successfully.But I don't know whether it will have something wrong in the future.
/home/hadoop/whx/CaffeOnSpark/caffe-grid/src/main/python export LD_LIBRARY_PATH=/usr/lib/nvidia-352:/usr/local/cuda/lib64::/home/hadoop/whx/CuDNN/lib64:/home/hadoop/whx/OpenBLAS/lib; export SPARK_HOME=/home/hadoop/whx/spark;GLOG_minloglevel=1 /home/hadoop/whx/CaffeOnSpark/caffe-grid/src/test/python/PythonTest.sh SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/home/hadoop/whx/CaffeOnSpark/caffe-grid/target/caffe-grid-0.1-SNAPSHOT-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/home/hadoop/whx/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 16/11/27 19:49:35 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable testTrain (main.PythonApiTest) ... 16/11/27 19:49:56 WARN Executor: 1 block locks were not released by TID = 10: [rdd_16_0] 16/11/27 19:49:56 WARN Config: both -test and -features are found, we will do test only (as it is latest), disabling feature mode. 16/11/27 19:50:00 WARN Executor: 1 block locks were not released by TID = 13: [rdd_40_0] 16/11/27 19:50:00 WARN Executor: 1 block locks were not released by TID = 16: [rdd_40_0] ok testTrainWithValidation (main.PythonApiTest) ... 16/11/27 19:50:01 WARN Executor: 1 block locks were not released by TID = 24: [rdd_60_0] 16/11/27 19:50:07 WARN CaffeProcessor: Some transformer threads haven't been terminated yet 16/11/27 19:50:08 WARN Executor: 1 block locks were not released by TID = 51: [rdd_495_0] +------------+------------+ | accuracy| loss| +------------+------------+ |[0.70999974]| [1.5267749]| | [0.8000009]|[0.62335044]| +------------+------------+ only showing top 2 rows
ok
guody5
commented
8 years ago The code I remove is:
chris0927
commented
8 years ago I have this problem, but I don`t know why?
guody5
commented
8 years ago @chris0927 do you have any better way to solve it?
chris0927
commented
8 years ago @guody5 sorry ,no
mriduljain
commented
8 years ago This appears to be spark/scala version issue. I use spark 1.6 and scala 2.10.5. I am not sure which version of scala you should use with spark 2.0, though I will give it a try once I upgrade my box. Please check your warnings about the same.
mriduljain
commented
8 years ago Looks like this line causes the issue: https://github.com/yahoo/CaffeOnSpark/blob/master/caffe-grid/src/main/scala/com/yahoo/ml/caffe/CaffeOnSpark.scala#L96
Could you check this on standalone spark/scala shell, to see if it works? I guess it doesn't have dependency on anything else.
guody5
commented
8 years ago @mriduljain
I check this line on standalone spark/scala shell,and it doesn't work:
The error is:floatarray2doubleUDF: org.apache.spark.sql.expressions.UserDefinedFunction = UserDefinedFunction(
But I have checked this example,and it works. https://github.com/yahoo/CaffeOnSpark/wiki/GetStarted_standalone
mriduljain
commented
8 years ago Sorry I am not clear yet if your problem is solved or not
JJJzheng
commented
7 years ago I got the same problem. i check the pom.xml in caffe-grid and found that my set up is not right. so I make this: maven 3.5.0 to 3.2.5 (with the problem of maven plugin : scala-test-plguin maven-scala-plugin) scala 2.11.10 to 2.11.7 spark 2.0.0 to 1.6.0 (with the problem of sqlcontext) after that, make build again.
When I make build,there are something wrong. Could someone help me?
2016-11-25 19:25:01,459 [Thread-2] INFO util.ShutdownHookManager - Shutdown hook called 2016-11-25 19:25:01,459 [Thread-2] INFO util.ShutdownHookManager - Deleting directory /tmp/spark-21b654cc-2608-4beb-afff-569cac5e5b1e [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] caffe .............................................. SUCCESS [ 0.001 s] [INFO] caffe-distri ....................................... SUCCESS [13:08 min] [INFO] caffe-grid ......................................... FAILURE [ 5.962 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 13:14 min [INFO] Finished at: 2016-11-25T19:25:01+08:00 [INFO] Final Memory: 59M/708M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.scalatest:scalatest-maven-plugin:1.0:test (test) on project caffe-grid: There are test failures -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :caffe-grid