yahoo / CaffeOnSpark

Distributed deep learning on Hadoop and Spark clusters.
Apache License 2.0
1.27k stars 358 forks source link

error while building #213

Open githubier opened 7 years ago

githubier commented 7 years ago

while I make build the CaffeOnSpark, I meet the problem below: cd caffe-public; make proto; make -j4 -e distribute; cd .. make[1]: Entering directory /home/master/Downloads/spark_sdk/CaffeOnSpark/caffe-public' make[1]: Nothing to be done forproto'. make[1]: Leaving directory /home/master/Downloads/spark_sdk/CaffeOnSpark/caffe-public' make[1]: Entering directory/home/master/Downloads/spark_sdk/CaffeOnSpark/caffe-public' CXX/LD -o .build_release/examples/cpp_classification/classification.bin CXX/LD -o python/caffe/_caffe.so python/caffe/_caffe.cpp python/caffe/_caffe.cpp:1:52: fatal error: Python.h: No such file or directory

include // NOLINT(build/include_alpha)

                                                ^

compilation terminated. make[1]: [python/caffe/_caffe.so] Error 1 make[1]: Waiting for unfinished jobs.... make[1]: Leaving directory `/home/master/Downloads/spark_sdk/CaffeOnSpark/caffe-public' export LD_LIBRARY_PATH="/home/master/Downloads/spark_sdk/CaffeOnSpark/caffe-public/distribute/lib:/home/master/Downloads/spark_sdk/CaffeOnSpark/caffe-distri/distribute/lib:~Downloads/spark_sdk/anaconda2/lib:/home/master/Downloads/spark_sdk/CaffeOnSpark/caffe-public/distribute/lib:/home/master/Downloads/spark_sdk/CaffeOnSpark/caffe-distri/distribute/lib:~Downloads/spark_sdk/anaconda2/lib::/home/master/Downloads/spark_sdk/CaffeOnSpark/caffe-public/distribute/lib:/home/master/Downloads/spark_sdk/CaffeOnSpark/caffe-distri/distribute/lib:/usr/lib64:/lib64 "; GLOG_minloglevel=1 mvn -B package -DskipTests [INFO] Scanning for projects... [WARNING] [WARNING] Some problems were encountered while building the effective model for com.yahoo.ml:caffe-grid:jar:0.1-SNAPSHOT [WARNING] The expression ${version} is deprecated. Please use ${project.version} instead. [WARNING] [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build. [WARNING] [WARNING] For this reason, future Maven versions might no longer support building such malformed projects. [WARNING] [INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] [INFO] caffe [INFO] caffe-distri [INFO] caffe-grid [INFO]
[INFO] ------------------------------------------------------------------------ [INFO] Building caffe 0.1-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO]
[INFO] ------------------------------------------------------------------------ [INFO] Building caffe-distri 0.1-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-antrun-plugin:1.7:run (proto) @ caffe-distri --- [INFO] Executing tasks

protoc: [exec] make[1]: No rule to make target ../caffe-public/distribute/proto/caffe.proto', needed bysrc/main/java/caffe/Caffe.java'. make[1]: Entering directory /home/master/Downloads/spark_sdk/CaffeOnSpark/caffe-distri' [exec] make[1]: Leaving directory/home/maste Stop. [exec] r/Downloads/spark_sdk/CaffeOnSpark/caffe-distri' [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] caffe .............................................. SUCCESS [ 0.003 s] [INFO] caffe-distri ....................................... FAILURE [ 1.205 s] [INFO] caffe-grid ......................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1.534 s [INFO] Finished at: 2016-12-26T06:06:34-05:00 [INFO] Final Memory: 8M/102M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (proto) on project caffe-distri: An Ant BuildException has occured: exec returned: 2 [ERROR] around Ant part ...... @ 5:111 in /home/master/Downloads/spark_sdk/CaffeOnSpark/caffe-distri/target/antrun/build-protoc.xml [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :caffe-distri make: [build] Error 1

and my protobuf is 2.5.0, i don't know how to solve it, so can you help me?

junshi15 commented 7 years ago

It looks like you had problems with python, which failed caffe compilation. Were you able to compile BVLC caffe? Your python setting may not be correct. The proto error later on is probably due to failure in caffe.

githubier commented 7 years ago

thank you for your help, I have resolved this problem. However, when I build caffe on another computer, I met a new problem below .buildrelease/src/caffe/proto/caffe.pb.h:24303:42: error: ‘kEmptyString’ is not a member of ‘google::protobuf::internal’ name = const_cast< ::std::string>(&::google::protobuf::internal::kEmptyString); ^ .build_release/src/caffe/proto/caffe.pb.h: In member function ‘void caffe::CoSDataParameter::clear_source()’: .buildrelease/src/caffe/proto/caffe.pb.h:24559:19: error: ‘kEmptyString’ is not a member of ‘google::protobuf::internal’ if (source != &::google::protobuf::internal::kEmptyString) { ^ .build_release/src/caffe/proto/caffe.pb.h: In member function ‘void caffe::CoSDataParameter::set_source(const string&)’: .buildrelease/src/caffe/proto/caffe.pb.h:24569:19: error: ‘kEmptyString’ is not a member of ‘google::protobuf::internal’ if (source == &::google::protobuf::internal::kEmptyString) { ^ .build_release/src/caffe/proto/caffe.pb.h: In member function ‘void caffe::CoSDataParameter::set_source(const char)’: .buildrelease/src/caffe/proto/caffe.pb.h:24576:19: error: ‘kEmptyString’ is not a member of ‘google::protobuf::internal’ if (source == &::google::protobuf::internal::kEmptyString) { ^ .build_release/src/caffe/proto/caffe.pb.h: In member function ‘void caffe::CoSDataParameter::set_source(const char, size_t)’: .buildrelease/src/caffe/proto/caffe.pb.h:24583:19: error: ‘kEmptyString’ is not a member of ‘google::protobuf::internal’ if (source == &::google::protobuf::internal::kEmptyString) { ^ .build_release/src/caffe/proto/caffe.pb.h: In member function ‘std::string caffe::CoSDataParameter::mutable_source()’: .buildrelease/src/caffe/proto/caffe.pb.h:24590:19: error: ‘kEmptyString’ is not a member of ‘google::protobuf::internal’ if (source == &::google::protobuf::internal::kEmptyString) { ^ .build_release/src/caffe/proto/caffe.pb.h: In member function ‘std::string caffe::CoSDataParameter::release_source()’: .buildrelease/src/caffe/proto/caffe.pb.h:24597:19: error: ‘kEmptyString’ is not a member of ‘google::protobuf::internal’ if (source == &::google::protobuf::internal::kEmptyString) { ^ .buildrelease/src/caffe/proto/caffe.pb.h:24601:44: error: ‘kEmptyString’ is not a member of ‘google::protobuf::internal’ source = const_cast< ::std::string>(&::google::protobuf::internal::kEmptyString); ^ .build_release/src/caffe/proto/caffe.pb.h: In member function ‘void caffe::CoSDataParameter::set_allocated_source(std::string)’: .buildrelease/src/caffe/proto/caffe.pb.h:24606:19: error: ‘kEmptyString’ is not a member of ‘google::protobuf::internal’ if (source != &::google::protobuf::internal::kEmptyString) { ^ .buildrelease/src/caffe/proto/caffe.pb.h:24614:44: error: ‘kEmptyString’ is not a member of ‘google::protobuf::internal’ source = const_cast< ::std::string>(&::google::protobuf::internal::kEmptyString); ^ make[1]: *** [.build_release/src/caffe/layer.o] Error 1 make[1]: Leaving directory `/home/master/Downloads/spark_sdk/CaffeOnSpark/caffe-public' export LD_LIBRARY_PATH="/home/master/Downloads/spark_sdk/CaffeOnSpark/caffe-public/distribute/lib:/home/master/Downloads/spark_sdk/CaffeOnSpark/caffe-distri/distribute/lib:/home/master/Downloads/spark_sdk/anaconda2/lib::/home/master/Downloads/spark_sdk/CaffeOnSpark/caffe-public/distribute/lib:/home/master/Downloads/spark_sdk/CaffeOnSpark/caffe-distri/distribute/lib:/usr/lib64:/lib64 "; GLOG_minloglevel=1 mvn -B package -DskipTests [INFO] Scanning for projects... [WARNING] [WARNING] Some problems were encountered while building the effective model for com.yahoo.ml:caffe-grid:jar:0.1-SNAPSHOT [WARNING] The expression ${version} is deprecated. Please use ${project.version} instead. [WARNING] [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build. [WARNING] [WARNING] For this reason, future Maven versions might no longer support building such malformed projects. [WARNING] [INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] [INFO] caffe [INFO] caffe-distri [INFO] caffe-grid [INFO]
[INFO] ------------------------------------------------------------------------ [INFO] Building caffe 0.1-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO]
[INFO] ------------------------------------------------------------------------ [INFO] Building caffe-distri 0.1-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-antrun-plugin:1.7:run (proto) @ caffe-distri --- [INFO] Executing tasks

protoc: [exec] make[1]: Entering directory /home/master/Downloads/spark_sdk/CaffeOnSpark/caffe-distri' [exec] make[1]: Leaving directory/home/master/Downloads/spark_sdk/CaffeOnSpark/caffe-distri' [exec] make[1]: No rule to make target ../caffe-public/distribute/proto/caffe.proto', needed bysrc/main/java/caffe/Caffe.java'. Stop. [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] caffe .............................................. SUCCESS [ 0.002 s] [INFO] caffe-distri ....................................... FAILURE [ 0.944 s] [INFO] caffe-grid ......................................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1.191 s [INFO] Finished at: 2016-12-27T08:27:36-05:00 [INFO] Final Memory: 8M/150M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (proto) on project caffe-distri: An Ant BuildException has occured: exec returned: 2 [ERROR] around Ant part ...... @ 5:111 in /home/master/Downloads/spark_sdk/CaffeOnSpark/caffe-distri/target/antrun/build-protoc.xml [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :caffe-distri make: [build] Error 1

also, the protoc is 2.5.0

junshi15 commented 7 years ago

what do you get from "protoc --version"? It looks like your protobuf version is incorrect.

githubier commented 7 years ago

yes, my protoc version is 2.5.0. There is two protoc in my computer, which in /usr/local/bin is 2.6.1,and after I remove it, the verision of my protoc is 2.5.0. So it is why I confused.

junshi15 commented 7 years ago

I wonder if you built it with 2.6 previously. Maybe you can do a "make clean" or "mvn clean" to start a clean build.

githubier commented 7 years ago

Ah, thank you for your suggestion, but I have "make clean",and it's no help for this issue.....