search

BVLC / caffe

Caffe: a fast open framework for deep learning.
http://caffe.berkeleyvision.org/
Other
34.14k stars 18.68k forks source link

OSX10.10 matcaffe protobuf conflict Matlab and Google #3423

Closed DCurro closed 7 years ago

DCurro commented 8 years ago

I can compile caffe normally.

When I add this to the LIBRARY_DIRS, I get the error seen at the bottom:

/Applications/MATLAB_R2015b.app/bin/maci64

If I try to make matcaffe, it also doesn't build.

I read somewhere that Matlab now distributes their own protobuf and opencv, and that this is likely the conflict.

error:

Domenics-MacBook-Pro:caffe DCurro$ make -j8
PROTOC src/caffe/proto/caffe.proto
CXX src/caffe/blob.cpp
CXX src/caffe/data_reader.cpp
CXX src/caffe/data_transformer.cpp
CXX src/caffe/internal_thread.cpp
CXX src/caffe/layer.cpp
CXX src/caffe/layer_factory.cpp
CXX src/caffe/layers/absval_layer.cpp
CXX src/caffe/layers/accuracy_layer.cpp
CXX src/caffe/layers/argmax_layer.cpp
CXX src/caffe/layers/base_conv_layer.cpp
CXX src/caffe/layers/base_data_layer.cpp
CXX src/caffe/layers/batch_norm_layer.cpp
CXX src/caffe/layers/batch_reindex_layer.cpp
CXX src/caffe/layers/bnll_layer.cpp
CXX src/caffe/layers/concat_layer.cpp
CXX src/caffe/layers/contrastive_loss_layer.cpp
CXX src/caffe/layers/conv_layer.cpp
CXX src/caffe/layers/cudnn_conv_layer.cpp
CXX src/caffe/layers/cudnn_lcn_layer.cpp
CXX src/caffe/layers/cudnn_lrn_layer.cpp
CXX src/caffe/layers/cudnn_pooling_layer.cpp
CXX src/caffe/layers/cudnn_relu_layer.cpp
CXX src/caffe/layers/cudnn_sigmoid_layer.cpp
CXX src/caffe/layers/cudnn_softmax_layer.cpp
CXX src/caffe/layers/cudnn_tanh_layer.cpp
CXX src/caffe/layers/data_layer.cpp
CXX src/caffe/layers/deconv_layer.cpp
CXX src/caffe/layers/dropout_layer.cpp
CXX src/caffe/layers/dummy_data_layer.cpp
CXX src/caffe/layers/eltwise_layer.cpp
CXX src/caffe/layers/embed_layer.cpp
CXX src/caffe/layers/euclidean_loss_layer.cpp
CXX src/caffe/layers/exp_layer.cpp
CXX src/caffe/layers/filter_layer.cpp
CXX src/caffe/layers/flatten_layer.cpp
CXX src/caffe/layers/hdf5_data_layer.cpp
CXX src/caffe/layers/hdf5_output_layer.cpp
CXX src/caffe/layers/hinge_loss_layer.cpp
CXX src/caffe/layers/im2col_layer.cpp
CXX src/caffe/layers/image_data_layer.cpp
CXX src/caffe/layers/infogain_loss_layer.cpp
CXX src/caffe/layers/inner_product_layer.cpp
CXX src/caffe/layers/log_layer.cpp
CXX src/caffe/layers/loss_layer.cpp
CXX src/caffe/layers/lrn_layer.cpp
CXX src/caffe/layers/memory_data_layer.cpp
CXX src/caffe/layers/multinomial_logistic_loss_layer.cpp
CXX src/caffe/layers/mvn_layer.cpp
CXX src/caffe/layers/neuron_layer.cpp
CXX src/caffe/layers/pooling_layer.cpp
CXX src/caffe/layers/power_layer.cpp
CXX src/caffe/layers/prelu_layer.cpp
CXX src/caffe/layers/reduction_layer.cpp
CXX src/caffe/layers/relu_layer.cpp
CXX src/caffe/layers/reshape_layer.cpp
CXX src/caffe/layers/sigmoid_cross_entropy_loss_layer.cpp
CXX src/caffe/layers/sigmoid_layer.cpp
CXX src/caffe/layers/silence_layer.cpp
CXX src/caffe/layers/slice_layer.cpp
CXX src/caffe/layers/softmax_layer.cpp
CXX src/caffe/layers/softmax_loss_layer.cpp
CXX src/caffe/layers/split_layer.cpp
CXX src/caffe/layers/spp_layer.cpp
CXX src/caffe/layers/tanh_layer.cpp
CXX src/caffe/layers/threshold_layer.cpp
CXX src/caffe/layers/tile_layer.cpp
CXX src/caffe/layers/window_data_layer.cpp
CXX src/caffe/net.cpp
CXX src/caffe/parallel.cpp
CXX src/caffe/solver.cpp
CXX src/caffe/solvers/adadelta_solver.cpp
CXX src/caffe/solvers/adagrad_solver.cpp
CXX src/caffe/solvers/adam_solver.cpp
CXX src/caffe/solvers/nesterov_solver.cpp
CXX src/caffe/solvers/rmsprop_solver.cpp
CXX src/caffe/solvers/sgd_solver.cpp
CXX src/caffe/syncedmem.cpp
CXX src/caffe/util/benchmark.cpp
CXX src/caffe/util/blocking_queue.cpp
CXX src/caffe/util/cudnn.cpp
CXX src/caffe/util/db.cpp
CXX src/caffe/util/db_leveldb.cpp
CXX src/caffe/util/db_lmdb.cpp
CXX src/caffe/util/hdf5.cpp
CXX src/caffe/util/im2col.cpp
CXX src/caffe/util/insert_splits.cpp
CXX src/caffe/util/io.cpp
CXX src/caffe/util/math_functions.cpp
CXX src/caffe/util/signal_handler.cpp
CXX src/caffe/util/upgrade_proto.cpp
CXX tools/caffe.cpp
CXX tools/compute_image_mean.cpp
CXX tools/convert_imageset.cpp
CXX tools/device_query.cpp
CXX tools/extract_features.cpp
CXX tools/finetune_net.cpp
CXX tools/net_speed_benchmark.cpp
CXX tools/test_net.cpp
CXX tools/train_net.cpp
CXX tools/upgrade_net_proto_binary.cpp
src/caffe/util/blocking_queue.cpp:50:7: warning: unused typedef 'INVALID_REQUESTED_LOG_SEVERITY' [-Wunused-local-typedef]
      LOG_EVERY_N(INFO, 1000)<< log_on_wait;
      ^
/usr/local/include/glog/logging.h:917:30: note: expanded from macro 'LOG_EVERY_N'
                             INVALID_REQUESTED_LOG_SEVERITY);           \
                             ^
/usr/local/include/glog/logging.h:912:73: note: expanded from macro 'GOOGLE_GLOG_COMPILE_ASSERT'
  typedef google::glog_internal_namespace_::CompileAssert<(bool(expr))> msg[bool(expr) ? 1 : -1]
                                                                        ^
1 warning generated.
CXX tools/upgrade_net_proto_text.cpp
CXX tools/upgrade_solver_proto_text.cpp
CXX examples/cifar10/convert_cifar_data.cpp
CXX examples/cpp_classification/classification.cpp
CXX examples/mnist/convert_mnist_data.cpp
CXX examples/siamese/convert_mnist_siamese_data.cpp
CXX .build_release/src/caffe/proto/caffe.pb.cc
AR -o .build_release/lib/libcaffe.a
LD -o .build_release/lib/libcaffe.so
clang: warning: argument unused during compilation: '-pthread'
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_conv_layer.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_lcn_layer.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_lrn_layer.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_pooling_layer.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_relu_layer.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_sigmoid_layer.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_softmax_layer.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn_tanh_layer.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: .build_release/lib/libcaffe.a(cudnn.o) has no symbols
Undefined symbols for architecture x86_64:
  "google::protobuf::io::CodedOutputStream::WriteStringWithSizeToArray(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned char*)", referenced from:
      caffe::Datum::SerializeWithCachedSizesToArray(unsigned char*) const in caffe.pb.o
      caffe::FillerParameter::SerializeWithCachedSizesToArray(unsigned char*) const in caffe.pb.o
      caffe::NetParameter::SerializeWithCachedSizesToArray(unsigned char*) const in caffe.pb.o
      caffe::SolverParameter::SerializeWithCachedSizesToArray(unsigned char*) const in caffe.pb.o
      caffe::SolverState::SerializeWithCachedSizesToArray(unsigned char*) const in caffe.pb.o
      caffe::NetState::SerializeWithCachedSizesToArray(unsigned char*) const in caffe.pb.o
      caffe::NetStateRule::SerializeWithCachedSizesToArray(unsigned char*) const in caffe.pb.o
      ...
  "google::protobuf::internal::empty_string_", referenced from:
      caffe::Datum::SharedCtor() in caffe.pb.o
      caffe::Datum::MergeFrom(caffe::Datum const&) in caffe.pb.o
      caffe::Datum::~Datum() in caffe.pb.o
      caffe::Datum::SharedDtor() in caffe.pb.o
      caffe::Datum::Clear() in caffe.pb.o
      caffe::Datum::MergePartialFromCodedStream(google::protobuf::io::CodedInputStream*) in caffe.pb.o
      caffe::NetParameter::SharedCtor() in caffe.pb.o
      ...
  "google::protobuf::internal::WireFormatLite::WriteBytesMaybeAliased(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::io::CodedOutputStream*)", referenced from:
      caffe::Datum::SerializeWithCachedSizes(google::protobuf::io::CodedOutputStream*) const in caffe.pb.o
  "google::protobuf::internal::WireFormatLite::WriteStringMaybeAliased(int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, google::protobuf::io::CodedOutputStream*)", referenced from:
      caffe::FillerParameter::SerializeWithCachedSizes(google::protobuf::io::CodedOutputStream*) const in caffe.pb.o
      caffe::NetParameter::SerializeWithCachedSizes(google::protobuf::io::CodedOutputStream*) const in caffe.pb.o
      caffe::SolverParameter::SerializeWithCachedSizes(google::protobuf::io::CodedOutputStream*) const in caffe.pb.o
      caffe::SolverState::SerializeWithCachedSizes(google::protobuf::io::CodedOutputStream*) const in caffe.pb.o
      caffe::ParamSpec::SerializeWithCachedSizes(google::protobuf::io::CodedOutputStream*) const in caffe.pb.o
      caffe::LayerParameter::SerializeWithCachedSizes(google::protobuf::io::CodedOutputStream*) const in caffe.pb.o
      caffe::TransformationParameter::SerializeWithCachedSizes(google::protobuf::io::CodedOutputStream*) const in caffe.pb.o
      ...
  "google::protobuf::internal::InitEmptyString()", referenced from:
      caffe::Datum::SharedCtor() in caffe.pb.o
      caffe::FillerParameter::SharedCtor() in caffe.pb.o
      caffe::NetParameter::SharedCtor() in caffe.pb.o
      caffe::SolverParameter::SharedCtor() in caffe.pb.o
      caffe::SolverState::SharedCtor() in caffe.pb.o
      caffe::NetState::SharedCtor() in caffe.pb.o
      caffe::NetStateRule::SharedCtor() in caffe.pb.o
      ...
  "google::protobuf::internal::empty_string_once_init_", referenced from:
      caffe::Datum::SharedCtor() in caffe.pb.o
      caffe::FillerParameter::SharedCtor() in caffe.pb.o
      caffe::NetParameter::SharedCtor() in caffe.pb.o
      caffe::SolverParameter::SharedCtor() in caffe.pb.o
      caffe::SolverState::SharedCtor() in caffe.pb.o
      caffe::NetState::SharedCtor() in caffe.pb.o
      caffe::NetStateRule::SharedCtor() in caffe.pb.o
      ...
  "google::protobuf::io::CodedInputStream::BytesUntilTotalBytesLimit() const", referenced from:
      caffe::BlobProto::MergePartialFromCodedStream(google::protobuf::io::CodedInputStream*) in caffe.pb.o
      bool google::protobuf::internal::WireFormatLite::ReadPackedPrimitiveNoInline<float, (google::protobuf::internal::WireFormatLite::FieldType)2>(google::protobuf::io::CodedInputStream*, google::protobuf::RepeatedField<float>*) in caffe.pb.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [.build_release/lib/libcaffe.so] Error 1
ronghanghu commented 8 years ago

Yes, this is exactly the reason of the error. Matlab is now distributing its own OpenCV and Protobuf and causing conflicts.

DCurro commented 8 years ago

Is there any good resolution to this? How can I ignore the matlab versions, and use the one's from the tutorial?

ronghanghu commented 8 years ago

It is quite tricky, as the mex compiling tool always appends the Matlab's internal library path before LIBRARY_DIRS. I tried to temporarily removed these matlab libraries during compiling, and use DYLD_INSERT_LIBRARIES at runtime.

DCurro commented 8 years ago

Could you produce a step by step tutorial? This sounds a little intense for someone like myself.

RobSalzwedel commented 8 years ago

Having exactly the same problem, Haven't been able to resolve it, even after uninstalling Matlab completely I still get the same error.

DCurro commented 8 years ago

I will try again over the christmas break. If I find a solution I will post it here. Hopefully someone beats me to it.

phillipi commented 8 years ago

A hacky solution that worked for me was to temporarily remove matlab's version of libprotobuf, compile, then restore matlab's version (the same might work for the opencv libs):

mv /Applications/MATLAB_R2015a.app/bin/maci64/libprotobuf.dylib /Applications/MATLAB_R2015a.app/bin/maci64/libprotobuf.dylib_bk
mv /Applications/MATLAB_R2015a.app/bin/maci64/libprotobuf.8.dylib /Applications/MATLAB_R2015a.app/bin/maci64/libprotobuf.8.dylib_bk
make matcaffe
mv /Applications/MATLAB_R2015a.app/bin/maci64/libprotobuf.dylib_bk /Applications/MATLAB_R2015a.app/bin/maci64/libprotobuf.dylib
mv /Applications/MATLAB_R2015a.app/bin/maci64/libprotobuf.8.dylib_bk /Applications/MATLAB_R2015a.app/bin/maci64/libprotobuf.8.dylib

I had a similar error on a linux system. In that case, matcaffe did compile, but at runtime still linked against matlab's incompatible libprotobuf. The following invocation of matlab fixed this:

LD_PRELOAD=/usr/local/lib/libprotobuf.so.10 matlab

where the path specifies a compatible version of protobuf.

cdicle commented 8 years ago

Thanks @phillipi that solved my compilation problem. For reference to others, I am trying to compile on El Capitan for Matlab 2016a. I needed to rename cudnn library as well, so mine looked like 

mv /Applications/MATLAB_R2016a.app/bin/maci64/libprotobuf.dylib /Applications/MATLAB_R2016a.app/bin/maci64/libprotobuf.dylib_bk
mv /Applications/MATLAB_R2016a.app/bin/maci64/libprotobuf.8.dylib /Applications/MATLAB_R2016a.app/bin/maci64/libprotobuf.8.dylib_bk
mv /Applications/MATLAB_R2016a.app/bin/maci64/libcudnn.dylib /Applications/MATLAB_R2016a.app/bin/maci64/libcudnn.dylib_bk
mv /Applications/MATLAB_R2016a.app/bin/maci64/libcudnn.7.0.dylib /Applications/MATLAB_R2016a.app/bin/maci64/libcudnn.7.0.dylib_bk
make matcaffe
mv /Applications/MATLAB_R2016a.app/bin/maci64/libprotobuf.dylib_bk /Applications/MATLAB_R2016a.app/bin/maci64/libprotobuf.dylib
mv /Applications/MATLAB_R2016a.app/bin/maci64/libprotobuf.8.dylib_bk /Applications/MATLAB_R2016a.app/bin/maci64/libprotobuf.8.dylib
mv /Applications/MATLAB_R2016a.app/bin/maci64/libcudnn.dylib_bk /Applications/MATLAB_R2016a.app/bin/maci64/libcudnn.dylib
mv /Applications/MATLAB_R2016a.app/bin/maci64/libcudnn.7.0.dylib_bk /Applications/MATLAB_R2016a.app/bin/maci64/libcudnn.7.0.dylib
shelhamer commented 7 years ago

Closing as duplicate of #3005