Mac OS X build instructions are not right for protobufs

[ezyang]

Steps to reproduce:

Follow the instructions at https://caffe2.ai/docs/getting-started.html?platform=mac&configuration=compile for Mac OS X build from source. The most important things:

1. brew install protobufs. At the time of writing the protobufs that brew installs by default is 3.3.0
2. Run conda install -y --channel https://conda.anaconda.org/conda-forge gflags glog numpy protobuf=3.2.0 (as specified in the docs)
3. cmake -DUSE_CUDA=OFF .. and make

Expected result: Successful build

Actual result: You get the error:

In file included from /Users/ezyang/Dev/caffe2/build/caffe/proto/caffe.pb.cc:5:
/Users/ezyang/Dev/caffe2/build/caffe/proto/caffe.pb.h:17:2: error: This file was
      generated by an older version of protoc which is
#error This file was generated by an older version of protoc which is
 ^
/Users/ezyang/Dev/caffe2/build/caffe/proto/caffe.pb.h:18:2: error: incompatible
      with your Protocol Buffer headers. Please
#error incompatible with your Protocol Buffer headers.  Please
 ^
/Users/ezyang/Dev/caffe2/build/caffe/proto/caffe.pb.h:19:2: error: regenerate
      this file with a newer version of protoc.
#error regenerate this file with a newer version of protoc.
 ^
3 errors generated.

This is actually the same error as https://github.com/caffe2/caffe2/issues/474 but I want to reopen the issue because there is legitimately a problem with caffe2 and its build documentation. I can see several problems:

1. The documentation on the build page suggests installing protobufs with both brew and Anaconda. There are multiple levels of wrongness here. First, brew installs 3.3.0, and Anaconda installs 3.2.0. That can't be right. Second, if you run both instructions, you end up with TWO copies of protobufs; only one is necessary. Finally, these aren't even the right version of protobufs for HEAD: HEAD needs 3.1.0 as of https://github.com/caffe2/caffe2/commit/d9e90a968d29116d9a60e61f7f358de7aef84498

2. cmake should test the version of protobufs, and error if it's the wrong version. That will give a lot more interpretable error, because you will be able to see what the wrong version is.

3. I haven't tried reproducing this carefully, but it seems that no matter how you setup your PATH, cmake will always end up using the brew installed protobufs rather than the Anaconda one, even if you have Anaconda earlier in your PATH. I'm not too sure what is going on here.

My suggestion is that we remove brew install protobufs from the instructions, suggest protobufs be installed with Anaconda, fix the version specified in the docs, and figure out why Anaconda isn't being picked up when there is both a brew and Anaconda version of protobufs. I'd be happy to contribute doc patches but I couldn't find the website sources.

[aaronmarkham]

Docs are in the gh-pages branch. Glad to have your help diagnosing as the protobuf problems seem to keep changing as versions get updated, plus there's inconsistencies in the conda packages. Generally you'll want system protobuf and python protobuf to match. Try downgrading the brew install to 3.2.0 unless you can find a conda package that is 3.3.0 to match the brew installation.

You can also try out the latest wheel and see if that works for you. https://caffe2.ai/docs/getting-started.html?platform=mac&configuration=prebuilt

[ezyang]

OK, I've diagnosed why we're picking up the wrong version of protobuf on a OS X system with Brew. Look at this:

-- Found gflags  (include: /usr/local/include, library: /usr/local/lib/libgflags.dylib)
-- Found system gflags install.
-- Found glog    (include: /usr/local/include, library: /usr/local/lib/libglog.dylib)
-- Found system glog install.

This is disaster! What this means is that we will add /usr/local/include to our include path at some arbitrary point. If we are unlucky (as is the case here), /usr/local/include may have higher precedence over my local Anaconda include directory, so even though CMake explicitly configured Anaconda in one directory, /usr/local/include ends up clobbering the local definition.

Here's a relevant StackOverflow question: https://stackoverflow.com/questions/7752609/cmake-ordering-of-include-directories-how-to-mix-system-and-user-based-includ (except that the problem in their case is the other way around.)

[rodamn]

FIX: Running conda install -y --channel https://conda.anaconda.org/conda-forge gflags glog numpy protobuf=3.3.0 will install a version of protobuf that matches the version installed by brew. This will allow the sudo make install step to properly compile (if it doesn't, from what I can tell you need to re-run cmake -DUSE_CUDA=OFF ...

HTH

[Heightened]

The documentation for Ubuntu is off as well. It lists 3.2.0 while it should be 3.1.0.

In addition, the required protobuf release should be installed manually since using 'sudo apt-get install ...' will install 2.6.x, which will lead to a version mismatch.

[NickGeneva]

Wanted to confirm that I also had a similar problem on Ubuntu and its fixed by updating the protobuf compiler, protoc, to match the Python version installed by Conda which was mentioned in the comment above. My steps to fix this issue follow:

1. Check the protobuf package version for Python via:

   conda list protobuf

2. Check protobuf-compiler version installed on Ubuntu:

   protoc --version

3. Odds are they are different if you used sudo apt-get protobuf-compiler. Head on over to the protobuf releases on Github and download the precompiled protoc binary (e.g. protoc-3.5.0-linux-x86_64.zip) for the version that corresponds to the Python version you installed with Conda.
4. Extract and replace your local protoc files with the updated version (found in usr/bin and usr/include/google). For example inside your extracted protoc folder I used the following:

   sudo cp bin/protoc /usr/bin/
   sudo rm -r /usr/include/google/protobuf/
   sudo cp -r include/google/protobuf/ /usr/include/google/protobuf/

5. Verify that your protobuf-compiler is indeed updated with the command in step 2.

   protoc --version

6. Re-clone and remake caffe2 from the beginning.

   git clone --recursive https://github.com/caffe2/caffe2.git && cd caffe2
   make && cd build && sudo make install

Hopefully this will save someone some time.

[cheetah132]

@AbsoluteStratos save my life.

In Mac OSX, I need little bit different command, because nowadays protobuf for caffe2 changed to ver 3.1 and the path of protobuf in OSX is different from that ubuntu.

I firstly do

brew install protobuf@3.1
conda install -y --channel https://conda.anaconda.org/conda-forge  \
future \
gflags \
glog  \
numpy \
protobuf=3.1.0 \
six

and secondly

cd /usr/local/Cellar/protobuf@3.1/3.1.0
sudo cp bin/protoc /usr/local/bin/
sudo rm -r /usr/local/include/google/protobuf/
sudo cp -r include/google/protobuf/ /usr/local/include/google/protobuf/
sudo cp -r lib /usr/local/

below is for Korean who don't want to translate English.

이 글을 읽고 계신다면 분명 까페2를 설치하는데 애를 먹고 계신 것이겠죠. 까페2 공식 홈페이지에 나온대로 설치를 하면 cmake가 brew나 conda가 설치한 protobuf가 아닌 시스템에 기본으로 내장된 protobuf를 사용해서 문제가 발생하실 겁니다. (현재 기준으로 까페2 공식 홈페이지에서는 3.1을 쓰라고 하고 있습니다.)

만약 우분투를 쓰신다면 제 윗분이 하신 방식대로 하면 되고요. 맥을 쓰신다면 제가 써놓은 커맨드를 입력하시면 될 겁니다.

[thnkim]

The comment of @cheetah132 was very helpful! One more thing I found (on Mac OSX) is, when I installed conda install libgcc, cmake selects anaconda's c++ compiler and causes weird syntax errors while building caffe2.

So I removed conda, installed again, and succeeded in building caffe2.

[Angel-Jia]

I post the wrong message. This comment should be in #1684