building drake_cmake_installed with clang++ causes a 'google::protobuf::FatalException' at runtime

[dmsj]

I was debugging linker errors on a different project which uses drake as an external and discovered that building with clang++ solves the linker issues (will submit this as a separate issue once I understand it better) but introduces a libprotobuf exception at runtime.

This is on a fresh install of Ubuntu 16.04. This problem does not occur on macOS.

to reproduce (following the README.md instructions in drake_cmake_installed): mkdir build cd build cmake -Ddrake_DIR=$HOME/drake/build/lib/cmake/drake -DCMAKE_CXX_COMPILER=/usr/bin/clang++ .. which produces the following output:

-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is Clang 3.8.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/clang++
-- Check for working CXX compiler: /usr/bin/clang++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Could NOT find GTest (missing:  GTEST_LIBRARY GTEST_INCLUDE_DIR GTEST_MAIN_LIBRARY) 
-- Could NOT find GTest (missing:  GTEST_LIBRARY GTEST_INCLUDE_DIR GTEST_MAIN_LIBRARY) 
-- Configuring done
-- Generating done
-- Build files have been written to: /home/momap/exp/sham/drake_cmake_installed/build

then make -j and cd src/particles followed by ./uniformly_accelerated_particle_demo which results in:

[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/descriptor_database.cc:58] File already exists in database: google/protobuf/any.proto
[libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/descriptor.cc:1394] CHECK failed: generated_database_->Add(encoded_file_descriptor, size): 
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  CHECK failed: generated_database_->Add(encoded_file_descriptor, size): 
Aborted (core dumped)

The whole pipeline works fine if you omit the -DCMAKE_CXX_COMPILER=/usr/bin/clang++ and I wouldn't care - except my other project won't link if I don't use clang.

An example failure in my custom project:

[ 81%] Linking CXX executable test_ik
libdracula.a(constraint_solver.cc.o): In function `ConstraintSolver::getJointNames[abi:cxx11]()':
constraint_solver.cc:(.text+0x2291): undefined reference to `RigidBodyTree<double>::get_position_name(int) const'
collect2: error: ld returned 1 exit status

Thanks for the help! In the meantime I'll try and get the other project building without clang.

[stonier]

Can you confirm whether the installed bazel you have is gcc or clang?

(If you got the binaries via wget from the server, they will have been built with gcc, if instead you just did a bazel run //:install, it will be clang). Not an ideal situation, but we're looking into that).

[dmsj]

I believe it was originally clang. I was able to fix my libprotobuf error by first building drake with bazel build --compiler=gcc-5 --config snopt //... and then compiling my cmake project as usual.

However, I'm now running into segmentation faults in my external project, which are not present in the macOS build which uses clang.

[dmsj]

results so far (build / run) drake-clang & drake_cmake_installed-clang : SUCCESS / FAIL (libprotobuf) drake-clang & drake_cmake_installed_gcc : SUCCESS / SUCCESS drake-gcc & drake_cmake_installed_gcc : SUCCESS / SUCCESS drake-clang & my_project-clang : SUCCESS / FAIL (libprotobuf) drake-clang & my_project-gcc : FAIL ( won't link - can't find the drake symbols) drake-gcc & my_project-gcc : SUCCESS / FAIL (segmentation fault - though this looks fixable)

[dmsj]

to clarify drake-gcc & my_project-gcc - I can build drake examples and run them, so from your standpoint this is a SUCCESS / SUCCESS

[stonier]

The drake-clang & my_project-clang is probably where we should focus. Is my_project_clang also linking to libprotobuf by another path (non-drake, e.g. directly or transitively via another library)?

[dmsj]

Not unless drake_cmake_installed-clang is somewhere. That fails in the same way as my project with the libprotobuf error. I would have expected clang / clang to work fine - and it does on macOS.

[stonier]

drake-clang & drake_cmake_installed-clang is actually working for me. I am using

# drake
bazel run --config snopt //:install -- /opt/drake

# drake_cmake_installed
cd build && cmake -Ddrake_DIR=/opt/drake/lib/cmake/drake -DCMAKE_CXX_COMPILER=/usr/bin/clang++-4.0  .. && make VERBOSE=1 -j5

when I build the actual project (you used /usr/bin/clang++, which I do not have).

[dmsj]

hmm.. I just tried that with clang-4.0 - and it still failed. Both for my project and for drake_cmake_installed.

That seems to point towards a problem with my particular installation.

[dmsj]

@stonier are you building from the precompiled binaries?

[stonier]

For this, I was testing with bazel run install method as shown above, so I could switch it from gcc to clang as you have done.

[dmsj]

I did verify that the gcc method (both bazel and precompiled) fails due to (I believe) my project also including Eigen. As soon as drake calls a function which uses Eigen, it seg faults.

I'm testing bazel-clang on a fresh ubuntu install now.

[stonier]

Are you making sure you find_package(Drake CONFIG REQUIRED) before anything else (if your project or another down the line is doing find_package(EIGEN ...) before Drake it will find the system Eigen which is incompatible with Drake's Eigen, but the other way around is fine.

Some notes on this in the PCL example README. I should ass some additional notes to the drake_cmake_installed README.

[dmsj]

I was able to reproduce the libprotobuf error on a fresh, completely clean ubuntu 16.04 install trying to do drake-clang and drake_cmake_installed-clang

# drake
bazel build --compiler=clang-4.0 --config snopt //...
mkdir build
bazel run --compiler=clang-4.0 --config snopt //:install -- $PWD/build

# drake_cmake_installed
mkdir build; cd build
cmake -Ddrake_DIR="$HOME/drake/build/lib/cmake/drake" -DCMAKE_CXX_COMPILER="/usr/bin/clang++-4.0"  ..
make -j

~/sham/drake_cmake_installed/build/src/particles$ ./uniformly_accelerated_particle_demo 
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/descriptor_database.cc:58] File already exists in database: google/protobuf/any.proto
[libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/descriptor.cc:1394] CHECK failed: generated_database_->Add(encoded_file_descriptor, size): 
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  CHECK failed: generated_database_->Add(encoded_file_descriptor, size): 
Aborted (core dumped)

maybe note-worth - the CXX flag only specifies the CXX complier, while the C complier is still GNU

The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is Clang 4.0.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/clang++-4.0
-- Check for working CXX compiler: /usr/bin/clang++-4.0 -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Could NOT find GTest (missing:  GTEST_LIBRARY GTEST_INCLUDE_DIR GTEST_MAIN_LIBRARY) 
-- Could NOT find GTest (missing:  GTEST_LIBRARY GTEST_INCLUDE_DIR GTEST_MAIN_LIBRARY) 
-- Configuring done
-- Generating done

[dmsj]

I did, however, fix the GNU gcc error. It turns out that one of our externals had packaged the eigen includes as well - by removing that, everything works.

[EricCousineau-TRI]

Testing the above instructions, but for Bionic with clang-6.0 (specified for both CC and CXX), and using environment variables since we've stopped using CROSSTOOL in Drake.

$ bash-isolate  # See https://bit.ly/2FtluQs
$ export CC=clang-6.0 CXX=clang++-6.0 SNOPT_PATH=git

$ cd drake
$ git rev-parse --short HEAD
a57d3198a
$ drake_dir=${PWD}/build/py2
$ bazel run --config=snopt //:install -- ${drake_dir}

$ cd .../drake-external-examples/drake_cmake_installed
$ git rev-parse --short HEAD
43ff258
$ mkdir build && cd build
$ cmake -DCMAKE_PREFIX_PATH=${drake_dir} ..
$ make -j
$ ctest -V -R
# No failures

Since this succeeds, I'm closing the issue for now. However, @dmsj, if you still have this as an active pain point, I can re-open the issue. (Also, are you still on Xenail, or have y'all migrated to Bionic?)