crosstool-NG compiled libraries compatibility problem

[MrMYHuang]

As I posted an issue at here crosstool-NG libraries are incompatible with CentOS 7 GNU linker!?, I find crosstool-NG compiler and CentOS 7.4.1708 native g++ compiler generate different symbols in some cases and fail to link to each other compiled libaries!

I create a minimal program to reproduce the problem: https://github.com/MrMYHuang/HelloWorld Please install Anaconda 3 5.0.1 64-bit on /opt/anaconda3 on CentOS 7.4.1708 64-bit first and install crosstool-NG compiler and libprotobuf:

/opt/anaconda3/bin/conda install gxx_linux-64=7.2.0=25
/opt/anaconda3/bin/conda install libprotobuf=3.4.1=h5b8497f_0

Then, the following shows that the crosstool-NG compiled libprotobuf can be linked with crosstool-NG compiled libHelloWorld:

$ git clone https://github.com/MrMYHuang/HelloWorld.git
$ cd HelloWorld
$ /opt/anaconda3/bin/x86_64-conda_cos6-linux-gnu-g++ -shared -fPIC -I/opt/anaconda3/include HelloWorld.cpp -o libHelloWorld.so
$ # Good to link.
$ /opt/anaconda3/bin/x86_64-conda_cos6-linux-gnu-g++ -L/opt/anaconda3/lib -lprotobuf -L. -lHelloWorld  Main.cpp -o Main
$ LD_LIBRARY_PATH=. ./Main
# You will see "Hello world."
Hello world.
$ nm -S libHelloWorld.so |grep fixed
                 U _ZN6google8protobuf8internal26fixed_address_empty_stringB5cxx11E

However, if we compile libHelloWorld with CentOS 7 native compiler, it will fail to link to libprotobuf:

$ g++ -shared -fPIC -I/opt/anaconda3/include HelloWorld.cpp -o libHelloWorld.so
$ # Fail to link.
$ g++ -L/opt/anaconda3/lib -lprotobuf -L. -lHelloWorld  Main.cpp -o Main
./libHelloWorld.so: undefined reference to `google::protobuf::internal::fixed_address_empty_string'
collect2: error: ld returned 1 exit status
$ nm -S libHelloWorld.so |grep fixed
                 U _ZN6google8protobuf8internal26fixed_address_empty_stringE

Actually, nm -S libHelloWorld.so |grep fixed shows the compiled symbols of the variable fixed_address_empty_string are different!: _ZN6google8protobuf8internal26fixed_address_empty_stringB5cxx11E vs _ZN6google8protobuf8internal26fixed_address_empty_stringE

This incompatibility problem makes me fail to compile caffe on CentOS 7... Is there any solution?

[mingwandroid]

Yes. You need to use our compilers.

[MrMYHuang]

I updated my repo with a new test set: HelloWold2.h, HelloWorld2.cpp, HelloWorld2Imp.cpp and Main2.cpp. Please checkout from: https://github.com/MrMYHuang/HelloWorld

These test files are self contained and have no dependency with libprotobuf. The following test shows there are linking problems in each combination: native lib + crosstool-NG compiler, or crosstool-NG compiled lib + native compiler.

$ # Fail to link.
$ g++ -shared -fPIC HelloWorld2Imp.cpp -o libHelloWorld2Imp.so
$ nm -S libHelloWorld2Imp.so |grep fix
0000000000201031 0000000000000001 B _ZN8internal26fixed_address_empty_stringE
$ /opt/anaconda3/bin/x86_64-conda_cos6-linux-gnu-g++ -shared -fPIC HelloWorld2.cpp -o libHelloWorld2.so
$ nm -S libHelloWorld2.so |grep fix
                 U _ZN8internal26fixed_address_empty_stringB5cxx11E
$ /opt/anaconda3/bin/x86_64-conda_cos6-linux-gnu-g++ -L. -lHelloWorld2 -lHelloWorld2Imp  Main2.cpp -o Main2
./libHelloWorld2.so: undefined reference to `internal::fixed_address_empty_string[abi:cxx11]'
$ 
$ # Fail to link.
$ /opt/anaconda3/bin/x86_64-conda_cos6-linux-gnu-g++ -shared -fPIC HelloWorld2Imp.cpp -o libHelloWorld2Imp.so
$ nm -S libHelloWorld2Imp.so |grep fix
0000000000201038 0000000000000001 B _ZN8internal26fixed_address_empty_stringB5cxx11E
$ g++ -shared -fPIC HelloWorld2.cpp -o libHelloWorld2.so
$ nm -S libHelloWorld2.so |grep fix
                 U _ZN8internal26fixed_address_empty_stringE
$ g++ -L. -lHelloWorld2 -lHelloWorld2Imp  Main2.cpp -o Main2
./libHelloWorld2.so: undefined reference to `internal::fixed_address_empty_string'
collect2: error: ld returned 1 exit status
$ 
$ # Good to link
$ /opt/anaconda3/bin/x86_64-conda_cos6-linux-gnu-g++ -shared -fPIC HelloWorld2Imp.cpp -o libHelloWorld2Imp.so
$ /opt/anaconda3/bin/x86_64-conda_cos6-linux-gnu-g++ -shared -fPIC HelloWorld2.cpp -o libHelloWorld2.so
$ /opt/anaconda3/bin/x86_64-conda_cos6-linux-gnu-g++ -L. -lHelloWorld2 -lHelloWorld2Imp  Main2.cpp -o Main2
$ LD_LIBRARY_PATH=. ./Main2
Hello world.
$ 
$ # Good to link
$ g++ -shared -fPIC HelloWorld2Imp.cpp -o libHelloWorld2Imp.so
$ g++ -shared -fPIC HelloWorld2.cpp -o libHelloWorld2.so
$ g++ -L. -lHelloWorld2 -lHelloWorld2Imp  Main2.cpp -o Main2
$ LD_LIBRARY_PATH=. ./Main2
Hello world.
$ 

[mingwandroid]

Ah, ok, CentOS7 system compilers do not even support the new C++11 ABI since they are version 4.8.5 only.

You can read more about this at: https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html

If you want to do this mixing (but you should use our compilers instead, there are about 8 years of toolchain development work between the 4 series and 7 series of compilers) then you need to do the following:

/opt/anaconda3/bin/x86_64-conda_cos6-linux-gnu-g++ -shared -fPIC HelloWorld2Imp.cpp -o libHelloWorld2Imp.so -D_GLIBCXX_USE_CXX11_ABI=0

This small string optimization you are missing is just one example of why you should prefer newer toolchains. Without SSO, for a small std::string on the heap, two allocations are required, but now-a-days, only one is required.

[mingwandroid]

And just to be clear, we use the C++11 ABI in all of our C++ libraries. This is something that standards compliance forces. We also support C++14 and 17 and that's not possible without adopting the new ABI.

[MrMYHuang]

Thanks. I see.