jeremy-rifkin / cpptrace

Simple, portable, and self-contained stacktrace library for C++11 and newer
MIT License
565 stars 52 forks source link

Mysterious linkage error, hard to debug #25

Closed CharlesJQuarra closed 10 months ago

CharlesJQuarra commented 11 months ago

Hi,

I successfully tested yesterday cpptrace on a test app with cmake and FetchContent_MakeAvailable. I went ahead to integrate it into a more complex cmake project using exactly the same strategy, but I kept getting

undefined reference to `cpptrace::generate_trace(unsigned int)'
collect2: error: ld returned 1 exit status 

I tried literally everything I could think of to debug the issue: from adding the hardcoded path as target_link_directories and hardcoding the -lcpptrace with target_link_options, to moving it around, with no luck.

Needless to say, I checked the symbols are indeed present in the library:

$ nm  -C _deps/cpptrace-build/libcpptrace.so | grep generate_trace
0000000000002be0 t cpptrace::generate_trace(unsigned int)
00000000000027e0 t cpptrace::generate_trace(unsigned int) [clone .cold.46]
0000000000003f20 t cpptrace::detail::generate_trace(unsigned long)
000000000000287a t cpptrace::detail::generate_trace(unsigned long) [clone .cold.47]

I tried adding verbose options to the linker, but besides adding more contextual environment variables to the console output, there is really no hint on why it failed to find the linker reference

One of the libraries I'm depending was an older version of libtorch, and after reading a while, I noticed that since a few versions ago it started shipping with both pre-CXX11-ABI and CXX11-ABI builds, so just in case this was causing the incompatibility, I updated the libtorch version with one built with c++11 ABI, but the same problem keeps happening. In any case I tried this despite there is no hint the source of the linker failure was even remotely related to an ABI incompatibility, just because some people on the internet seemed to have similar mysterious undefined reference errors that were related to this.

I've tried to read more on how to debug linker errors, but it seems there is not a lot of available information short of becoming an amateur compiler developer myself. Do you have any better ideas how could I debug this error?

This is the linker command in question with lots of verbose output:

/usr/bin/g++-8 -std=c++17 -O3 -g -Wl,--no-as-needed -g  -rdynamic -DENABLE_DEBUG_MACROS -rdynamic --verbose CMakeFiles/test_ad_higher_order.dir/test_ad_higher_order.cpp.o -o test_ad_higher_order   -L/home/charlesq/Projects/NousFix/build/_deps/cpptrace-build  -Wl,-rpath,/home/charlesq/Projects/NousFix/build/_deps/cpptrace-build:/home/charlesq/Projects/libtorch/1.8.0/libtorch/lib:/usr/lib/cuda-11.2/lib64:/usr/lib/cuda-11.2/lib64/stubs -ldl -lrt -ltinfo -lpthread -lm -lunwind /home/charlesq/Projects/libtorch/1.8.0/libtorch/lib/libtorch.so /home/charlesq/Projects/libtorch/1.8.0/libtorch/lib/libc10.so /usr/lib/cuda-11.2/lib64/stubs/libcuda.so /usr/lib/cuda-11.2/lib64/libnvrtc.so /usr/lib/x86_64-linux-gnu/libnvToolsExt.so /usr/lib/cuda-11.2/lib64/libcudart.so /home/charlesq/Projects/libtorch/1.8.0/libtorch/lib/libc10_cuda.so ../lib/libgtest_maind.a ../lib/libgtestd.a -lpthread ../libs/fmt/libfmtd.a ../common/libnous-common.a /usr/lib/libafcpu.so.3.8.0 ../common/libnous-common.a /usr/lib/x86_64-linux-gnu/libopencv_stitching.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_aruco.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_bgsegm.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_bioinspired.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_ccalib.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_dnn_objdetect.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_dnn_superres.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_dpm.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_face.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_freetype.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_fuzzy.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_hdf.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_hfs.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_img_hash.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_line_descriptor.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_quality.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_reg.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_rgbd.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_saliency.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_shape.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_stereo.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_structured_light.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_superres.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_surface_matching.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_tracking.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_videostab.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_viz.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_xobjdetect.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_xphoto.so.4.2.0 -ldl -lrt -ltinfo -lpthread -lm -lunwind ../_deps/cpptrace-build/libcpptrace.so /home/charlesq/Projects/libtorch/1.8.0/libtorch/lib/libc10.so /usr/lib/cuda-11.2/lib64/stubs/libcuda.so /usr/lib/cuda-11.2/lib64/libnvrtc.so /home/charlesq/Projects/libtorch/1.8.0/libtorch/lib/libc10_cuda.so -Wl,--no-as-needed,"/home/charlesq/Projects/libtorch/1.8.0/libtorch/lib/libtorch_cpu.so" -Wl,--as-needed -Wl,--no-as-needed,"/home/charlesq/Projects/libtorch/1.8.0/libtorch/lib/libtorch_cuda.so" -Wl,--as-needed /home/charlesq/Projects/libtorch/1.8.0/libtorch/lib/libc10_cuda.so /home/charlesq/Projects/libtorch/1.8.0/libtorch/lib/libc10.so /usr/lib/cuda-11.2/lib64/libcudart.so /usr/lib/x86_64-linux-gnu/libnvToolsExt.so /usr/lib/cuda-11.2/lib64/libcufft.so /usr/lib/cuda-11.2/lib64/libcurand.so /usr/lib/cuda-11.2/lib64/libcublas.so /usr/lib/cuda-11.2/lib64/libcudnn.so -Wl,--no-as-needed,"/home/charlesq/Projects/libtorch/1.8.0/libtorch/lib/libtorch.so" -Wl,--as-needed -lpthread -ldl ../libs/fmt/libfmtd.a -lbfd -lpthread /usr/lib/x86_64-linux-gnu/libopencv_highgui.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_datasets.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_plot.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_text.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_dnn.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_ml.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_phase_unwrapping.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_optflow.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_ximgproc.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_video.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_videoio.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_imgcodecs.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_objdetect.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_calib3d.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_features2d.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_flann.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_photo.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_imgproc.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_core.so.4.2.0 
Using built-in specs.
COLLECT_GCC=/usr/bin/g++-8
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/8/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 8.4.0-3ubuntu2' --with-bugurl=file:///usr/share/doc/gcc-8/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-8 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 8.4.0 (Ubuntu 8.4.0-3ubuntu2) 
COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/8/:/usr/lib/gcc/x86_64-linux-gnu/8/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/8/:/usr/lib/gcc/x86_64-linux-gnu/
LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/8/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../../lib/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/8/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-std=c++17' '-O3' '-g' '-g' '-rdynamic' '-D' 'ENABLE_DEBUG_MACROS' '-rdynamic' '-v' '-o' 'test_ad_higher_order' '-L/home/charlesq/Projects/NousFix/build/_deps/cpptrace-build' '-shared-libgcc' '-mtune=generic' '-march=x86-64'
 /usr/lib/gcc/x86_64-linux-gnu/8/collect2 -plugin /usr/lib/gcc/x86_64-linux-gnu/8/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/8/lto-wrapper -plugin-opt=-fresolution=/tmp/ccjVKd3R.res -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc --build-id --eh-frame-hdr -m elf_x86_64 --hash-style=gnu --as-needed -export-dynamic -dynamic-linker /lib64/ld-linux-x86-64.so.2 -pie -z now -z relro -o test_ad_higher_order /usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/Scrt1.o /usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/8/crtbeginS.o -L/home/charlesq/Projects/NousFix/build/_deps/cpptrace-build -L/usr/lib/gcc/x86_64-linux-gnu/8 -L/usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/8/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/8/../../.. --no-as-needed CMakeFiles/test_ad_higher_order.dir/test_ad_higher_order.cpp.o -rpath /home/charlesq/Projects/NousFix/build/_deps/cpptrace-build:/home/charlesq/Projects/libtorch/1.8.0/libtorch/lib:/usr/lib/cuda-11.2/lib64:/usr/lib/cuda-11.2/lib64/stubs -ldl -lrt -ltinfo -lpthread -lunwind /home/charlesq/Projects/libtorch/1.8.0/libtorch/lib/libtorch.so /home/charlesq/Projects/libtorch/1.8.0/libtorch/lib/libc10.so /usr/lib/cuda-11.2/lib64/stubs/libcuda.so /usr/lib/cuda-11.2/lib64/libnvrtc.so /usr/lib/x86_64-linux-gnu/libnvToolsExt.so /usr/lib/cuda-11.2/lib64/libcudart.so /home/charlesq/Projects/libtorch/1.8.0/libtorch/lib/libc10_cuda.so ../lib/libgtest_maind.a ../lib/libgtestd.a -lpthread ../libs/fmt/libfmtd.a ../common/libnous-common.a /usr/lib/libafcpu.so.3.8.0 ../common/libnous-common.a /usr/lib/x86_64-linux-gnu/libopencv_stitching.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_aruco.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_bgsegm.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_bioinspired.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_ccalib.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_dnn_objdetect.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_dnn_superres.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_dpm.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_face.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_freetype.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_fuzzy.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_hdf.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_hfs.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_img_hash.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_line_descriptor.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_quality.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_reg.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_rgbd.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_saliency.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_shape.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_stereo.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_structured_light.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_superres.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_surface_matching.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_tracking.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_videostab.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_viz.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_xobjdetect.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_xphoto.so.4.2.0 -ldl -lrt -ltinfo -lpthread -lm -lunwind ../_deps/cpptrace-build/libcpptrace.so /home/charlesq/Projects/libtorch/1.8.0/libtorch/lib/libc10.so /usr/lib/cuda-11.2/lib64/stubs/libcuda.so /usr/lib/cuda-11.2/lib64/libnvrtc.so /home/charlesq/Projects/libtorch/1.8.0/libtorch/lib/libc10_cuda.so --no-as-needed /home/charlesq/Projects/libtorch/1.8.0/libtorch/lib/libtorch_cpu.so --as-needed --no-as-needed /home/charlesq/Projects/libtorch/1.8.0/libtorch/lib/libtorch_cuda.so --as-needed /home/charlesq/Projects/libtorch/1.8.0/libtorch/lib/libc10_cuda.so /home/charlesq/Projects/libtorch/1.8.0/libtorch/lib/libc10.so /usr/lib/cuda-11.2/lib64/libcudart.so /usr/lib/x86_64-linux-gnu/libnvToolsExt.so /usr/lib/cuda-11.2/lib64/libcufft.so /usr/lib/cuda-11.2/lib64/libcurand.so /usr/lib/cuda-11.2/lib64/libcublas.so /usr/lib/cuda-11.2/lib64/libcudnn.so --no-as-needed /home/charlesq/Projects/libtorch/1.8.0/libtorch/lib/libtorch.so --as-needed -lpthread -ldl ../libs/fmt/libfmtd.a -lbfd -lpthread /usr/lib/x86_64-linux-gnu/libopencv_highgui.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_datasets.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_plot.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_text.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_dnn.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_ml.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_phase_unwrapping.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_optflow.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_ximgproc.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_video.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_videoio.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_imgcodecs.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_objdetect.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_calib3d.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_features2d.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_flann.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_photo.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_imgproc.so.4.2.0 /usr/lib/x86_64-linux-gnu/libopencv_core.so.4.2.0 -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /usr/lib/gcc/x86_64-linux-gnu/8/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/8/../../../x86_64-linux-gnu/crtn.o
/usr/bin/ld: CMakeFiles/test_ad_higher_order.dir/test_ad_higher_order.cpp.o: in function `trace()':
/home/charlesq/Projects/NousFix/adcpp/test_ad_higher_order.cpp:23: undefined reference to `cpptrace::generate_trace(unsigned int)'
collect2: error: ld returned 1 exit status
jeremy-rifkin commented 11 months ago

Hello, thank you for the report!

The first thing would probably be to check if the library is being built correctly and if the symbol actually exists in the library, for example on my machine

$ find . -name libcpptrace.so
./libcpptrace.so
$ nm ./libcpptrace.so | grep generate_trace
0000000000057873 T _ZN8cpptrace14generate_traceEj

There's a good chance the symbol could be discarded if the compiler thinks it's not used, which I ran into briefly on windows. There's a fix in #24 (which is a work-in-progress but hopefully can be merged soon) but I can apply a hot-fix on the main branch if needed.

CharlesJQuarra commented 11 months ago

Hi Jeremy, it seems I updated the issue while you were replying, the updated message shows the symbols are indeed present

jeremy-rifkin commented 11 months ago

Oh it looks like it, thanks! This is then very puzzling 😄 The link order looks fine to me. I tried a simpler version of the same command locally and it builds fine. If you try something like this, does it work for you? (Where foo.cpp is just a dummy program that calls cpptrace::generate_trace)

g++ foo.cpp -o foo.o -c -Iwherever
g++ foo.o -Lwherever -Wl,-rpath,wherever libcpptrace.so -o foo -Wl,--as-needed -rdynamic
CharlesJQuarra commented 11 months ago

The original successful test example I had was linking like this:

/usr/bin/c++ -g -DENABLE_DEBUG_MACROS -rdynamic CMakeFiles/simple.dir/src/simple_test.cpp.o -o simple -Wl,-rpath,/home/charlesq/Projects/testbt/build/_deps/cpptrace-build -lasan -ldl -lrt -ltinfo -lpthread -lm -lunwind _deps/cpptrace-build/libcpptrace.so

which is similar to yours, except in yours the -rdynamic option comes after the libraries, I tried to see if there is some way in cmake to add link options in specific order relative to libraries but I couldn't find one. But since it works in the test example perhaps the order of that specific option is not essential

g++ foo.cpp -o foo.o -c -Iwherever
g++ foo.o -Lwherever -Wl,-rpath,wherever libcpptrace.so -o foo -Wl,--as-needed -rdynamic
jeremy-rifkin commented 11 months ago

I'm pretty puzzled by what's going on here. I'll think on it a little more but I'm not immediately thinking of anything else. If there's a way I could reproduce locally that'd be helpful.

CharlesJQuarra commented 11 months ago

I'm pretty puzzled by what's going on here. I'll think on it a little more but I'm not immediately thinking of anything else. If there's a way I could reproduce locally that'd be helpful.

I spent half my day trying to reproduce the undefined reference of the complex project into the test project by adding all the dependencies like libtorch, opencv, etc. and nothing I brought resulted in the problem happening

I gave up and decided to install cpptrace system-wide, and using find_package() and the cpptrace::cpptrace target now links and works correctly on the complex project.. The only reason I could discern is that using FetchContent_MakeAvailable the library is added as relative path "../_deps/cpptrace-build/libcpptrace.so", but running the linking command directly with the full path didn't help. It bothers me that it would work with the system-wide installation but not the other, but the only lead is that perhaps the cmake scripts being installed are doing something different than the FetchContent strategy?

jeremy-rifkin commented 11 months ago

That I'm not sure, I thought FetchContent relied on the same definitions that cmake uses for the install target. I would not expect a relative or absolute path to cause an error here for the linker step at least.

jeremy-rifkin commented 10 months ago

Hi @CharlesJQuarra, I'm going to close this issue for now since a lot of time has passed and also a lot has changed in the library. There's a chance if things never ended up resolved on your end, it may work now or when the next release comes out.