Incompatibility between Ubuntu 22.04 libdwarf-dev package and the dwarf version cpptrace uses

[twfry]

Previously I used backward-cpp for development on Ubuntu and had the Ubuntu package 'libdwarf-dev' preinstalled. Apparently this dwarf version conflicts with the dwarf version cpptrace uses and installs. However the cpptrace install process skips updating dwarf to the version needed if it already exists. I wanted to report this because it took me a bit to figure out and hopefully helps anyone else with the same issue.

Here is the cpptrace installation console output, the install process reported all dwarf files as pre-existing and 'Up-to-date' and so it skips them.

 $ sudo make install
Consolidate compiler generated dependencies of target libzstd_static
[ 29%] Built target libzstd_static
Consolidate compiler generated dependencies of target dwarf
[ 82%] Built target dwarf
Consolidate compiler generated dependencies of target cpptrace-lib
[100%] Built target cpptrace-lib
Install the project...
-- Install configuration: "Release"
-- Installing: /usr/local/lib/cmake/zstd/zstdTargets.cmake
-- Installing: /usr/local/lib/cmake/zstd/zstdTargets-release.cmake
-- Installing: /usr/local/lib/cmake/zstd/zstdConfig.cmake
-- Installing: /usr/local/lib/cmake/zstd/zstdConfigVersion.cmake
-- Installing: /usr/local/lib/pkgconfig/libzstd.pc
-- Installing: /usr/local/include/zstd.h
-- Installing: /usr/local/include/zdict.h
-- Installing: /usr/local/include/zstd_errors.h
-- Installing: /usr/local/lib/libzstd.a
-- Installing: /usr/local/lib/libdwarf.a
-- Installing: /usr/local/include/libdwarf.h
-- Installing: /usr/local/include/dwarf.h
-- Installing: /usr/local/lib/cmake/libdwarf/libdwarfConfig.cmake
-- Installing: /usr/local/lib/cmake/libdwarf/libdwarfConfigVersion.cmake
-- Up-to-date: /usr/local/lib/libdwarf.a
-- Up-to-date: /usr/local/include/libdwarf.h
-- Up-to-date: /usr/local/include/dwarf.h
-- Installing: /usr/local/lib/cmake/libdwarf/libdwarf-targets.cmake
-- Installing: /usr/local/lib/cmake/libdwarf/libdwarf-targets-release.cmake
-- Installing: /usr/local/lib/pkgconfig/libdwarf.pc
-- Installing: /usr/local/lib/cmake/libdwarf/Findzstd.cmake
-- Up-to-date: /usr/local/include
-- Installing: /usr/local/include/ctrace
-- Installing: /usr/local/include/ctrace/ctrace.h
-- Installing: /usr/local/include/cpptrace
-- Installing: /usr/local/include/cpptrace/cpptrace.hpp
-- Installing: /usr/local/lib/libcpptrace.a
-- Installing: /usr/local/lib/cmake/cpptrace/cpptrace-config.cmake
-- Installing: /usr/local/lib/cmake/cpptrace/cpptrace-config-version.cmake
-- Installing: /usr/local/lib/cmake/cpptrace/cpptrace-targets.cmake
-- Installing: /usr/local/lib/cmake/cpptrace/cpptrace-targets-release.cmake
-- Installing: /usr/local/lib/cmake/cpptrace/Findzstd.cmake

However when attempting to build the demo.cpp file to test, several unresolved errors were reported, all relating to dwarf.

$ g++ demo.cpp -o demo -I /usr/local/include -g -Wall -lcpptrace -ldwarf -lz -lzstd -ldl
/usr/bin/ld: /usr/local/lib/libcpptrace.a(symbols_with_libdwarf.cpp.o): in function `cpptrace::detail::libdwarf::get_resolver_for_object(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
symbols_with_libdwarf.cpp:(.text+0xc9a): undefined reference to `dwarf_init_path_a'
/usr/bin/ld: /usr/local/lib/libcpptrace.a(symbols_with_libdwarf.cpp.o): in function `void cpptrace::detail::libdwarf::die_object::dwarf_ranges<cpptrace::detail::libdwarf::die_object::pc_in_die(int, unsigned long long) const::{lambda(unsigned long long, unsigned long long)#1}>(int, cpptrace::detail::optional<unsigned long long, 0>, cpptrace::detail::libdwarf::die_object::pc_in_die(int, unsigned long long) const::{lambda(unsigned long long, unsigned long long)#1}) const':
symbols_with_libdwarf.cpp:(.text._ZNK8cpptrace6detail8libdwarf10die_object12dwarf_rangesIZNKS2_9pc_in_dieEiyEUlyyE_EEviNS0_8optionalIyLi0EEET_[_ZNK8cpptrace6detail8libdwarf10die_object12dwarf_rangesIZNKS2_9pc_in_dieEiyEUlyyE_EEviNS0_8optionalIyLi0EEET_]+0x223): undefined reference to `dwarf_dealloc_ranges'
/usr/bin/ld: symbols_with_libdwarf.cpp:(.text._ZNK8cpptrace6detail8libdwarf10die_object12dwarf_rangesIZNKS2_9pc_in_dieEiyEUlyyE_EEviNS0_8optionalIyLi0EEET_[_ZNK8cpptrace6detail8libdwarf10die_object12dwarf_rangesIZNKS2_9pc_in_dieEiyEUlyyE_EEviNS0_8optionalIyLi0EEET_]+0x2ab): undefined reference to `dwarf_dealloc_ranges'
/usr/bin/ld: /usr/local/lib/libcpptrace.a(symbols_with_libdwarf.cpp.o): in function `cpptrace::detail::raii_wrapper<Dwarf_Ranges_s*, cpptrace::detail::libdwarf::die_object::dwarf4_ranges<cpptrace::detail::libdwarf::die_object::get_rangelist_entries(int) const::{lambda(unsigned long long, unsigned long long)#1}>(unsigned long long, cpptrace::detail::libdwarf::die_object::get_rangelist_entries(int) const::{lambda(unsigned long long, unsigned long long)#1}) const::{lambda(Dwarf_Ranges_s*)#2}, 0, 0, 0>::~raii_wrapper()':
symbols_with_libdwarf.cpp:(.text._ZN8cpptrace6detail12raii_wrapperIP14Dwarf_Ranges_sZNKS0_8libdwarf10die_object13dwarf4_rangesIZNKS5_21get_rangelist_entriesEiEUlyyE_EEvyT_EUlS3_E0_Li0ELi0ELi0EED2Ev[_ZN8cpptrace6detail12raii_wrapperIP14Dwarf_Ranges_sZNKS0_8libdwarf10die_object13dwarf4_rangesIZNKS5_21get_rangelist_entriesEiEUlyyE_EEvyT_EUlS3_E0_Li0ELi0ELi0EED5Ev]+0x23): undefined reference to `dwarf_dealloc_ranges'
/usr/bin/ld: /usr/local/lib/libcpptrace.a(symbols_with_libdwarf.cpp.o): in function `std::_Function_handler<bool (cpptrace::detail::libdwarf::die_object const&), cpptrace::detail::libdwarf::dwarf_resolver::lazy_generate_cu_cache()::{lambda(cpptrace::detail::libdwarf::die_object const&)#1}>::_M_invoke(std::_Any_data const&, cpptrace::detail::libdwarf::die_object const&)':
symbols_with_libdwarf.cpp:(.text._ZNSt17_Function_handlerIFbRKN8cpptrace6detail8libdwarf10die_objectEEZNS2_14dwarf_resolver22lazy_generate_cu_cacheEvEUlS5_E_E9_M_invokeERKSt9_Any_dataS5_[_ZNSt17_Function_handlerIFbRKN8cpptrace6detail8libdwarf10die_objectEEZNS2_14dwarf_resolver22lazy_generate_cu_cacheEvEUlS5_E_E9_M_invokeERKSt9_Any_dataS5_]+0x589): undefined reference to `dwarf_dealloc_ranges'
/usr/bin/ld: /usr/local/lib/libcpptrace.a(symbols_with_libdwarf.cpp.o): in function `std::_Function_handler<bool (cpptrace::detail::libdwarf::die_object const&), cpptrace::detail::libdwarf::dwarf_resolver::preprocess_subprograms(cpptrace::detail::libdwarf::die_object const&, unsigned short, std::vector<cpptrace::detail::libdwarf::subprogram_entry, std::allocator<cpptrace::detail::libdwarf::subprogram_entry> >&)::{lambda(cpptrace::detail::libdwarf::die_object const&)#1}>::_M_invoke(std::_Any_data const&, cpptrace::detail::libdwarf::die_object const&)':
symbols_with_libdwarf.cpp:(.text._ZNSt17_Function_handlerIFbRKN8cpptrace6detail8libdwarf10die_objectEEZNS2_14dwarf_resolver22preprocess_subprogramsES5_tRSt6vectorINS2_16subprogram_entryESaIS9_EEEUlS5_E_E9_M_invokeERKSt9_Any_dataS5_[_ZNSt17_Function_handlerIFbRKN8cpptrace6detail8libdwarf10die_objectEEZNS2_14dwarf_resolver22preprocess_subprogramsES5_tRSt6vectorINS2_16subprogram_entryESaIS9_EEEUlS5_E_E9_M_invokeERKSt9_Any_dataS5_]+0x739): undefined reference to `dwarf_dealloc_ranges'
collect2: error: ld returned 1 exit status

After searching for a while I suspected a package incompatibility issue, so I first tried to uninstall the Ubuntu libdwarf-dev package and reinstall cpptrace. This had no impact and the files were still reported as up to date by the install process.

The final solution was to manually delete all dwarf related files listed on the cpptrace install output and re-run cpptrace install. This forced all files included with cpptrace to be installed. After this the demo.cpp build and output works as expected.

The cpptrace install process ideally should have upgraded the existing dwarf files, or at least reported some sort of potential conflict. Again just wanted report this in case others run into it or the install process can handle it. Thanks for the package, it looks great from initial testing, I've needed a full signal-safe backtrace and need to play with that but it looks promising.

[twfry]

Some additional information

My application builds with cpptrace fine. However if I reinstall Ubuntu's libdwarf-dev package again with "sudo apt install libdwarf-dev" and then recompile I receive the following linker errors

/usr/bin/ld: /usr/local/lib/libcpptrace.a(symbols_with_libdwarf.cpp.o): in function `cpptrace::detail::libdwarf::get_resolver_for_object(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
symbols_with_libdwarf.cpp:(.text+0xc9a): undefined reference to `dwarf_init_path_a'
/usr/bin/ld: /usr/local/lib/libcpptrace.a(symbols_with_libdwarf.cpp.o): in function `void cpptrace::detail::libdwarf::die_object::dwarf_ranges<cpptrace::detail::libdwarf::die_object::pc_in_die(int, unsigned long long) const::{lambda(unsigned long long, unsigned long long)#1}>(int, cpptrace::detail::optional<unsigned long long, 0>, cpptrace::detail::libdwarf::die_object::pc_in_die(int, unsigned long long) const::{lambda(unsigned long long, unsigned long long)#1}) const':
symbols_with_libdwarf.cpp:(.text._ZNK8cpptrace6detail8libdwarf10die_object12dwarf_rangesIZNKS2_9pc_in_dieEiyEUlyyE_EEviNS0_8optionalIyLi0EEET_[_ZNK8cpptrace6detail8libdwarf10die_object12dwarf_rangesIZNKS2_9pc_in_dieEiyEUlyyE_EEviNS0_8optionalIyLi0EEET_]+0x223): undefined reference to `dwarf_dealloc_ranges'
/usr/bin/ld: symbols_with_libdwarf.cpp:(.text._ZNK8cpptrace6detail8libdwarf10die_object12dwarf_rangesIZNKS2_9pc_in_dieEiyEUlyyE_EEviNS0_8optionalIyLi0EEET_[_ZNK8cpptrace6detail8libdwarf10die_object12dwarf_rangesIZNKS2_9pc_in_dieEiyEUlyyE_EEviNS0_8optionalIyLi0EEET_]+0x2ab): undefined reference to `dwarf_dealloc_ranges'
/usr/bin/ld: /usr/local/lib/libcpptrace.a(symbols_with_libdwarf.cpp.o): in function `cpptrace::detail::raii_wrapper<Dwarf_Ranges_s*, cpptrace::detail::libdwarf::die_object::dwarf4_ranges<cpptrace::detail::libdwarf::die_object::get_rangelist_entries(int) const::{lambda(unsigned long long, unsigned long long)#1}>(unsigned long long, cpptrace::detail::libdwarf::die_object::get_rangelist_entries(int) const::{lambda(unsigned long long, unsigned long long)#1}) const::{lambda(Dwarf_Ranges_s*)#2}, 0, 0, 0>::~raii_wrapper()':
symbols_with_libdwarf.cpp:(.text._ZN8cpptrace6detail12raii_wrapperIP14Dwarf_Ranges_sZNKS0_8libdwarf10die_object13dwarf4_rangesIZNKS5_21get_rangelist_entriesEiEUlyyE_EEvyT_EUlS3_E0_Li0ELi0ELi0EED2Ev[_ZN8cpptrace6detail12raii_wrapperIP14Dwarf_Ranges_sZNKS0_8libdwarf10die_object13dwarf4_rangesIZNKS5_21get_rangelist_entriesEiEUlyyE_EEvyT_EUlS3_E0_Li0ELi0ELi0EED5Ev]+0x23): undefined reference to `dwarf_dealloc_ranges'
/usr/bin/ld: /usr/local/lib/libcpptrace.a(symbols_with_libdwarf.cpp.o): in function `std::_Function_handler<bool (cpptrace::detail::libdwarf::die_object const&), cpptrace::detail::libdwarf::dwarf_resolver::lazy_generate_cu_cache()::{lambda(cpptrace::detail::libdwarf::die_object const&)#1}>::_M_invoke(std::_Any_data const&, cpptrace::detail::libdwarf::die_object const&)':
symbols_with_libdwarf.cpp:(.text._ZNSt17_Function_handlerIFbRKN8cpptrace6detail8libdwarf10die_objectEEZNS2_14dwarf_resolver22lazy_generate_cu_cacheEvEUlS5_E_E9_M_invokeERKSt9_Any_dataS5_[_ZNSt17_Function_handlerIFbRKN8cpptrace6detail8libdwarf10die_objectEEZNS2_14dwarf_resolver22lazy_generate_cu_cacheEvEUlS5_E_E9_M_invokeERKSt9_Any_dataS5_]+0x589): undefined reference to `dwarf_dealloc_ranges'
/usr/bin/ld: /usr/local/lib/libcpptrace.a(symbols_with_libdwarf.cpp.o): in function `std::_Function_handler<bool (cpptrace::detail::libdwarf::die_object const&), cpptrace::detail::libdwarf::dwarf_resolver::preprocess_subprograms(cpptrace::detail::libdwarf::die_object const&, unsigned short, std::vector<cpptrace::detail::libdwarf::subprogram_entry, std::allocator<cpptrace::detail::libdwarf::subprogram_entry> >&)::{lambda(cpptrace::detail::libdwarf::die_object const&)#1}>::_M_invoke(std::_Any_data const&, cpptrace::detail::libdwarf::die_object const&)':
symbols_with_libdwarf.cpp:(.text._ZNSt17_Function_handlerIFbRKN8cpptrace6detail8libdwarf10die_objectEEZNS2_14dwarf_resolver22preprocess_subprogramsES5_tRSt6vectorINS2_16subprogram_entryESaIS9_EEEUlS5_E_E9_M_invokeERKSt9_Any_dataS5_[_ZNSt17_Function_handlerIFbRKN8cpptrace6detail8libdwarf10die_objectEEZNS2_14dwarf_resolver22preprocess_subprogramsES5_tRSt6vectorINS2_16subprogram_entryESaIS9_EEEUlS5_E_E9_M_invokeERKSt9_Any_dataS5_]+0x739): undefined reference to `dwarf_dealloc_ranges'
collect2: error: ld returned 1 exit status

If I then remove libdwarf-dev with "sudo apt remove libdwarf-dev", the application builds fine again.

This is with Ubuntu 22.04 LTS. Again the only change is installing the libdwarf-dev package and then removing it. Everything with the application and build process remains the same. I am using make for this and calling g++ directly.

[jeremy-rifkin]

Thanks for opening this. I'm sorry to hear you've ran into a subtle challenge here, this is a tricky problem and I'm not sure much can be done on the cpptrace side of things. I don't think it would be ideal to clobber an existing libdwarf installation. The libdwarf installation is all handled by upstream libdwarf's cmake configuration however I were to try adding a warning for this on the cpptrace side of things I would need to detect libdwarf was already installed and then I'd need to try to detect what version to check if it's actually a compatibility issue and not already installed. This is easier said than done and gets tricky quickly, especially given there are libdwarf installations aren't consistent in what files are installed and where depending on version, build system, etc.

In general I'd probably recommend leaning away from a system-wide installation, however it certainly is convenient.

At the very least I should document that a relatively recent libdwarf is needed.

[twfry]

Thanks for looking into it, understand there are no easy solutions here but it was subtle enough to root cause that it seemed worthwhile to at least document. In my case it is on the Ubuntu 22.04 package which is still fairly recent so others will likely run into it as well.

I still use Makefiles and have not migrated over to the CMake build process. If I had been using CMake to build do you think that would have prevented libdwarf from being clobbered by the system's version?

[jeremy-rifkin]

Thanks, I'll go ahead and close this as completed for now since I've updated the docs but I'll keep this in mind as a potential place to improve things.

If you'd been using cmake it would probably be easier to avoid a system-wide installation as the easiest installation mechanism, e.g. by using FetchContent or a package manager or installing to an alternative location and setting the CMAKE_PREFIX_PATH which controls where cmake looks for packages.