Missing symbol after compilation

pmdaye commented 3 years ago

Hello,

I followed the process to compile libedgetpu using the makefile_build approach. Everything went OK during the compilation. However, we I run the test using:

python3 examples/classify_image.py --model test_data/mobilenet_v2_1.0_224_inat_bird_quant_edgetpu.tflite --labels test_data/inat_bird_labels.txt --input test_data/parrot.jpg

I have a missing symbol:

ImportError: /usr/lib/libedgetpu.so.1: undefined symbol: _ZN4absl12lts_2021032419str_format_internal13FormatArgImpl8DispatchIjEEbNS2_4DataENS1_24FormatConversionSpecImplEPv

I checked that this was not a mismatch between my tensorflow commit using this:

python3 -c 'print(import("tflite_runtime").__git_version__)' a4dfb8d1a71385bd6d122e4f27f86dcebb96712d

Which corresponds to the commit provided in https://github.com/google-coral/libedgetpu#makefile Therefore, I think that I do have the correct version running... Maybe I did miss something?

Thank you for any help you could provide!

Pi-r

PS: I cannot use the bazel nor the docker approach as I am compiling this on an Ultrascale+ from Xilinx.

pmdaye commented 3 years ago

I did another test to check if the symbol is actually missing or not...

root@uz4cg-kol-2020-2:~# readelf -Ws /usr/lib/libedgetpu.so.1 | grep _ZN4absl12lts_2021032419str_format_internal13FormatArgImpl8DispatchIjEEbNS2_4DataENS1_24FormatConversionSpecImplEPv 26: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND _ZN4absl12lts_2021032419str_format_internal13FormatArgImpl8DispatchIjEEbNS2_4DataENS1_24FormatConversionSpecImplEPv 31247: 0000000000000000 0 NOTYPE GLOBAL DEFAULT UND _ZN4absl12lts_2021032419str_format_internal13FormatArgImpl8DispatchIjEEbNS2_4DataENS1_24FormatConversionSpecImplEPv

It seems that the symbol is present in the library... I must admit that I am a bit lost now!

dmitriykovalev commented 3 years ago

This is a symbol from Abseil library:

% c++filt -n _ZN4absl12lts_2021032419str_format_internal13FormatArgImpl8DispatchIjEEbNS2_4DataENS1_24FormatConversionSpecImplEPv
bool absl::lts_20210324::str_format_internal::FormatArgImpl::Dispatch<unsigned int>(absl::lts_20210324::str_format_internal::FormatArgImpl::Data, absl::lts_20210324::str_format_internal::FormatConversionSpecImpl, void*)

It's undefined in libedgetpu.so.1 and should be loaded from the corresponding Abseil shared library:

readlef -s --wide /usr/lib/<multiarch>/libabsl_str_format_internal.so

It's weird because you've just compiled libedgetpu without errors. Please copy the output of ldd command:

ldd  /usr/lib/libedgetpu.so.1
ldd -r  /usr/lib/libedgetpu.so.1  # just in case

pmdaye commented 3 years ago

@dmitriykovalev , thanks for the quick reply!

Some additional details, if I compile using my CMakeLists.txt, I can follow the procedure explained in the abseil-gcc git (see [https://github.com/abseil/abseil-cpp/blob/master/CMake/README.md](this url) for the details). I had to do that as my development board doesn't have abseil in its package database. If I tried to compile and install abseil-cpp, and then I compile libedgetpu, it doesn't work. I think that this is because the CMakeLists.txt of abseil-cpp doesn't allow to compile a shared library. It has to be a static library. If I use my CMakeLists.txt, I can compile abseil (and flatbuffers) within the same project and everything is properly integrated.

In the first case (makefile_build), this is the ldd -r of the created library:

 linux-vdso.so.1 (0x0000ffff83762000)
        libflatbuffers.so.2 => /usr/local/lib64/libflatbuffers.so.2 (0x0000ffff83375000)
        libusb-1.0.so.0 => /lib/libusb-1.0.so.0 (0x0000ffff8334b000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x0000ffff8316c000)
        libm.so.6 => /lib/libm.so.6 (0x0000ffff830c2000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x0000ffff8309d000)
        libc.so.6 => /lib/libc.so.6 (0x0000ffff82f32000)
        libudev.so.1 => /lib/libudev.so.1 (0x0000ffff82ef9000)
        libpthread.so.0 => /lib/libpthread.so.0 (0x0000ffff82ec9000)
        /lib/ld-linux-aarch64.so.1 (0x0000ffff83734000)
undefined symbol: _ZN4absl12lts_2021032419str_format_internal13FormatArgImpl8DispatchIjEEbNS2_4DataENS1_24FormatConversionSpecImplEPv   (out/direct/k8/libedgetpu.so.1)
undefined symbol: _ZN4absl12lts_2021032419str_format_internal13FormatArgImpl8DispatchINS0_11string_viewEEEbNS2_4DataENS1_24FormatConversionSpecImplEPv  (out/direct/k8/libedgetpu.so.1)
undefined symbol: _ZN4absl12lts_2021032413hash_internal9HashState5kSeedE        (out/direct/k8/libedgetpu.so.1)
undefined symbol: _ZN4absl12lts_2021032419str_format_internal13FormatArgImpl8DispatchIxEEbNS2_4DataENS1_24FormatConversionSpecImplEPv   (out/direct/k8/libedgetpu.so.1)
undefined symbol: _ZN4absl12lts_2021032419str_format_internal13FormatArgImpl8DispatchImEEbNS2_4DataENS1_24FormatConversionSpecImplEPv   (out/direct/k8/libedgetpu.so.1)
undefined symbol: _ZN4absl12lts_2021032419str_format_internal13FormatArgImpl8DispatchIyEEbNS2_4DataENS1_24FormatConversionSpecImplEPv   (out/direct/k8/libedgetpu.so.1)
undefined symbol: _ZN4absl12lts_2021032419str_format_internal13FormatArgImpl8DispatchIPKcEEbNS2_4DataENS1_24FormatConversionSpecImplEPv (out/direct/k8/libedgetpu.so.1)
undefined symbol: _ZN4absl12lts_2021032419str_format_internal13FormatArgImpl8DispatchIiEEbNS2_4DataENS1_24FormatConversionSpecImplEPv   (out/direct/k8/libedgetpu.so.1)
undefined symbol: _ZTVN4absl12lts_2021032414flags_internal8FlagImplE    (out/direct/k8/libedgetpu.so.1)
undefined symbol: _ZN4absl12lts_2021032410SimpleAtobENS0_11string_viewEPb       (out/direct/k8/libedgetpu.so.1)
undefined symbol: _ZN4absl12lts_2021032414flags_internal23RegisterCommandLineFlagERNS0_15CommandLineFlagEPKc    (out/direct/k8/libedgetpu.so.1)
undefined symbol: _ZN4absl12lts_2021032416numbers_internal17safe_strto32_baseENS0_11string_viewEPii     (out/direct/k8/libedgetpu.so.1)
undefined symbol: _ZN4absl12lts_2021032414flags_internal7UnparseB5cxx11Eb       (out/direct/k8/libedgetpu.so.1)
undefined symbol: _ZN4absl12lts_2021032414flags_internal7UnparseB5cxx11Ei       (out/direct/k8/libedgetpu.so.1)
undefined symbol: _ZN4absl12lts_2021032414flags_internal13AbslParseFlagENS0_11string_viewEPbPNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE       (out/direct/k8/libedgetpu.so.1)
undefined symbol: _ZN4absl12lts_2021032414flags_internal13AbslParseFlagENS0_11string_viewEPiPNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE       (out/direct/k8/libedgetpu.so.1)
undefined symbol: _ZNK4absl12lts_2021032414flags_internal8FlagImpl15AssertValidTypeEPKvPFPKSt9type_infovE       (out/direct/k8/libedgetpu.so.1)
undefined symbol: _ZNK4absl12lts_2021032414flags_internal8FlagImpl4ReadEPv      (out/direct/k8/libedgetpu.so.1)
undefined symbol: _ZN4absl12lts_2021032419str_format_internal10FormatPackB5cxx11ENS1_21UntypedFormatSpecImplENS0_4SpanIKNS1_13FormatArgImplEEE  (out/direct/k8/libedgetpu.so.1)
undefined symbol: _ZN4absl12lts_2021032418container_internal21ShouldInsertBackwardsEmPa (out/direct/k8/libedgetpu.so.1)
undefined symbol: _ZN4absl12lts_2021032418container_internal37ConvertDeletedToEmptyAndFullToDeletedEPam (out/direct/k8/libedgetpu.so.1)
undefined symbol: _ZN4absl12lts_2021032417optional_internal25throw_bad_optional_accessEv        (out/direct/k8/libedgetpu.so.1)

In the CMakeLists.txt case, this is the corresponding output of ldd -r:

        linux-vdso.so.1 (0x0000ffff8d7a7000)
        libusb-1.0.so.0 => /lib/libusb-1.0.so.0 (0x0000ffff8d4bb000)
        libpthread.so.0 => /lib/libpthread.so.0 (0x0000ffff8d48b000)
        librt.so.1 => /lib/librt.so.1 (0x0000ffff8d473000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x0000ffff8d294000)
        libm.so.6 => /lib/libm.so.6 (0x0000ffff8d1ea000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x0000ffff8d1c5000)
        libc.so.6 => /lib/libc.so.6 (0x0000ffff8d058000)
        libudev.so.1 => /lib/libudev.so.1 (0x0000ffff8d021000)
        /lib/ld-linux-aarch64.so.1 (0x0000ffff8d779000)

It seems that in the Makefile there are a couple of missing symbols.

dmitriykovalev commented 3 years ago

Makefile doesn't define any symbols, it only specifies which libraries to link with using -l flags. I think it should be fine to link with Abseil library statically if needed.

I guess you should follow Traditional CMake Set-Up for Abseil in order to install all generated static/shared libraries to some common location like /usr/local and then build and link libedgetpu.

hjonnala commented 3 years ago

@pmdaye are you still having the issue?

mtiftikci commented 3 years ago

Hello everyone,

Im facing the same issue with @pmdaye. But there are a few little differences:

1) I tried to build for 32 bit arm architecture (armv7a).

In order to do so, I had to build flatbuffers (both for x86_64 and armv7a) and abseil libraries as well. Luckily I already has a cross compile toolchain supported by the supplier company.
Then I clone the tensorflow project which is needed by libedgetpu and checkout tf2.5 branch.
Then I could cross-compile with small changes on the makefile-build/Makefile. I had to make small changes on it because makefile is not suitable for cross-compile. Using python or bazel is not an option for my case unfortunately.

I did the the things above, then I need to test it on the beagleboard X15. In order to do so, I used tflite repo. I tried to compile "classification" example, but it could not be compiled unless adding linker parameter "-fuse-ld=gold"

Although the "classification" example was compiled successfully with gold linker, when I run the classification example on the target, I got the same "undefined symbol" error like @pmdaye's case.

2) When I failed on the arm target, I decided to compile and run libedgetpu on my x86_64 PC. The compilation was done successfully and the example application run without any glitches. No linker issues.

My comments:

Flatbuffer library is used while in compilation and run time. So that, when I try a cross compilation, I need both intel and arm32 architecture of library at the same time. Maybe makefile takes wrong architecture libraries while compiling cause makefile is designed to use in native compilation, not a cross-compilation case.
libedgetpu makefile uses gold linker, maybe this linker parameter causes an issue while cross-compilation.

Thanks in advance,

NobuoTsukamoto commented 2 years ago

When you source build abseil-cpp, specify the shared library build options.

cmake .. \
  -DBUILD_SHARED_LIBS=ON \
  -DBUILD_TESTING=OFF \
  -DCMAKE_CXX_STANDARD=14

It seems that some symbols are missing in the static link library of abseil-cpp (Default compilation options). When linking abseil-cpp with the static link library on Raspberry Pi OS 64bit, the same problem occurred.

google-coral / libedgetpu

Missing symbol after compilation #28