Add libbpftime.a archive when BPFTIME_BUILD_STATIC_LIB option is provided

hp77-creator commented 4 months ago

Description

Fixes #232 Changes taken from #271

Type of change

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?

[ ] Run build with BPFTIME_BUILD_STATIC_LIB

Test Configuration:

Firmware version: 10151.101.3
Hardware: Macbook Air M2 chip
Toolchain: Xcode
SDK:

Checklist

[ ] My code follows the style guidelines of this project
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I have made corresponding changes to the documentation
[ ] My changes generate no new warnings
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] New and existing unit tests pass locally with my changes
[ ] Any dependent changes have been merged and published in downstream modules
[ ] I have checked my code and corrected any misspellings

hp77-creator commented 4 months ago

@Officeyutong @yunwei37 I have added targets, I can see when running on a fedora vm that a archive file is generated with the above changes. I am looking for comments from you two on how can I enhance this more, Shall I directly install the CLI with our archive if BPFTIME_BUILD_STATIC is ON ? I haven't added that change. Also on my open PR I had asked for help regarding testing the archive, I will try to do that in a separate program, looking into that now.

Officeyutong commented 4 months ago

Please add an example to demonstrate how to use the static library, such as a small program that dumps loaded programs from the shared memory. The program should use headers from bpftime, and links the static library you built. It's ok if this program was not built using CMake, since the main purpose that we release standalone static library is to make it more convenient for users who doesn't want to use CMake

hp77-creator commented 4 months ago

Okay @Officeyutong , I am working on that.

hp77-creator commented 4 months ago

I wrote one program, I only included "bpftime_shm.hpp" but I had to include libubpf.a for that, I tried to add it into our single archive but I am getting following error:

- Configuring done (0.6s)
CMake Error at CMakeLists.txt:140 (add_custom_command):
  Error evaluating generator expression:

    $<TARGET_FILE:ubpf>

  Target "ubpf" is not an executable or library.

To compile the program, I had to write the following compilation command:

g++ -o example main.cpp -I/home/fedora/codes/bpftime-hp/runtime/include/ -I/home/fedora/codes/bpftime-hp/vm/vm-core/compat/include -I/home/fedora/codes/bpftime-hp/vm/compat/include/ -L/home/fedora/.bpftime -lbpftime -lboost_system -lrt -lbpf -L/home/fedora/codes/bpftime-hp/build/vm/ubpf-vm/ubpf/lib -lubpf

I am trying to add ubpf as well but unable to, any tips?? @Officeyutong @yunwei37

Officeyutong commented 4 months ago

I wrote one program, I only included "bpftime_shm.hpp" but I had to include libubpf.a for that, I tried to add it into our single archive but I am getting following error:
- Configuring done (0.6s)
CMake Error at CMakeLists.txt:140 (add_custom_command):
  Error evaluating generator expression:

    $<TARGET_FILE:ubpf>

  Target "ubpf" is not an executable or library.
To compile the program, I had to write the following compilation command:
g++ -o example main.cpp -I/home/fedora/codes/bpftime-hp/runtime/include/ -I/home/fedora/codes/bpftime-hp/vm/vm-core/compat/include -I/home/fedora/codes/bpftime-hp/vm/compat/include/ -L/home/fedora/.bpftime -lbpftime -lboost_system -lrt -lbpf -L/home/fedora/codes/bpftime-hp/build/vm/ubpf-vm/ubpf/lib -lubpf
I am trying to add ubpf as well but unable to, any tips?? @Officeyutong @yunwei37

Because ubpf was not in our project structure. We add ubpf into bpftime through externalproject, not add_subdirectory (because add_subdirectory may lead to a series of directory conflicts).

Besides, your static library should also contain dependencies like ubpf or spdlog. So a user who wants to use bpftime won't need to fetch these libraries by himself

hp77-creator commented 4 months ago

Yea, I saw, it is being added via ExternalProject, but still unlike spdlog, I am not able to add it, I don't know why 🤔, will have to look into it.

Officeyutong commented 4 months ago

Yea, I saw, it is being added via ExternalProject, but still unlike spdlog, I am not able to add it, I don't know why 🤔, will have to look into it.

spdlog was added using add_subdirectory. You may search keyword spdlog among all cmake files to get detailed configurations

hp77-creator commented 4 months ago

I meant, I am able to add spdlog to the static archive. But I am not able to add ubpf. 😅

yunwei37 commented 4 months ago

Maybe you can try to manually build ubpf in cmake if It's used, and add it to the static library?

Officeyutong commented 4 months ago

I meant, I am able to add spdlog to the static archive. But I am not able to add ubpf. 😅

This is how bpftime links ubpf, you may link it in the same way https://github.com/eunomia-bpf/bpftime/blob/307b16f27a57d3243f4929aa920b280bb21864d3/vm/ubpf-vm/CMakeLists.txt#L18

hp77-creator commented 4 months ago

sure, I will see that, meanwhile I tried to run a program with just "spdlog/spdlog.h" header and linking libbpftime.a, I am getting a lot of undefined reference for the program

/usr/bin/ld: /home/fedora/.bpftime/libbpftime.a(frida_attach_utils.cpp.o): in function `bpftime::attach::find_function_addr_by_name(char const*)':
frida_attach_utils.cpp:(.text+0x3f0): undefined reference to `gum_find_function'
/usr/bin/ld: frida_attach_utils.cpp:(.text+0x410): undefined reference to `gum_module_find_export_by_name'
/usr/bin/ld: /home/fedora/.bpftime/libbpftime.a(frida_attach_utils.cpp.o): in function `bpftime::attach::get_module_base_addr(char const*)':
frida_attach_utils.cpp:(.text+0x444): undefined reference to `gum_module_load'
/usr/bin/ld: frida_attach_utils.cpp:(.text+0x454): undefined reference to `gum_module_find_base_address'

I am figuring its some error because headers are not added but even after including header with -I/home/fedora/bpftime-hp/attach/frida_uprobe_attach_impl/include error exists, @Officeyutong @yunwei37 can you folks try at your ends as well, if possible and let me know if its giving same error for you for some program that you have Meanwhile I will look on how we can add libubpf.a to our archive.

Officeyutong commented 4 months ago

sure, I will see that, meanwhile I tried to run a program with just "spdlog/spdlog.h" header and linking libbpftime.a, I am getting a lot of undefined reference for the program
/usr/bin/ld: /home/fedora/.bpftime/libbpftime.a(frida_attach_utils.cpp.o): in function `bpftime::attach::find_function_addr_by_name(char const*)':
frida_attach_utils.cpp:(.text+0x3f0): undefined reference to `gum_find_function'
/usr/bin/ld: frida_attach_utils.cpp:(.text+0x410): undefined reference to `gum_module_find_export_by_name'
/usr/bin/ld: /home/fedora/.bpftime/libbpftime.a(frida_attach_utils.cpp.o): in function `bpftime::attach::get_module_base_addr(char const*)':
frida_attach_utils.cpp:(.text+0x444): undefined reference to `gum_module_load'
/usr/bin/ld: frida_attach_utils.cpp:(.text+0x454): undefined reference to `gum_module_find_base_address'
I am figuring its some error because headers are not added but even after including header with -I/home/fedora/bpftime-hp/attach/frida_uprobe_attach_impl/include error exists, @Officeyutong @yunwei37 can you folks try at your ends as well, if possible and let me know if its giving same error for you for some program that you have Meanwhile I will look on how we can add libubpf.a to our archive.

Symbols like gum_XXX are defined in frida gum, which is a dependency of frida_uprobe_attach_impl, we link it at https://github.com/eunomia-bpf/bpftime/blob/307b16f27a57d3243f4929aa920b280bb21864d3/attach/frida_uprobe_attach_impl/CMakeLists.txt#L17

yunwei37 commented 4 months ago

You also need to link the frida-gum library to libbpftime.a, which is similar to ubpf because they are not built by the project.

Is it possible to call something like bpftime_add_static_lib_component_command on a xxx.a static library?

hp77-creator commented 4 months ago

I will check that, sorry, haven't been able to implement @Officeyutong 's suggestions, I guess we can do that on a target for our function but will let you know for sure in some time, by tomorrow or day after tomorrow EOD.

hp77-creator commented 4 months ago

Tried adding both lib-ubpf and libfrida-gum and was getting following error:

 g++ -o example spdlog_exam.cpp -I/home/fedora/codes/bpftime-hp/runtime/include/ -I/home/fedora/codes/bpftime-hp/vm/vm-core/compat/include -I/home/fedora/codes/bpftime-hp/vm/compat/include/ -I/home/fedora/codes/bpftime-hp/third_party/spdlog/include -L/home/fedora/.bpftime -lbpftime -lboost_system -lrt -lbpf
/usr/bin/ld: /home/fedora/.bpftime/libbpftime.a(dwarf_die_deliv.c.o): in function `dwarf_die_from_hash_signature':
/__w/frida/frida/build/fs-tmp-linux-arm64/libdwarf/../../../deps/libdwarf/libdwarf/dwarf_die_deliv.c:1825: undefined reference to `_frida_dwarf_get_cu_die_offset_given_cu_header_offset_b'
/usr/bin/ld: /home/fedora/.bpftime/libbpftime.a(inflate.c.o): in function `inflate':
/__w/frida/frida/build/fs-tmp-linux-arm64/zlib/../../../deps/zlib/inflate.c:670: undefined reference to `_frida_crc32'
/usr/bin/ld: /__w/frida/frida/build/fs-tmp-linux-arm64/zlib/../../../deps/zlib/inflate.c:671: undefined reference to `_frida_crc32'
/usr/bin/ld: /__w/frida/frida/build/fs-tmp-linux-arm64/zlib/../../../deps/zlib/inflate.c:725: undefined reference to `_frida_crc32'
/usr/bin/ld: /__w/frida/frida/build/fs-tmp-linux-arm64/zlib/../../../deps/zlib/inflate.c:734: undefined reference to `_frida_crc32'
/usr/bin/ld: /__w/frida/frida/build/fs-tmp-linux-arm64/zlib/../../../deps/zlib/inflate.c:745: undefined reference to `_frida_crc32'
/usr/bin/ld: /home/fedora/.bpftime/libbpftime.a(inflate.c.o):/__w/frida/frida/build/fs-tmp-linux-arm64/zlib/../../../deps/zlib/inflate.c:756: more undefined references to `_frida_crc32' follow
collect2: error: ld returned 1 exit status

i don't have much experience with frida do I need to include something or link something else?

Officeyutong commented 4 months ago

_frida_crc32

Symbol _frida_crc32 is defined in crc32.c.o of libfrida-gum.a, and _frida_dwarf_get_cu_die_offset_given_cu_header_offset_b is defined in dwarf_global.c.o of libfrida-gum.a. Please check if these two object files are in your static archive.

You may use nm libfrida-gum.a to check which symbols are defined in object files of the given archive.

............
dwarf_global.c.o:
0000000000000000 t dealloc_globals_chain
0000000000000000 t _dwarf_make_global_add_to_chain
                 U _frida__dwarf_check_string_valid
0000000000000000 T _frida_dwarf_CU_dieoffset_given_die
                 U _frida_dwarf_dealloc
                 U _frida__dwarf_error
                 U _frida__dwarf_error_string
                 U _frida__dwarf_get_alloc
0000000000000000 T _frida_dwarf_get_cu_die_offset_given_cu_header_offset
0000000000000000 T _frida_dwarf_get_cu_die_offset_given_cu_header_offset_b
...........

crc32.c.o:
0000000000000000 r crc_braid_table
0000000000000000 r crc_table
0000000000000000 t crc_word
0000000000000000 T _frida_crc32
.............

hp77-creator commented 4 months ago

I checked with nm both object files are there but their nm output is different in libbpftime.a and in libfrida-gum.a

hp77-creator commented 4 months ago

I tried checking separately. I checked nm libfrida-gum.a , I could see the same output that you pasted. But when I did ar -x libfrida-gum.a and then tried to check with nm dwarf_global.c.o, symbols were different. Is ar doing something here?

Officeyutong commented 4 months ago

I tried checking separately.

I checked nm libfrida-gum.a , I could see the same output that you pasted. But when I did ar -x libfrida-gum.a and then tried to check with nm dwarf_global.c.o, symbols were different. Is ar doing something here?

A possible reason is that there are object files in the same name among different archives. If you unpack these archives containing object files with the same name, and put the unpacked files into a folder, some object files might be overridden by others. Please check archives you used for duplicated files names, if existed, just do a rename for these object files to avoid overridden

hp77-creator commented 4 months ago

Understood, will try this. Thanks for your input. Will update on the same.

hp77-creator commented 4 months ago

Just a question, so if I rename the object files, will there be a problem during linking, like the symbols will exist but the object file might be different in those scenarios?

yunwei37 commented 4 months ago

No, the symbol remains the same

hp77-creator commented 4 months ago

I looked for many solution but unable to find exact way, seems ar is unable to unarchive different object files if they have the same name, it will always override the object files, https://stackoverflow.com/questions/4907121/static-library-having-object-files-with-same-name-ar https://www.linux.org/threads/same-filename-o-in-a.12182/ both sites above mention that we can extract object files individually and give them directory structure but then the whole process of unarchiving and archiving becomes tedious, how do I find the directory structure, I tried giving number to the extracted files from archive files but that doesn't help, because 'ar' by itself overwrite the object file, it doesn't extract two object file of same name.

hp77-creator commented 4 months ago

found this: https://stackoverflow.com/a/9215709/7116645 maybe using ar was a mistake, will try to use above script.

hp77-creator commented 4 months ago

I am able to extract object files separately, previous error is gone, now I am getting following output:

g++ -o example spdlog_exam.cpp -I/home/fedora/codes/bpftime-hp/runtime/include/ -I/home/fedora/codes/bpftime-hp/vm/vm-core/compat/include -I/home/fedora/codes/bpftime-hp/vm/compat/include/ -I/home/fedora/codes/bpftime-hp/third_party/spdlog/include -L/home/fedora/.bpftime -lbpftime -lboost_system -lelf -lm -lbpf
/usr/bin/ld: /home/fedora/.bpftime/libbpftime.a(dwarf_crc32.c.o): in function `dwarf_basic_crc32':
/__w/frida/frida/build/fs-tmp-linux-arm64/libdwarf/../../../deps/libdwarf/libdwarf/dwarf_crc32.c:67: undefined reference to `_frida__dwarf_crc32'
/usr/bin/ld: /home/fedora/.bpftime/libbpftime.a(dwarf_crc32.c.o): in function `dwarf_crc32':
/__w/frida/frida/build/fs-tmp-linux-arm64/libdwarf/../../../deps/libdwarf/libdwarf/dwarf_crc32.c:145: undefined reference to `_frida__dwarf_crc32'
collect2: error: ld returned 1 exit status

hp77-creator commented 4 months ago

for some reason, frida-gum is not linking properly if i try via libbpftime.a looking into the issue

Officeyutong commented 4 months ago

for some reason, frida-gum is not linking properly if i try via libbpftime.a looking into the issue

May I check your libbpftime.a?

hp77-creator commented 4 months ago

How do I share it here? I have added the change, you can checkout the branch and build it in your system? @Officeyutong

Officeyutong commented 4 months ago

How do I share it here? I have added the change, you can checkout the branch and build it in your system? @Officeyutong

You may directly drag a file to text box of GitHub issue comment. It supports attaching files. Also, I'll try build it myself

hp77-creator commented 4 months ago

I am creating everything in a VM, it has some issues when I try copy pasting from there to a browser, Due to those limitations, I won't be able to attach file here, let me know if there is any issue in building it on your machine, I will push fix if needed.

Officeyutong commented 4 months ago

Referenc

One error:

[cmake] CMake Error at CMakeLists.txt:163 (add_custom_command):
[cmake]   Error evaluating generator expression:
[cmake] 
[cmake]     $<TARGET_FILE:bpftime_ubpf_vm>
[cmake] 
[cmake]   No target "bpftime_ubpf_vm"
[cmake] 
[cmake] 
[cmake] CMake Error at CMakeLists.txt:163 (add_custom_command):
[cmake]   Error evaluating generator expression:
[cmake] 
[cmake]     $<TARGET_FILE:bpftime_ubpf_vm>
[cmake] 
[cmake]   No target "bpftime_ubpf_vm"
[cmake] 
[cmake] 
[cmake] CMake Error at CMakeLists.txt:163 (add_custom_command):
[cmake]   Error evaluating generator expression:
[cmake] 
[cmake]     $<TARGET_FILE:bpftime_ubpf_vm>
[cmake] 
[cmake]   No target "bpftime_ubpf_vm"
[cmake] 
[cmake] 
[cmake] -- Generating done
[cmake] CMake Generate step failed.  Build files cannot be regenerated correctly.
[proc] The command: /usr/bin/cmake --no-warn-unused-cli -DBPFTIME_ENABLE_LTO=YES -DBPFTIME_LLVM_JIT=YES -DBPFTIME_ENABLE_MPK=YES -DBPFTIME_ENABLE_UNIT_TESTING=YES -DBUILD_ATTACH_IMPL_EXAMPLE=YES -DENABLE_LLVM_SHARED=YES -DWASMEDGE_PLUGIN_WASM_BPF=YES -DBPFTIME_BUILD_STATIC_LIB=YES -DCMAKE_BUILD_TYPE:STRING=Debug -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=TRUE -DCMAKE_C_COMPILER:FILEPATH=/usr/bin/clang-15 -DCMAKE_CXX_COMPILER:FILEPATH=/usr/bin/clang++-15 -S/root/bpftime-hp -B/root/bpftime-hp/build -G Ninja exited with code: 1

Seems you referenced target bpftime_ubpf_vm, but this target is only enabled when BPFTIME_LLVM_JIT is set to off

Officeyutong commented 4 months ago

Another thing is that BPFTIME_ENABLE_LTO must be set to OFF, or clang will emit LLVM IR files, instead of native ELF object files

Officeyutong commented 4 months ago

Seems there are two crc32.c.o in libfrida-gum.a, and they are different files..

hp77-creator commented 4 months ago

well, I have taken care of two different files with my latest changes, you can comment out 'remove_directory' step and see in objs there are two files created. Also can you share the build command that you are using for cmake? @Officeyutong

hp77-creator commented 4 months ago

I was using these two:

cmake -Bbuild  -DCMAKE_BUILD_TYPE:STRING=Release -DSPDLOG_ACTIVE_LEVEL=SPDLOG_LEVEL_INFO -DBPFTIME_BUILD_STATIC_LIB=ON
cmake --build build --config Release --target install

Officeyutong commented 4 months ago

Seems you may use ar xN <N> XXX.a YYY.o to extract the N-th entry of duplicated name YYY.o

call ar t XXX.a of each archive to extract, and gets a list of files in it (duplicated named occurred multiple times)
count how many times each name occurred. You may call list(REMOVE_DUPLICATES <list>) to dedup a list, then iterate over each unique element and count how many times it occurred. This is O(n^2), but CMake doesn't provide any mapping data structure..
For each name occurred multiple times, call ar xN <I> XXX.a YYY.o (I iterates from 1 to the number of occurrance) to extract the I-th entry, then rename it to a different name, such as YYY.1.o (or other names you like). For other names, just call ar x XXX.a YYY.o
Then we should have all entries extracted. repack them into a new archive

Officeyutong commented 4 months ago

well, I have taken care of two different files with my latest changes, you can comment out 'remove_directory' step and see in objs there are two files created. Also can you share the build command that you are using for cmake? @Officeyutong

usr/bin/cmake --no-warn-unused-cli -DBPFTIME_ENABLE_LTO=NO -DBPFTIME_LLVM_JIT=NO -DBPFTIME_ENABLE_MPK=YES -DBPFTIME_ENABLE_UNIT_TESTING=YES -DBUILD_ATTACH_IMPL_EXAMPLE=YES -DENABLE_LLVM_SHARED=YES -DWASMEDGE_PLUGIN_WASM_BPF=YES -DBPFTIME_BUILD_STATIC_LIB=YES -DCMAKE_BUILD_TYPE:STRING=Debug -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=TRUE -DCMAKE_C_COMPILER:FILEPATH=/usr/bin/clang-15 -DCMAKE_CXX_COMPILER:FILEPATH=/usr/bin/clang++-15 -S/root/bpftime-hp -B/root/bpftime-hp/build -G Ninja

Besides, I use CMake Tools of Visual Studio Code to handle tasks for cmake. This build command was generated by that plugin. Switches are defined by myself

hp77-creator commented 4 months ago

Seems you may use ar xN <N> XXX.a YYY.o to extract the N-th entry of duplicated name YYY.o

call ar t XXX.a of each archive to extract, and gets a list of files in it (duplicated named occurred multiple times)

count how many times each name occurred. You may call list(REMOVE_DUPLICATES <list>) to dedup a list, then iterate over each unique element and count how many times it occurred. This is O(n^2), but CMake doesn't provide any mapping data structure..

For each name occurred multiple times, call ar xN <I> XXX.a YYY.o (I iterates from 1 to the number of occurrance) to extract the I-th entry, then rename it to a different name, such as YYY.1.o (or other names you like). For other names, just call ar x XXX.a YYY.o

Then we should have all entries extracted. repack them into a new archive

is the extract_and_rename shell script still overwriting previous files?? I will try above approach then

Officeyutong commented 4 months ago

extract_and_rename

Seems you may use ar xN <N> XXX.a YYY.o to extract the N-th entry of duplicated name YYY.o

call ar t XXX.a of each archive to extract, and gets a list of files in it (duplicated named occurred multiple times)

count how many times each name occurred. You may call list(REMOVE_DUPLICATES <list>) to dedup a list, then iterate over each unique element and count how many times it occurred. This is O(n^2), but CMake doesn't provide any mapping data structure..

For each name occurred multiple times, call ar xN <I> XXX.a YYY.o (I iterates from 1 to the number of occurrance) to extract the I-th entry, then rename it to a different name, such as YYY.1.o (or other names you like). For other names, just call ar x XXX.a YYY.o

Then we should have all entries extracted. repack them into a new archive

is the extract_and_rename shell script still overwriting previous files?? I will try above approach then

Seems you may use ar xN <N> XXX.a YYY.o to extract the N-th entry of duplicated name YYY.o

call ar t XXX.a of each archive to extract, and gets a list of files in it (duplicated named occurred multiple times)

count how many times each name occurred. You may call list(REMOVE_DUPLICATES <list>) to dedup a list, then iterate over each unique element and count how many times it occurred. This is O(n^2), but CMake doesn't provide any mapping data structure..

For each name occurred multiple times, call ar xN <I> XXX.a YYY.o (I iterates from 1 to the number of occurrance) to extract the I-th entry, then rename it to a different name, such as YYY.1.o (or other names you like). For other names, just call ar x XXX.a YYY.o

Then we should have all entries extracted. repack them into a new archive

is the extract_and_rename shell script still overwriting previous files?? I will try above approach then

extract_and_rename doesn't work. I've tried it, and it just extracted the first occurrence of a duplicated name multiple times

hp77-creator commented 4 months ago

ok, there is an issue, i might have messed up while trying to write code from my VM to actual repo, checking.

hp77-creator commented 4 months ago

With your recent suggestion the examples that I have added are working, @Officeyutong I will now include the conditional options like for LLVM and others

hp77-creator commented 4 months ago

should we also include, libbpf in the archive, I see that it is also installed but the examples that I have added don't have that requirement. What do you suggest? @Officeyutong @yunwei37

yunwei37 commented 4 months ago

I think it would be better to include libbpf

Officeyutong commented 4 months ago

Besides, it would be better to have a Makefile to build the examples you wrote

hp77-creator commented 4 months ago

I have added the changes, please review. @Officeyutong @yunwei37

eunomia-bpf / bpftime