Open nalreddy opened 1 month ago
adding sample code for reproducing issue tracer_code.zip
I'm sorry that i'm a bit confusing. Could please make the following things clearer?
Which executable : https://github.com/ls1mardyn/ls1-mardyn Build steps mentioned in Readme cd ls1-mardyn mkdir build cd build CC=clang CXX=clang++ ccmake .. make -j $(nproc)
How to run : standalone application
export OMP_NUM_THREADS=4
cd ls1-mardy/build/src
./MarDyn ../../examples/EOX/305K_liq/config.xml --steps 10
===================================================== Success : With mardyn application and above kernel bpf program mentioned. we are able to attach uprobes. Please refer above comments for how get bpf program , how to build, and run.
Failure : Tried same app with bpftime.
Server : LD_PRELOAD=build/runtime/syscall-server/libbpftime-syscall-server.so /home/user/bpftracer
cd ls1-MarDyn/build/src LD_PRELOAD=/home/user/bpftime/build/runtime/agent/libbpftime-agent.so ./MarDyn ../../examples/EOX/305K_liq/config.xml --steps 50
Error : Failed to attach uprobe to some functions ; error : Failed to execute frida gum_interceptor_attach for function 55b2caf0a3a0
case GUM_INSTRUMENTATION_ERROR_WRONG_SIGNATURE: result = GUM_ATTACH_WRONG_SIGNATURE;
Please do let me know if you are anymore queries.
@Officeyutong please let me know if you need more clarity . I am kind off stuck here , we had to delay integration of bpftime into our project.
bpftime working fine with other application but failing for this application. https://github.com/ls1mardyn/ls1-mardyn
@Officeyutong please let me know if you need more clarity . I am kind off stuck here , we had to delay integration of bpftime into our project.
bpftime working fine with other application but failing for this application. https://github.com/ls1mardyn/ls1-mardyn
Sorry I'm too busy these days, I'll investigate into this issue and give you a reply in two days
Which executable : https://github.com/ls1mardyn/ls1-mardyn Build steps mentioned in Readme cd ls1-mardyn mkdir build cd build CC=clang CXX=clang++ ccmake .. make -j $(nproc)
How to run : standalone application
export OMP_NUM_THREADS=4 cd ls1-mardy/build/src ./MarDyn ../../examples/EOX/305K_liq/config.xml --steps 10
===================================================== Success : With mardyn application and above kernel bpf program mentioned. we are able to attach uprobes. Please refer above comments for how get bpf program , how to build, and run.
Failure : Tried same app with bpftime.
Server : LD_PRELOAD=build/runtime/syscall-server/libbpftime-syscall-server.so /home/user/bpftracer
cd ls1-MarDyn/build/src LD_PRELOAD=/home/user/bpftime/build/runtime/agent/libbpftime-agent.so ./MarDyn ../../examples/EOX/305K_liq/config.xml --steps 50
Error : Failed to attach uprobe to some functions ; error : Failed to execute frida gum_interceptor_attach for function 55b2caf0a3a0
case GUM_INSTRUMENTATION_ERROR_WRONG_SIGNATURE: result = GUM_ATTACH_WRONG_SIGNATURE;
Please do let me know if you are anymore queries.
Could you please send me a precompiled (statically linked is better) bpftracer for x86-64? It's difficult to get your source compiled
Attaching bpftracer built with following tools ldd (Ubuntu GLIBC 2.37-0ubuntu2.2) 2.37 gcc version 9.5.0 (Ubuntu 9.5.0-3ubuntu1) Ubuntu clang version 15.0.7
Need root access and check ulimit limits for that user , may need to increase base on number of uprobes. by default attaching to all functions in symtab
For attaching normal kernel uprobes sudo ./bpftracer "path to binary"
Unable to repdocude except limitation of DEFAULT_MAX_FD
DEFAULT_MAX_FD
, you can change https://github.com/eunomia-bpf/bpftime/blob/87a8901d6eca582812ee84f8ab2ae3d10740adce/runtime/src/handler/handler_manager.hpp#L45 to a larger value and recompile to temporarily resolve it. In a future version, we will support specifying this value at runtimeI didn't encounter any issues related with attach, I have done the following test:
bpftime load ./bpftracer /root/bpftime-312/ls1-mardyn/build/src/MarDyn
. bpftracer
is the binary you have uploadedbpftime start ./MarDyn ../../examples/EOX/305K_liq/config.xml --steps 50
After a few seconds, uprobe enter
and uprobe return
are kept being printed to console infinitelybpftime is built with the newest commit.
So now I think there might be some issues on the binary you produced. Can you send me a copy of MarDyn
, libbpftime-agent.so
, libbpftime-syscall-server.so
you compiled? (Debug profile is preferred)
I could attach uprobes for default binary and working fine. I will check if there are any issues with generated MarDyn binary. Thanks for looking into it and quick response.
sorry for deviating from main question :
Does bpftime support attaching multiple pids (of running processes)
like example sudo bpftime attach 101771,101772,...
https://github.com/eunomia-bpf/bpftime (Quick start section) You can also dynamically attach the eBPF program with a running process: Does with work with multiple pids, how do we provide
Hi @Officeyutong . I managed to create small reproducer for the issue.
Issue happening with AMD compiler(AOCC) you can download from here: https://www.amd.com/en/developer/aocc.html Frida seems to inject probes with wrong function address. AOCC compiler is llvm based.
Same application compiled with GCC and Clang working fine with bpftime.
Reproduced with : Compile below application or any small program with function with AOCC comipler (llvm based compiler). Try to attach function.
Bpf client side error :
2024-08-27 05:00:43][error][1401954] Failed to execute frida gum_interceptor_attach for function 555da3ee37b0 [2024-08-27 05:00:43][error][1401954] Unable to instantiate handlers: Failed to attach uprobe/uretprobe
Bpf Server side : Success
2024-08-27 05:56:45][info][1422282] Created uprobe/uretprobe perf event handler, module name /home/user/data/expr/probe_limits, offset 7b0 [2024-08-27 05:56:45][info][1422282] Created uprobe/uretprobe perf event handler, module name /home/user/data/expr/probe_limits, offset 7b0
Process maps :
Maps of app 555da3ee3000-555da3ee4000 r--p 00000000 08:04 28574075 /home/user/data/expr/probe_limits (555da3ee37b0 address in bpftime falls in non-executable region ?? ) 555da3ee4000-555da3ee5000 r-xp 00000000 08:04 28574075 /home/user/data/expr/probe_limits ( function address supposed to fall in this region ?) 555da3ee5000-555da3ee6000 r--p 00000000 08:04 28574075 /home/user/data/expr/probe_limits 555da3ee6000-555da3ee7000 rw-p 00000000 08:04 28574075 /home/user/data/expr/probe_limits 555da3ee7000-555da3ee8000 r--p 00000000 00:00 0 555da3ee8000-555da3eef000 rwxp 00000000 00:00 0 555da5bb3000-555da5bf5000 rw-p 00000000 00:00 0
Failed function address 555da3ee37b0, falling outside execution mapped region of process.
Server side attached using proper file offset
objdump -D -F -C binary
00000000000017b0
Application
void attribute ((noinline)) op_0() { for (int x = 0; x < 10; x++) { int sum = x+ 100; } return; }
int main()t { printf("test limits pid : %d\n",getpid()); getchar(); return 0; }
I could attach uprobes for default binary and working fine. I will check if there are any issues with generated MarDyn binary. Thanks for looking into it and quick response.
sorry for deviating from main question :
Does bpftime support attaching multiple pids (of running processes)
like example sudo bpftime attach 101771,101772,...
https://github.com/eunomia-bpf/bpftime (Quick start section) You can also dynamically attach the eBPF program with a running process: Does with work with multiple pids, how do we provide
Just attach them seperately, call bpftime attach 101771
and bpftime attach 101772
and so on. Currently bpftime attach
can only attach one pid at a time
Hi @Officeyutong . I managed to create small reproducer for the issue.
Issue happening with AMD compiler(AOCC) you can download from here: https://www.amd.com/en/developer/aocc.html Frida seems to inject probes with wrong function address. AOCC compiler is llvm based.
Same application compiled with GCC and Clang working fine with bpftime.
Reproduced with : Compile below application or any small program with function with AOCC comipler (llvm based compiler). Try to attach function.
Bpf client side error :
2024-08-27 05:00:43][error][1401954] Failed to execute frida gum_interceptor_attach for function 555da3ee37b0 [2024-08-27 05:00:43][error][1401954] Unable to instantiate handlers: Failed to attach uprobe/uretprobe
Bpf Server side : Success
2024-08-27 05:56:45][info][1422282] Created uprobe/uretprobe perf event handler, module name /home/user/data/expr/probe_limits, offset 7b0 [2024-08-27 05:56:45][info][1422282] Created uprobe/uretprobe perf event handler, module name /home/user/data/expr/probe_limits, offset 7b0
Process maps :
Maps of app 555da3ee3000-555da3ee4000 r--p 00000000 08:04 28574075 /home/user/data/expr/probe_limits (555da3ee37b0 address in bpftime falls in non-executable region ?? ) 555da3ee4000-555da3ee5000 r-xp 00000000 08:04 28574075 /home/user/data/expr/probe_limits ( function address supposed to fall in this region ?) 555da3ee5000-555da3ee6000 r--p 00000000 08:04 28574075 /home/user/data/expr/probe_limits 555da3ee6000-555da3ee7000 rw-p 00000000 08:04 28574075 /home/user/data/expr/probe_limits 555da3ee7000-555da3ee8000 r--p 00000000 00:00 0 555da3ee8000-555da3eef000 rwxp 00000000 00:00 0 555da5bb3000-555da5bf5000 rw-p 00000000 00:00 0
Failed function address 555da3ee37b0, falling outside execution mapped region of process.
Server side attached using proper file offset objdump -D -F -C binary
00000000000017b0
(File Offset: 0x7b0): we are u 17b0: c3 ret 17b1: 66 66 66 66 66 66 2e data16 data16 data16 data16 data16 cs nopw 0x0(%rax,%rax,1) 17b8: 0f 1f 84 00 00 00 00 17bf: 00 Application
include
#include void attribute ((noinline)) op_0() { for (int x = 0; x < 10; x++) { int sum = x+ 100; } return; }
int main()t { printf("test limits pid : %d\n",getpid()); getchar(); return 0; }
Will AOCC work on an Intel CPU? I don't have any AMD cpu machine now
AOCC should work with intel CPU, its same x86 . Except for some special optimization flags.
above application compiled with aocc/clang application.c -o application
Make sure its aocc/clange not system default clang . Please do "which clang" and Give absolute path of AOCC compiler.
Hi @Officeyutong . I managed to create small reproducer for the issue. Issue happening with AMD compiler(AOCC) you can download from here: https://www.amd.com/en/developer/aocc.html Frida seems to inject probes with wrong function address. AOCC compiler is llvm based. Same application compiled with GCC and Clang working fine with bpftime. Reproduced with : Compile below application or any small program with function with AOCC comipler (llvm based compiler). Try to attach function. Bpf client side error : 2024-08-27 05:00:43][error][1401954] Failed to execute frida gum_interceptor_attach for function 555da3ee37b0 [2024-08-27 05:00:43][error][1401954] Unable to instantiate handlers: Failed to attach uprobe/uretprobe Bpf Server side : Success 2024-08-27 05:56:45][info][1422282] Created uprobe/uretprobe perf event handler, module name /home/user/data/expr/probe_limits, offset 7b0 [2024-08-27 05:56:45][info][1422282] Created uprobe/uretprobe perf event handler, module name /home/user/data/expr/probe_limits, offset 7b0 Process maps : Maps of app 555da3ee3000-555da3ee4000 r--p 00000000 08:04 28574075 /home/user/data/expr/probe_limits (555da3ee37b0 address in bpftime falls in non-executable region ?? ) 555da3ee4000-555da3ee5000 r-xp 00000000 08:04 28574075 /home/user/data/expr/probe_limits ( function address supposed to fall in this region ?) 555da3ee5000-555da3ee6000 r--p 00000000 08:04 28574075 /home/user/data/expr/probe_limits 555da3ee6000-555da3ee7000 rw-p 00000000 08:04 28574075 /home/user/data/expr/probe_limits 555da3ee7000-555da3ee8000 r--p 00000000 00:00 0 555da3ee8000-555da3eef000 rwxp 00000000 00:00 0 555da5bb3000-555da5bf5000 rw-p 00000000 00:00 0 Failed function address 555da3ee37b0, falling outside execution mapped region of process. Server side attached using proper file offset objdump -D -F -C binary 00000000000017b0
(File Offset: 0x7b0): we are u 17b0: c3 ret 17b1: 66 66 66 66 66 66 2e data16 data16 data16 data16 data16 cs nopw 0x0(%rax,%rax,1) 17b8: 0f 1f 84 00 00 00 00 17bf: 00 Application include
#include void attribute ((noinline)) op_0() { for (int x = 0; x < 10; x++) { int sum = x+ 100; } return; } int main()t { printf("test limits pid : %d\n",getpid()); getchar(); return 0; }
Will AOCC work on an Intel CPU? I don't have any AMD cpu machine now
"yes, AOCC should work with intel CPU, its same x86 . Except for some special optimization flags.
above application compiled with aocc/clang application.c -o application
Make sure its aocc/clange not system default clang . Please do "which clang" and Give absolute path of AOCC compiler."
Describe the bug Trying to attach uprobe to application functions. Issue occurred when client trying to instrument application failed to inject or attach probe to certain functions.
Same bpf code able to attach uprobes to appilcation with kernel uprobes. After adding some debugs ,looks like we are hitting following issue GUM_INSTRUMENTATION_ERROR_WRONG_SIGNATURE.
To Reproduce Attaching sample programs user space bpf and kernel bpf code.
Application : https://github.com/ls1mardyn/ls1-mardyn
How to build :
make;g++ -g bpftracer.cpp -o bpftracer -lbpf -lelf Make sure bpftime bundled libbpf and system installed libbpf is same (libbpf.so.1.3.0)
How to run : ./bpftracer "absulute path to executable" "function pattern or name"
tracer_code.zip functionpattern "*" traces all functions in executable ".text" section.
Server : LD_PRELOAD=build/runtime/syscall-server/libbpftime-syscall-server.so /home/user/bpftracer
Client : application https://github.com/ls1mardyn/ls1-mardyn ( build clang one ) cd ls1-MarDyn/build/src LD_PRELOAD=/home/user/bpftime/build/runtime/agent/libbpftime-agent.so ./MarDyn ../../examples/EOX/305K_liq/config.xml --steps 50