nalreddy commented 4 months ago

Describe the bug Trying to attach uprobe to application functions. Issue occurred when client trying to instrument application failed to inject or attach probe to certain functions.

Same bpf code able to attach uprobes to appilcation with kernel uprobes. After adding some debugs ,looks like we are hitting following issue GUM_INSTRUMENTATION_ERROR_WRONG_SIGNATURE.

To Reproduce Attaching sample programs user space bpf and kernel bpf code.

Application : https://github.com/ls1mardyn/ls1-mardyn

you may need to increase file descriptor limit.(system limits ulimit -n  ). below app with "*" , comes around 3500 functions (return probes 2X that atleast)
bpftime code fix  DEFAULT_MAX_FD = 4024 * 6

How to build :
make;g++ -g bpftracer.cpp -o bpftracer -lbpf -lelf Make sure bpftime bundled libbpf and system installed libbpf is same (libbpf.so.1.3.0)

How to run : ./bpftracer "absulute path to executable" "function pattern or name"

tracer_code.zip functionpattern "*" traces all functions in executable ".text" section.

Server : LD_PRELOAD=build/runtime/syscall-server/libbpftime-syscall-server.so /home/user/bpftracer

Client : application https://github.com/ls1mardyn/ls1-mardyn ( build clang one ) cd ls1-MarDyn/build/src LD_PRELOAD=/home/user/bpftime/build/runtime/agent/libbpftime-agent.so ./MarDyn ../../examples/EOX/305K_liq/config.xml --steps 50

nalreddy commented 4 months ago

adding sample code for reproducing issue tracer_code.zip

Officeyutong commented 4 months ago

I'm sorry that i'm a bit confusing. Could please make the following things clearer?

Which executable are you attaching to? (i.e the one injected by libbpftime-agent.so) How to get this executable?
What's the behavior of the failure and success?

nalreddy commented 4 months ago

Which executable : https://github.com/ls1mardyn/ls1-mardyn Build steps mentioned in Readme cd ls1-mardyn mkdir build cd build CC=clang CXX=clang++ ccmake .. make -j $(nproc)

How to run : standalone application

    export OMP_NUM_THREADS=4
    cd ls1-mardy/build/src
    ./MarDyn ../../examples/EOX/305K_liq/config.xml --steps 10

===================================================== Success : With mardyn application and above kernel bpf program mentioned. we are able to attach uprobes. Please refer above comments for how get bpf program , how to build, and run.

Failure : Tried same app with bpftime.

Server : LD_PRELOAD=build/runtime/syscall-server/libbpftime-syscall-server.so /home/user/bpftracer

cd ls1-MarDyn/build/src LD_PRELOAD=/home/user/bpftime/build/runtime/agent/libbpftime-agent.so ./MarDyn ../../examples/EOX/305K_liq/config.xml --steps 50

Error : Failed to attach uprobe to some functions ; error : Failed to execute frida gum_interceptor_attach for function 55b2caf0a3a0

case GUM_INSTRUMENTATION_ERROR_WRONG_SIGNATURE: result = GUM_ATTACH_WRONG_SIGNATURE;

Please do let me know if you are anymore queries.

nalreddy commented 3 months ago

@Officeyutong please let me know if you need more clarity . I am kind off stuck here , we had to delay integration of bpftime into our project.

bpftime working fine with other application but failing for this application. https://github.com/ls1mardyn/ls1-mardyn

Officeyutong commented 3 months ago

@Officeyutong please let me know if you need more clarity . I am kind off stuck here , we had to delay integration of bpftime into our project.

bpftime working fine with other application but failing for this application. https://github.com/ls1mardyn/ls1-mardyn

Sorry I'm too busy these days, I'll investigate into this issue and give you a reply in two days

Officeyutong commented 3 months ago

Which executable : https://github.com/ls1mardyn/ls1-mardyn Build steps mentioned in Readme cd ls1-mardyn mkdir build cd build CC=clang CXX=clang++ ccmake .. make -j $(nproc)

How to run : standalone application
    export OMP_NUM_THREADS=4
    cd ls1-mardy/build/src
    ./MarDyn ../../examples/EOX/305K_liq/config.xml --steps 10
===================================================== Success : With mardyn application and above kernel bpf program mentioned. we are able to attach uprobes. Please refer above comments for how get bpf program , how to build, and run.

Failure : Tried same app with bpftime.

Server : LD_PRELOAD=build/runtime/syscall-server/libbpftime-syscall-server.so /home/user/bpftracer

cd ls1-MarDyn/build/src LD_PRELOAD=/home/user/bpftime/build/runtime/agent/libbpftime-agent.so ./MarDyn ../../examples/EOX/305K_liq/config.xml --steps 50

Error : Failed to attach uprobe to some functions ; error : Failed to execute frida gum_interceptor_attach for function 55b2caf0a3a0

case GUM_INSTRUMENTATION_ERROR_WRONG_SIGNATURE: result = GUM_ATTACH_WRONG_SIGNATURE;

Please do let me know if you are anymore queries.

Could you please send me a precompiled (statically linked is better) bpftracer for x86-64? It's difficult to get your source compiled

nalreddy commented 3 months ago

Attaching bpftracer built with following tools ldd (Ubuntu GLIBC 2.37-0ubuntu2.2) 2.37 gcc version 9.5.0 (Ubuntu 9.5.0-3ubuntu1) Ubuntu clang version 15.0.7

Need root access and check ulimit limits for that user , may need to increase base on number of uprobes. by default attaching to all functions in symtab

For attaching normal kernel uprobes sudo ./bpftracer "path to binary"

bpftracer.zip

Officeyutong commented 3 months ago

Unable to repdocude except limitation of DEFAULT_MAX_FD

For limitation of DEFAULT_MAX_FD, you can change https://github.com/eunomia-bpf/bpftime/blob/87a8901d6eca582812ee84f8ab2ae3d10740adce/runtime/src/handler/handler_manager.hpp#L45 to a larger value and recompile to temporarily resolve it. In a future version, we will support specifying this value at runtime

I didn't encounter any issues related with attach, I have done the following test:

In terminal 1, run bpftime load ./bpftracer /root/bpftime-312/ls1-mardyn/build/src/MarDyn. bpftracer is the binary you have uploaded
In terminal 2, run bpftime start ./MarDyn ../../examples/EOX/305K_liq/config.xml --steps 50 After a few seconds, uprobe enter and uprobe return are kept being printed to console infinitely

bpftime is built with the newest commit.

So now I think there might be some issues on the binary you produced. Can you send me a copy of MarDyn, libbpftime-agent.so, libbpftime-syscall-server.so you compiled? (Debug profile is preferred)

nalreddy commented 3 months ago

I could attach uprobes for default binary and working fine. I will check if there are any issues with generated MarDyn binary. Thanks for looking into it and quick response.

sorry for deviating from main question :

Does bpftime support attaching multiple pids (of running processes)

like example sudo bpftime attach 101771,101772,...

https://github.com/eunomia-bpf/bpftime (Quick start section) You can also dynamically attach the eBPF program with a running process: Does with work with multiple pids, how do we provide

nalreddy commented 3 months ago

Hi @Officeyutong . I managed to create small reproducer for the issue.

Issue happening with AMD compiler(AOCC) you can download from here: https://www.amd.com/en/developer/aocc.html Frida seems to inject probes with wrong function address. AOCC compiler is llvm based.

Same application compiled with GCC and Clang working fine with bpftime.

Reproduced with : Compile below application or any small program with function with AOCC comipler (llvm based compiler). Try to attach function.

Bpf client side error :

2024-08-27 05:00:43][error][1401954] Failed to execute frida gum_interceptor_attach for function 555da3ee37b0 [2024-08-27 05:00:43][error][1401954] Unable to instantiate handlers: Failed to attach uprobe/uretprobe

Bpf Server side : Success

2024-08-27 05:56:45][info][1422282] Created uprobe/uretprobe perf event handler, module name /home/user/data/expr/probe_limits, offset 7b0 [2024-08-27 05:56:45][info][1422282] Created uprobe/uretprobe perf event handler, module name /home/user/data/expr/probe_limits, offset 7b0

Process maps :

Maps of app 555da3ee3000-555da3ee4000 r--p 00000000 08:04 28574075 /home/user/data/expr/probe_limits (555da3ee37b0 address in bpftime falls in non-executable region ?? ) 555da3ee4000-555da3ee5000 r-xp 00000000 08:04 28574075 /home/user/data/expr/probe_limits ( function address supposed to fall in this region ?) 555da3ee5000-555da3ee6000 r--p 00000000 08:04 28574075 /home/user/data/expr/probe_limits 555da3ee6000-555da3ee7000 rw-p 00000000 08:04 28574075 /home/user/data/expr/probe_limits 555da3ee7000-555da3ee8000 r--p 00000000 00:00 0 555da3ee8000-555da3eef000 rwxp 00000000 00:00 0 555da5bb3000-555da5bf5000 rw-p 00000000 00:00 0

Failed function address 555da3ee37b0, falling outside execution mapped region of process.

Server side attached using proper file offset
objdump -D -F -C binary

00000000000017b0 (File Offset: 0x7b0): we are u 17b0: c3 ret 17b1: 66 66 66 66 66 66 2e data16 data16 data16 data16 data16 cs nopw 0x0(%rax,%rax,1) 17b8: 0f 1f 84 00 00 00 00 17bf: 00

Application

include

void attribute ((noinline)) op_0() { for (int x = 0; x < 10; x++) { int sum = x+ 100; } return; }

int main()t { printf("test limits pid : %d\n",getpid()); getchar(); return 0; }

Officeyutong commented 2 months ago

I could attach uprobes for default binary and working fine. I will check if there are any issues with generated MarDyn binary. Thanks for looking into it and quick response.

sorry for deviating from main question :

Does bpftime support attaching multiple pids (of running processes)

like example sudo bpftime attach 101771,101772,...

https://github.com/eunomia-bpf/bpftime (Quick start section) You can also dynamically attach the eBPF program with a running process: Does with work with multiple pids, how do we provide

Just attach them seperately, call bpftime attach 101771 and bpftime attach 101772 and so on. Currently bpftime attach can only attach one pid at a time

Officeyutong commented 2 months ago

Hi @Officeyutong . I managed to create small reproducer for the issue.

Issue happening with AMD compiler(AOCC) you can download from here: https://www.amd.com/en/developer/aocc.html Frida seems to inject probes with wrong function address. AOCC compiler is llvm based.

Same application compiled with GCC and Clang working fine with bpftime.

Reproduced with : Compile below application or any small program with function with AOCC comipler (llvm based compiler). Try to attach function.

Bpf client side error :

2024-08-27 05:00:43][error][1401954] Failed to execute frida gum_interceptor_attach for function 555da3ee37b0 [2024-08-27 05:00:43][error][1401954] Unable to instantiate handlers: Failed to attach uprobe/uretprobe

Bpf Server side : Success

2024-08-27 05:56:45][info][1422282] Created uprobe/uretprobe perf event handler, module name /home/user/data/expr/probe_limits, offset 7b0 [2024-08-27 05:56:45][info][1422282] Created uprobe/uretprobe perf event handler, module name /home/user/data/expr/probe_limits, offset 7b0

Process maps :

Maps of app 555da3ee3000-555da3ee4000 r--p 00000000 08:04 28574075 /home/user/data/expr/probe_limits (555da3ee37b0 address in bpftime falls in non-executable region ?? ) 555da3ee4000-555da3ee5000 r-xp 00000000 08:04 28574075 /home/user/data/expr/probe_limits ( function address supposed to fall in this region ?) 555da3ee5000-555da3ee6000 r--p 00000000 08:04 28574075 /home/user/data/expr/probe_limits 555da3ee6000-555da3ee7000 rw-p 00000000 08:04 28574075 /home/user/data/expr/probe_limits 555da3ee7000-555da3ee8000 r--p 00000000 00:00 0 555da3ee8000-555da3eef000 rwxp 00000000 00:00 0 555da5bb3000-555da5bf5000 rw-p 00000000 00:00 0

Failed function address 555da3ee37b0, falling outside execution mapped region of process.

Server side attached using proper file offset objdump -D -F -C binary

00000000000017b0 (File Offset: 0x7b0): we are u 17b0: c3 ret 17b1: 66 66 66 66 66 66 2e data16 data16 data16 data16 data16 cs nopw 0x0(%rax,%rax,1) 17b8: 0f 1f 84 00 00 00 00 17bf: 00

Application

include #include

void attribute ((noinline)) op_0() { for (int x = 0; x < 10; x++) { int sum = x+ 100; } return; }

int main()t { printf("test limits pid : %d\n",getpid()); getchar(); return 0; }

Will AOCC work on an Intel CPU? I don't have any AMD cpu machine now

nalreddy commented 2 months ago

AOCC should work with intel CPU, its same x86 . Except for some special optimization flags.

above application compiled with aocc/clang application.c -o application

Make sure its aocc/clange not system default clang . Please do "which clang" and Give absolute path of AOCC compiler.

nalreddy commented 2 months ago

Hi @Officeyutong . I managed to create small reproducer for the issue. Issue happening with AMD compiler(AOCC) you can download from here: https://www.amd.com/en/developer/aocc.html Frida seems to inject probes with wrong function address. AOCC compiler is llvm based. Same application compiled with GCC and Clang working fine with bpftime. Reproduced with : Compile below application or any small program with function with AOCC comipler (llvm based compiler). Try to attach function. Bpf client side error : 2024-08-27 05:00:43][error][1401954] Failed to execute frida gum_interceptor_attach for function 555da3ee37b0 [2024-08-27 05:00:43][error][1401954] Unable to instantiate handlers: Failed to attach uprobe/uretprobe Bpf Server side : Success 2024-08-27 05:56:45][info][1422282] Created uprobe/uretprobe perf event handler, module name /home/user/data/expr/probe_limits, offset 7b0 [2024-08-27 05:56:45][info][1422282] Created uprobe/uretprobe perf event handler, module name /home/user/data/expr/probe_limits, offset 7b0 Process maps : Maps of app 555da3ee3000-555da3ee4000 r--p 00000000 08:04 28574075 /home/user/data/expr/probe_limits (555da3ee37b0 address in bpftime falls in non-executable region ?? ) 555da3ee4000-555da3ee5000 r-xp 00000000 08:04 28574075 /home/user/data/expr/probe_limits ( function address supposed to fall in this region ?) 555da3ee5000-555da3ee6000 r--p 00000000 08:04 28574075 /home/user/data/expr/probe_limits 555da3ee6000-555da3ee7000 rw-p 00000000 08:04 28574075 /home/user/data/expr/probe_limits 555da3ee7000-555da3ee8000 r--p 00000000 00:00 0 555da3ee8000-555da3eef000 rwxp 00000000 00:00 0 555da5bb3000-555da5bf5000 rw-p 00000000 00:00 0 Failed function address 555da3ee37b0, falling outside execution mapped region of process. Server side attached using proper file offset objdump -D -F -C binary 00000000000017b0 (File Offset: 0x7b0): we are u 17b0: c3 ret 17b1: 66 66 66 66 66 66 2e data16 data16 data16 data16 data16 cs nopw 0x0(%rax,%rax,1) 17b8: 0f 1f 84 00 00 00 00 17bf: 00 Application

include #include

void attribute ((noinline)) op_0() { for (int x = 0; x < 10; x++) { int sum = x+ 100; } return; } int main()t { printf("test limits pid : %d\n",getpid()); getchar(); return 0; }

Will AOCC work on an Intel CPU? I don't have any AMD cpu machine now

"yes, AOCC should work with intel CPU, its same x86 . Except for some special optimization flags.

above application compiled with aocc/clang application.c -o application

Make sure its aocc/clange not system default clang . Please do "which clang" and Give absolute path of AOCC compiler."

nalreddy commented 2 months ago

@Officeyutong
Our current product ,bpf programs run without sudo ( we set bpf_cap to executable before running the bpf program).

How do we achieve same with bpftime .

Are there any places we have to run bpftime with sudo (must ?? ) from : https://eunomia.dev/bpftime/documents/usage/

Modes of running bpftime

1. ./example/malloc/victim & echo $! # The pid is 101771

attach to it:

**$ sudo bpftime attach 101771 # You may need to run make install in root Inject: "/root/.bpftime/libbpftime-agent.so" Successfully injected. ID: 1**

2. bpftime load ./example/malloc/malloc bpftime start ./example/malloc/victim

3. Run with LD_PRELOAD directly

Why do we need to use sudo in 1st method , is it necessary and also System call tracing examples are run with Sudo , is it must.

Could you please let me know which cases we need to use sudo and which use case not.

We would like to avoid use of Sudo along with bpf programs. (run bpf program as non root).

we would like to use attach method without sudo , is it possible.

eunomia-bpf / bpftime

[BUG] Failed to attach uprobe at client side #312

include

include

include #include

include #include