iovisor / ply

Dynamic Tracing in Linux
GNU General Public License v2.0
969 stars 90 forks source link

Add BEGIN and END providers #74

Closed namhyung closed 2 years ago

namhyung commented 2 years ago

Like awk(1) and bpftrace(8), add BEGIN and END (case-sensitive) providers to do things before and after other providers. This can be useful to print some header message and/or set up maps to predefined values.

The way it's implemented is to use basic uprobe infra (using xprobe code). Add two empty functions for each trigger and call it when probes are added. But I needed to calculate (file) offsets of the functions without having an ELF parser. So I used auxv to get ELF program header (and its virtual address). Not sure if there's a better way but it works for me. :)

Also I added the BEGIN and END keywords as PSPEC without a colon to match with the existing convention in awk/bpftrace.

For example:

# cat opensnoop.ply
BEGIN {
  printf("%7s  %16s  %s\n", "PID", "COMM", "FILE");
}

kprobe:__x64_sys_open {
  printf("%7d  %16s  %s\n", pid, comm, str(arg1));
}

END {
  printf("DONE\n");
}

The output looks like:


# ply -c 'sleep 1' opensnoop.ply 
    PID              COMM  FILE
1923473                sh  /etc/ld.so.cache
1923473                sh  /lib/x86_64-linux-gnu/libtinfo.so.6
2098992   futex-default-S  /proc/2299637
2098992   futex-default-S  /proc/2299650
2098992   futex-default-S  /proc/2299645
2098992   futex-default-S  /proc/2299643
2098992   futex-default-S  /proc/2299651
  32147   systemd-journal  /proc/1034/comm
  32147   systemd-journal  /proc/1034/cmdline
  32147   systemd-journal  /proc/1034/status
DONE
namhyung commented 2 years ago

It seems it failed to attach to the uprobe events. Will take a look..

wkz commented 2 years ago

Love the way you solved this! Here I was, thinking about implementing an interpreter to run in these scenarios - this is much better!

I just have one concern: IIRC, uprobes are attached to a specific binary/solib rather than a process. So don't we need a filter that ensures that the begin/end-probes are associated with the correct instance of ply? (In the, admittedly unlikely, scenario where you have multiple ply processes running concurrently) I.e. there should be an implicit if (pid == mypid) compiled in at the start of every begin/end.

Re: checks: The x86_64 image did not have uprobes compiled in, I will fix that ASAP. Not obvious why fhe other targets failed. I guess you are running on x86_64 exclusively?

namhyung commented 2 years ago

Thanks, I was testing on my laptop mainly as I found it difficult to run the tests with modified code. But it seems to work after deleting test/work directory. It'd be nice if it can detect it automatically.

For the pid check, that's why I changed the perf_event_attach to work on a process only. But actually I'm not sure BPF would work that way. So I'll take a look at how to add the if statement as you said.

And I found a bug in calculating offset for the begin/end symbol. It should use phdr->p_offset instead of phdr->p_vaddr. My laptop build PIE only and they were same so I missed it. Also I added a NULL check for the ply->stdbuf.

But armv5 and armv7 were stuck in the test. Not sure what' the problem...

wkz commented 2 years ago

I do not feel comfortable merging this until we know that we can make it work on all architectures.

I will try to find some time to help out with debugging this on ARM.

namhyung commented 2 years ago

OK, I changed it to parse '/proc/self/maps' to get the base address instead of using the auxv. It seems ARM binary layout is little bit different than others. Anyway it now works on ARM too.

Still I don't have the pid check. Maybe it needs provider->rewrite() callback.

wkz commented 2 years ago

I think specifying pid to perf_event_open should be enough.

Thanks!