add sigqueue instrumentation to killsnoop-bpfcc

[vladak]

As noted in https://stackoverflow.com/a/62434615/11582827 the killsnoop-bpfcc does not capture signals sent via the sigqueue library call, i.e. using the rt_sigqueueinfo system call.

Running https://github.com/devnull-cz/unix-linux-prog-in-c-src/blob/701db3599460ab03ed1aa45a8bc7e1dccb462588/signals/sigqueue.c will send the signal via sigqueue(3):

$ cc sigqueue.c
$ ./a.out 
2043940 sleeping
2043941 waiting
2043940 sending SIGINT signal to 2043941
2043941 exiting with received 42

and using the killsnoop-bpfcc program with simple modification to instrument the rt_sigqueueinfo syscall, it now displays the signal:

$ sudo ./killsnoop-bpfcc 
TIME      PID    COMM             SIG  TPID   RESULT
15:33:44  2043940 a.out            2    2043941 0

[vladak]

Here's what I used to make it work:

--- /usr/sbin/killsnoop-bpfcc   2020-02-07 14:31:42.000000000 +0100
+++ killsnoop-bpfcc 2021-07-01 15:33:40.985373732 +0200
@@ -60,6 +60,21 @@
 BPF_HASH(infotmp, u32, struct val_t);
 BPF_PERF_OUTPUT(events);

+int syscall__sigqueue(struct pt_regs *ctx, int tpid, int sig, siginfo_t *info)
+{
+    u32 pid = bpf_get_current_pid_tgid();
+    FILTER
+
+    struct val_t val = {.pid = pid};
+    if (bpf_get_current_comm(&val.comm, sizeof(val.comm)) == 0) {
+        val.tpid = tpid;
+        val.sig = sig;
+        infotmp.update(&pid, &val);
+    }
+
+    return 0;
+};
+
 int syscall__kill(struct pt_regs *ctx, int tpid, int sig)
 {
     u32 pid = bpf_get_current_pid_tgid();
@@ -114,6 +129,9 @@
 kill_fnname = b.get_syscall_fnname("kill")
 b.attach_kprobe(event=kill_fnname, fn_name="syscall__kill")
 b.attach_kretprobe(event=kill_fnname, fn_name="do_ret_sys_kill")
+sigqueue_fnname = b.get_syscall_fnname("rt_sigqueueinfo")
+b.attach_kprobe(event=sigqueue_fnname, fn_name="syscall__sigqueue")
+b.attach_kretprobe(event=sigqueue_fnname, fn_name="do_ret_sys_kill")

 # header
 print("%-9s %-6s %-16s %-4s %-6s %s" % (

[vladak]

There is also the rt_tgsigqueueinfo syscall so perhaps the script should be renamed to sigsnoop-bpfcc if the changes go in.

[yonghong-song]

I check the kernel signal.c file, besides kill, rt_sigqueueinfo and rt_tgsigqueueinfo, there are three more syscalls which could send signals pidfd_send_signal, tgkill and tkill. I do agree that sigsnoop might be a better tool name and we can add more functionality there to cover more use cases.

But kill still the most common way to send signals and killsnoop has been around for some time and complete rename might not be good? Maybe we could do the following:

• implement sigsnoop.py tool which cover more send signal syscalls then just kill
• implement killsnoop as sigsnoop.py --kill-only [other parameters].

Or we just add functionality to killsnoop.py. It is fine to me too as long as we have some explanations in man page, example and killsnoop.py --help output.

@brendangregg what do you think?

[chenhengqi]

@yonghong-song

I tried to implement what you suggested here as a libbpf tool, see #3570, but it seems like that these syscall tracepoints are NOT triggered by the kernel.

I notice that there is a signal/signal_generate tracepoint, which might be a promising target to instrument.

https://github.com/torvalds/linux/blob/85a90500f9a1717c4e142ce92e6c1cb1a339ec78/include/trace/events/signal.h#L36-L80

[yonghong-song]

Agree. indeed signal/signal_generate is better and stable to catch all kill events (probably more), but I think it should be fine.

[brendangregg]

I just debugged something that was tgkill(2), so killsnoop didn't see it. I filed #3592.

Looks like in the short-term we should have a killsnoop(8) that does all the kill* syscalls, and sigsnoop(8) for the kernel catch-all. Maybe in the long term future (years away) we can retire killsnoop(8); give me some time to start documenting sigsnoop(8).