search

DataDog / datadog-process-agent

Datadog Process Agent
https://datadoghq.com
20 stars 9 forks source link

[network-tracer] Allow to enable debug logs at runtime #285

Closed sfluor closed 5 years ago

sfluor commented 5 years ago

This allows to have debug logs output in /sys/kernel/debug/tracing/trace_pipe when enabling network_bpf_debug, for instance:

rake ebpf:nettop

Output example:


$ sudo cat /sys/kernel/debug/tracing/trace_pipe | grep -v tcp_send
           <...>-28184 [001] ....  6126.735061: 0x00000001: kprobe/tcp_close: pid_tgid: 28184, ns: -268435303
            sshd-1910  [001] ....  6132.176617: 0x00000001: kprobe/tcp_cleanup_rbuf: pid_tgid: 1910, copied: 72
            sshd-1910  [001] ....  6132.473164: 0x00000001: kprobe/tcp_cleanup_rbuf: pid_tgid: 1910, copied: 36
            sshd-1910  [001] ....  6132.474111: 0x00000001: kprobe/tcp_cleanup_rbuf: pid_tgid: 1910, copied: 152
            sshd-1910  [000] ....  6132.908773: 0x00000001: kprobe/tcp_cleanup_rbuf: pid_tgid: 1910, copied: 216
            sshd-1910  [000] ....  6133.576691: 0x00000001: kprobe/tcp_cleanup_rbuf: pid_tgid: 1910, copied: 324
            sshd-1910  [000] ....  6135.714521: 0x00000001: kprobe/tcp_cleanup_rbuf: pid_tgid: 1910, copied: 36
            sshd-1910  [000] ....  6135.794720: 0x00000001: kprobe/tcp_cleanup_rbuf: pid_tgid: 1910, copied: 36
            sshd-1910  [000] ....  6135.930663: 0x00000001: kprobe/tcp_cleanup_rbuf: pid_tgid: 1910, copied: 36
            sshd-1910  [000] ....  6136.023574: 0x00000001: kprobe/tcp_cleanup_rbuf: pid_tgid: 1910, copied: 36
            sshd-1910  [000] ....  6136.153803: 0x00000001: kprobe/tcp_cleanup_rbuf: pid_tgid: 1910, copied: 36
            sshd-1910  [000] ....  6136.264604: 0x00000001: kprobe/tcp_cleanup_rbuf: pid_tgid: 1910, copied: 36
            sshd-1910  [000] ....  6136.344841: 0x00000001: kprobe/tcp_cleanup_rbuf: pid_tgid: 1910, copied: 36
            sshd-1910  [000] ....  6136.480980: 0x00000001: kprobe/tcp_cleanup_rbuf: pid_tgid: 1910, copied: 36
            sshd-1910  [000] ....  6136.591774: 0x00000001: kprobe/tcp_cleanup_rbuf: pid_tgid: 1910, copied: 36
            sshd-1910  [000] ....  6136.691593: 0x00000001: kprobe/tcp_cleanup_rbuf: pid_tgid: 1910, copied: 36
            sshd-1910  [001] ....  6136.828420: 0x00000001: kprobe/tcp_cleanup_rbuf: pid_tgid: 1910, copied: 36
            sshd-1910  [001] ....  6137.230548: 0x00000001: kprobe/tcp_cleanup_rbuf: pid_tgid: 1910, copied: 36
            sshd-1910  [001] ....  6137.390314: 0x00000001: kprobe/tcp_cleanup_rbuf: pid_tgid: 1910, copied: 36
            sshd-1910  [001] ....  6137.454566: 0x00000001: kprobe/tcp_cleanup_rbuf: pid_tgid: 1910, copied: 36
            sshd-1910  [001] ....  6137.595633: 0x00000001: kprobe/tcp_cleanup_rbuf: pid_tgid: 1910, copied: 36
            curl-28202 [001] ....  6137.613937: 0x00000001: kprobe/udp_sendmsg: pid_tgid: 28202, size: 27
 systemd-resolve-668   [001] ....  6137.614930: 0x00000001: kprobe/udp_recvmsg: pid_tgid: 668
 systemd-resolve-668   [001] ....  6137.614999: 0x00000001: kprobe/udp_recvmsg: pid_tgid: 668
 systemd-resolve-668   [001] ....  6137.615175: 0x00000001: kprobe/udp_sendmsg: pid_tgid: 668, size: 61
 systemd-resolve-668   [001] ....  6137.615340: 0x00000001: kprobe/udp_sendmsg: pid_tgid: 668, size: 61
            curl-28202 [001] ....  6137.616450: 0x00000001: kprobe/udp_sendmsg: pid_tgid: 28202, size: 27
 systemd-resolve-668   [001] ....  6137.616954: 0x00000001: kprobe/udp_recvmsg: pid_tgid: 668
 systemd-resolve-668   [001] ....  6137.616976: 0x00000001: kprobe/udp_recvmsg: pid_tgid: 668
 systemd-resolve-668   [001] ....  6137.617130: 0x00000001: kprobe/udp_sendmsg: pid_tgid: 668, size: 61
 systemd-resolve-668   [001] ....  6137.617258: 0x00000001: kprobe/udp_sendmsg: pid_tgid: 668, size: 61
 systemd-resolve-668   [001] ....  6137.631124: 0x00000001: kprobe/udp_recvmsg: pid_tgid: 668
 systemd-resolve-668   [001] ....  6137.631155: 0x00000001: kprobe/udp_recvmsg: pid_tgid: 668
 systemd-resolve-668   [001] ....  6137.631251: 0x00000001: kprobe/udp_sendmsg: pid_tgid: 668, size: 61
 systemd-resolve-668   [001] ....  6137.631398: 0x00000001: kprobe/udp_recvmsg: pid_tgid: 668
 systemd-resolve-668   [001] ....  6137.631406: 0x00000001: kprobe/udp_recvmsg: pid_tgid: 668
 systemd-resolve-668   [001] ....  6137.631540: 0x00000001: kprobe/udp_sendmsg: pid_tgid: 668, size: 61
 systemd-resolve-668   [001] ....  6137.640161: 0x00000001: kprobe/udp_recvmsg: pid_tgid: 668
 systemd-resolve-668   [001] ....  6137.640242: 0x00000001: kprobe/udp_recvmsg: pid_tgid: 668
 systemd-resolve-668   [001] ....  6137.642760: 0x00000001: kprobe/udp_recvmsg: pid_tgid: 668
 systemd-resolve-668   [001] ....  6137.642840: 0x00000001: kprobe/udp_recvmsg: pid_tgid: 668
 systemd-resolve-668   [001] ....  6137.655755: 0x00000001: kprobe/udp_recvmsg: pid_tgid: 668
 systemd-resolve-668   [001] ....  6137.655852: 0x00000001: kprobe/udp_recvmsg: pid_tgid: 668
 systemd-resolve-668   [001] ....  6137.656431: 0x00000001: kprobe/udp_sendmsg: pid_tgid: 668, size: 43
 systemd-resolve-668   [001] ....  6137.656664: 0x00000001: kprobe/udp_sendmsg: pid_tgid: 668, size: 55
            curl-28202 [001] ....  6137.658276: 0x00000001: kprobe/udp_recvmsg: pid_tgid: 28202
            curl-28202 [001] ....  6137.658914: 0x00000001: kprobe/udp_recvmsg: pid_tgid: 28202
            curl-28201 [001] ....  6137.725226: 0x00000001: kprobe/tcp_cleanup_rbuf: pid_tgid: 28201, copied: 526
            curl-28201 [001] ....  6137.725385: 0x00000001: kprobe/tcp_close: pid_tgid: 28201, ns: -268435303
            sshd-1910  [001] ....  6138.976211: 0x00000001: kprobe/tcp_cleanup_rbuf: pid_tgid: 1910, copied: 108```
kevinconaway commented 5 years ago

Does it make sense to use a runtime flag for this instead?

sfluor commented 5 years ago

👍

sunhay commented 5 years ago

Should we generate two versions of the ELF object file? One with debugging enabled and one without to allow for enabling debug logs via a config option?

sfluor commented 5 years ago

The size did not really changed (18M before and 18M now)

sfluor commented 5 years ago

Nice idea 👍 ! I will add that in a follow up PR