iovisor / kubectl-trace

Schedule bpftrace programs on your kubernetes cluster using the kubectl
MIT License
2.04k stars 165 forks source link

No bpf execution result displayed #177

Open no1asian opened 2 years ago

no1asian commented 2 years ago

I've install kubectl-trace without error in my K8S cluster.

But no result displayed.

$ kubectl trace run ip-10-250-27-39.eu-central-1.compute.internal -e "tracepoint:syscalls:sysenter* { @[probe] = count(); }" trace e894f5e4-fd28-40a6-bce8-ed3d11c082d4 created

Finished without result data. Any other method or path to check the result?

rainest commented 2 years ago

Are you able to see the trace containers/jobs' output? I had a similar issue running under KIND, where I lacked kernel headers, causing the trace pods to end quickly. It looks like the trace plugin isn't quite able to properly deal with situations where its pods disappear, so it just hangs there and shows nothing, or will indicate it can't find any trace if you try trace logs.

The jobs use a short ttlSecondsAfterFinished value, so seeing the affected pods is a bit difficult. The best option I've found is immediately running kubectl edit job immediately after starting trace and deleting that line, but that probably won't work as well on a cluster with other activity.

--fetch-headers should fix this, but it's a bit outdated at the moment: https://github.com/iovisor/kubectl-trace/pull/179

mYu4N commented 1 year ago

run trace: ~]# kubectl trace run cn-beijing.192.168.88.23 -e "tracepoint:syscalls:sysenter* { @[probe] = count(); }" trace b4bf4d65-957d-11ed-8c73-00163e0c327b created

log pod: ~]# kubectl logs kubectl-trace-b4bf4d65-957d-11ed-8c73-00163e0c327b-9w9z6 if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two times Attaching 336 probes...

login node ,and use kill send sigint : ~]# pgrep bpftrace 252665 ~]# pstree -sp 252665 systemd(1)───containerd-shim(252587)───timeout(252641)───trace-runner(252653)───bpftrace(252665) ~]# kill -2 252653

then,view pods log:

~]# kubectl logs kubectl-trace-b4bf4d65-957d-11ed-8c73-00163e0c327b-9w9z6 if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two times Attaching 336 probes...

first SIGINT received, now if your program had maps and did not free them it should print them out

so,where find my bpftrace result ?

use trace xxx -a, Still can't see the result ]# kubectl trace run cn-beijing.192.168.88.23 -e "tracepoint:syscalls:sysenter* { @[probe] = count(); }" -a

kill -2 timeout pid ,i can use logs pod to view bpftrace result ,and pod status is completed

]# pstree -sp 257811 systemd(1)───containerd-shim(257734)───timeout(257788)───trace-runner(257800)───bpftrace(257811)

kubectl-trace-d411da1d-957e-11ed-a75c-00163e0c327b-btltq 0/1 Completed

]# kubectl logs kubectl-trace-d411da1d-957e-11ed-a75c-00163e0c327b-btltq if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two times Attaching 336 probes...

first SIGINT received, now if your program had maps and did not free them it should print them out

@[tracepoint:syscalls:sys_enter_symlink]: 1 @[tracepoint:syscalls:sys_enter_fallocate]: 1 @[tracepoint:syscalls:sys_enter_restart_syscall]: 1 @[tracepoint:syscalls:sys_enter_time]: 1 @[tracepoint:syscalls:sys_enter_add_key]: 1 @[tracepoint:syscalls:sys_enter_sendmmsg]: 1

LIRUILONGS commented 6 months ago

I have the same problem and I want to know why and where the problem is, any help would be greatly appreciated

┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$kubectl trace run node/vms103.liruilongs.github.io -f opensnoop.bt
trace fc86c785-e513-11ee-8e04-000c290e5d5f created
┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$kubectl get pods | grep fc86c785-e513-11ee-8e04-000c290e5d5f
kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm   0/1     ContainerCreating   0             8s
┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$kubectl get pods kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm -w
NAME                                                       READY   STATUS      RESTARTS   AGE
kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm   0/1     Completed   0          19s
kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm   0/1     Completed   0          20s
kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm   0/1     Completed   0          21s
kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm   0/1     Terminating   0          26s
kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm   0/1     Terminating   0          26s
^C┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$
┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$kubectl logs  kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm
Error from server (NotFound): pods "kubectl-trace-fc86c785-e513-11ee-8e04-000c290e5d5f-g67gm" not found
if your program has maps to print, send a SIGINT using Ctrl-C, if you want to interrupt the execution send SIGINT two ││ times                                                                                                                │
│/bpftrace/include/clang_workarounds.h:14:10: fatal error: 'linux/types.h' file not found                              ││exit status 1

Add --fetch-headers to try

┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl trace run vms105.liruilongs.github.io -e "tracepoint:syscalls:sys_enter_execve { @[comm] = count() }" --fetch-headers
trace 3d9981a0-e59b-11ee-a570-000c290e5d5f created
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl get pods
NAME                                                       READY   STATUS     RESTARTS        AGE
hello-webhook-deployment-7f599b95c4-hjx86                  1/1     Running    1 (2d16h ago)   18d
kubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f-wwprt   0/1     Init:0/1   0               12s
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl get pods -w
NAME                                                       READY   STATUS     RESTARTS        AGE
hello-webhook-deployment-7f599b95c4-hjx86                  1/1     Running    1 (2d16h ago)   18d
kubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f-wwprt   0/1     Init:0/1   0               33s
kubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f-wwprt   0/1     Init:0/1   0               41s
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl get pods -w
NAME                                                       READY   STATUS     RESTARTS        AGE
hello-webhook-deployment-7f599b95c4-hjx86                  1/1     Running    1 (2d16h ago)   18d
kubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f-wwprt   0/1     Init:0/1   0               64s
┌──[root@vms100.liruilongs.github.io]-[~/ansible]
└─$kubectl get pods
NAME                                                       READY   STATUS    RESTARTS        AGE
hello-webhook-deployment-7f599b95c4-hjx86                  1/1     Running   1 (2d16h ago)   18d
kubectl-trace-3d9981a0-e59b-11ee-a570-000c290e5d5f-wwprt   1/1     Running   0               16m

Os

┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$hostnamectl
   Static hostname: vms100.liruilongs.github.io
         Icon name: computer-vm
           Chassis: vm
        Machine ID: e93ae3f6cb354f3ba509eeb73568087e
           Boot ID: 51ad5f1933914654affd2dcf9ebca862
    Virtualization: vmware
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 5.4.266-1.el7.elrepo.x86_64
      Architecture: x86-64

script

┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$cat opensnoop.bt
#!/usr/bin/bpftrace
/*
 * opensnoop    Trace open() syscalls.
 *              For Linux, uses bpftrace and eBPF.
 *
 * Also a basic example of bpftrace.
 *
 * USAGE: opensnoop.bt
 *
 * This is a bpftrace version of the bcc tool of the same name.
 *
 * Copyright 2018 Netflix, Inc.
 * Licensed under the Apache License, Version 2.0 (the "License")
 *
 * 08-Sep-2018  Brendan Gregg   Created this.
 */

BEGIN
{
        printf("Tracing open syscalls... Hit Ctrl-C to end.\n");
        printf("%-6s %-16s %4s %3s %s\n", "PID", "COMM", "FD", "ERR", "PATH");
}

tracepoint:syscalls:sys_enter_open,
tracepoint:syscalls:sys_enter_openat
{
        @filename[tid] = args->filename;
}

tracepoint:syscalls:sys_exit_open,
tracepoint:syscalls:sys_exit_openat
/@filename[tid]/
{
        $ret = args->ret;
        $fd = $ret >= 0 ? $ret : -1;
        $errno = $ret >= 0 ? 0 : - $ret;

        printf("%-6d %-16s %4d %3d %s\n", pid, comm, $fd, $errno,
            str(@filename[tid]));
        delete(@filename[tid]);
}

END
{
        clear(@filename);
}
┌──[root@vms100.liruilongs.github.io]-[~/ansible/trace]
└─$