search

deepflowio / deepflow

eBPF Observability - Distributed Tracing and Profiling
https://deepflow.io
Apache License 2.0
2.88k stars 319 forks source link

[BUG] deepflow-agent v6.5 ebpf program EbpfRunningError #7988

Closed Fancyki1 closed 1 month ago

Fancyki1 commented 1 month ago

Search before asking

DeepFlow Component

Agent

What you expected to happen

不确定有什么影响，ebpf程序加载出错了

[2024-08-29 11:12:56.187319 +08:00] ERROR [src/rpc/session.rs:241] Dial server(10.244.91.163 20035) failed: transport error
bpf: Failed to load program: Permission denied
0: (bf) r9 = r1
1: (79) r6 = *(u64 *)(r9 +48)
func '__sys_recvmmsg' doesn't have 7-th argument
invalid bpf_context access off=48 size=8
processed 2 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0

[2024-08-29 11:12:56.364209 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] WARN func load_obj__progs() [user/load.c:585] bcc_prog_load() failed. name: kretfunc____sys_recvmmsg, Permission denied errno: 13
[2024-08-29 11:12:56.364307 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] WARN func load_obj__progs() [user/load.c:590] The number of EBPF instructions (17568) exceeded the maximum limit (4096).
[2024-08-29 11:12:56.364325 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] WARN func ebpf_obj_load() [user/load.c:939] eBPF load programs failed. (errno 13)
[2024-08-29 11:12:56.364340 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO release object ("socket-trace-bpf-linux-kfunc") ...
[2024-08-29 11:12:56.364695 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO release object done
[2024-08-29 11:12:56.364748 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] WARN func tracer_bpf_load() [user/tracer.c:525] bpf load "socket-trace-bpf-linux-kfunc" failed, error:Permission denied (13).
[2024-08-29 11:12:56.364769 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] WARN func tracer_bpf_load() [user/tracer.c:544] Check the selinux status, if found SELinux 'status: enabled' and 'Current mode:enforcing', please try the following way to solve:
1 Create file 'deepflow-agent.te',contents:

module deepflow-agent 1.0;
require {
  type container_runtime_t;
  class bpf { map_create map_read map_write prog_load prog_run };
}
allow container_runtime_t self:bpf { map_create map_read map_write prog_load prog_run };

2 checkmodule -M -m -o deepflow-agent.mod deepflow-agent.te
3 semodule_package -o deepflow-agent.pp -m deepflow-agent.mod
4 semodule -i deepflow-agent.pp
5 restart pods
[2024-08-29 11:12:56.364816 +08:00] ERROR [src/trident.rs:2339] ebpf collector error: EbpfRunningError

报错版本

10862-c73c030a0f43f2179137038e37e0474b1c877de0
Name: deepflow-agent community edition
Branch: v6.5
CommitId: c73c030a0f43f2179137038e37e0474b1c877de0
RevCount: 10862
Compiler: rustc 1.77.1 (7cf61ebde 2024-03-27)
CompileTime: 2024-08-27 11:51:36

切换老版本无报错

10841-e0a10484155463453b61b20bd5fcd6222d59f829
Name: deepflow-agent community edition
Branch: v6.5
CommitId: e0a10484155463453b61b20bd5fcd6222d59f829
RevCount: 10841
Compiler: rustc 1.77.1 (7cf61ebde 2024-03-27)
CompileTime: 2024-08-18 14:17:18

@yinjiping

How to reproduce

不确定有什么影响，ebpf程序加载出错了

DeepFlow version

10862-c73c030a0f43f2179137038e37e0474b1c877de0
Name: deepflow-agent community edition
Branch: v6.5
CommitId: c73c030a0f43f2179137038e37e0474b1c877de0
RevCount: 10862
Compiler: rustc 1.77.1 (7cf61ebde 2024-03-27)
CompileTime: 2024-08-27 11:51:36

DeepFlow agent list

ID   NAME                TYPE     CTRL_IP      CTRL_MAC            STATE    GROUP     EXCEPTIONS   REVISION     UPGRADE_REVISION
11   master-b9846-0-V1   K8S_VM   10.107.4.2   fe:fc:fe:ed:59:47   NORMAL   default                v6.5 10862

Kubernetes CNI

无

Operation-System/Kernel version

4.18.0-372.32.1.90.po1.x86_64

Anything else

[root@master-b9846-0 ~]# sestatus
SELinux status:                 disabled

Are you willing to submit a PR?

Code of Conduct

yinjiping commented 1 month ago

@Fancyki1 感谢您的ISSUE。

Linux 4.18的接口如下：

int __sys_recvmmsg(int fd, struct mmsghdr __user *mmsg, unsigned int vlen,
           unsigned int flags, struct timespec *timeout)

而这个接口在Linux 5.0+，参数列表发生了变动：

int __sys_recvmmsg(int fd, struct mmsghdr __user *mmsg,
           unsigned int vlen, unsigned int flags,
           struct __kernel_timespec __user *timeout,
           struct old_timespec32 __user *timeout32)

在deeflow v6.5 最新版本中我们提交了支持fentry/fexit的特性，详细可以看这里 https://github.com/deepflowio/deepflow/pull/7913 fentry/fexit特性开始于社区版本Linux 5.5，我们在代码中使用了新的__sys_recvmmsg()接口，没有考虑到旧接口。我们尽快提交PR修复这个问题，让其做到兼容。

Fancyki1 commented 1 month ago

@yinjiping @sharang 感谢大佬的修复，我用了修复后最新版本的v6.5 deepflow-agent 依然存在相同的报错，确认一下修复有效或者报错是否有影响 流水线打包： image

root@deepflow-agent-77bpx:/# deepflow-agent --version
10865-4aa4a4f34df9bc9156e4aecd02a30ad576d967f0
Name: deepflow-agent community edition
Branch: v6.5
CommitId: 4aa4a4f34df9bc9156e4aecd02a30ad576d967f0
RevCount: 10865
Compiler: rustc 1.77.1 (7cf61ebde 2024-03-27)
CompileTime: 2024-08-30 09:14:49
bpf: Failed to load program: Permission denied
0: (bf) r9 = r1
1: (79) r6 = *(u64 *)(r9 +48)
func '__sys_recvmmsg' doesn't have 7-th argument
invalid bpf_context access off=48 size=8
processed 2 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0

[2024-09-03 15:27:57.405449 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] WARN func load_obj__progs() [user/load.c:585] bcc_prog_load() failed. name: kretfunc____sys_recvmmsg, Permission denied errno: 13
[2024-09-03 15:27:57.405571 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] WARN func load_obj__progs() [user/load.c:590] The number of EBPF instructions (17568) exceeded the maximum limit (4096).
[2024-09-03 15:27:57.405601 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] WARN func ebpf_obj_load() [user/load.c:939] eBPF load programs failed. (errno 13)
[2024-09-03 15:27:57.405619 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO release object ("socket-trace-bpf-linux-kfunc") ...
[2024-09-03 15:27:57.406771 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO release object done
[2024-09-03 15:27:57.406831 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] WARN func tracer_bpf_load() [user/tracer.c:525] bpf load "socket-trace-bpf-linux-kfunc" failed, error:Permission denied (13).
[2024-09-03 15:27:57.406851 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] WARN func tracer_bpf_load() [user/tracer.c:544] Check the selinux status, if found SELinux 'status: enabled' and 'Current mode:enforcing', please try the following way to solve:
1 Create file 'deepflow-agent.te',contents:

module deepflow-agent 1.0;
require {
  type container_runtime_t;
  class bpf { map_create map_read map_write prog_load prog_run };
}
allow container_runtime_t self:bpf { map_create map_read map_write prog_load prog_run };

2 checkmodule -M -m -o deepflow-agent.mod deepflow-agent.te
3 semodule_package -o deepflow-agent.pp -m deepflow-agent.mod
4 semodule -i deepflow-agent.pp
5 restart pods
[2024-09-03 15:27:57.406904 +08:00] ERROR [src/trident.rs:2339] ebpf collector error: EbpfRunningError
yinjiping commented 1 month ago

@Fancyki1 抱歉，应该是修复方式没有完全解决问题，我尽快修正这个问题。

yinjiping commented 1 month ago

@Fancyki1 , 已做了修复可使用最新v6.5版本，看是否解决了问题。

Fancyki1 commented 1 month ago

@yinjiping issue验证完毕，已经解决了问题

[root@master-15d62-0 fx]# kubectl get pod -n deepflow
NAME                   READY   STATUS    RESTARTS   AGE
deepflow-agent-ghdvw   1/1     Running   0          3m20s
deepflow-agent-pctct   1/1     Running   0          2m30s
deepflow-agent-v5psp   1/1     Running   0          2m50s
[root@master-15d62-0 fx]# kubectl exec -it deepflow-agent-v5psp -n deepflow -- deepflow-agent --version
Defaulted container "deepflow-agent" out of: deepflow-agent, configure-sysctl (init)
10870-7fe8add4b63b01e2e8b627e298b7b7b3a7cb41d1
Name: deepflow-agent community edition
Branch: v6.5
CommitId: 7fe8add4b63b01e2e8b627e298b7b7b3a7cb41d1
RevCount: 10870
Compiler: rustc 1.77.1 (7cf61ebde 2024-03-27)
CompileTime: 2024-09-04 15:20:44
[root@master-15d62-0 fx]# uname -r
4.18.0-372.19.1.30.po1.x86_64

deepflow-agent的一些关键的日志

[2024-09-05 16:37:51.598525 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO ctrl_main begin !!!
[2024-09-05 16:37:51.688859 +08:00] INFO [src/platform/kubernetes/active_poller.rs:86] kubernetes poller updated to version (1)
[2024-09-05 16:37:52.104222 +08:00] INFO [src/sender/uniform_sender.rs:184] stats id 1 update agent id from 6 to 6, team id from 0 to 1, organization id from 0 to 1.
[2024-09-05 16:37:52.109580 +08:00] INFO [src/sender/uniform_sender.rs:448] stats sender tcp connection to 10.107.19.118:30033 succeed.
[2024-09-05 16:37:52.598227 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO cpus_kick begin !!!
[2024-09-05 16:37:52.696587 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO [eBPF Kernel Adapt] kernel_offset_infer_server close.client_count:40
[2024-09-05 16:37:53.592787 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] WARN func socket_tracer_start() [user/socket.c:2277] [eBPF Kernel Adapt] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
[2024-09-05 16:37:53.668106 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO [eBPF Kernel Adapt] Linux 4.18.0-372.19.1.30.po1.x86_64 adapt success. Set the status to TRACER_RUNNING
[2024-09-05 16:37:53.668194 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO set_period_event_invalid 'check-kern-adapt' set invalid succeed.
[2024-09-05 16:37:53.668204 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO set_period_event_invalid 'trigger_kern_adapt' set invalid succeed.
[2024-09-05 16:37:53.760147 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO Set current state: TRACER_WAIT_STOP.
[2024-09-05 16:37:53.760263 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO tracer(socket-trace) detach ...
[2024-09-05 16:37:53.798768 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach enter kprobe: 'kprobe/__sys_sendmsg', success!
[2024-09-05 16:37:53.863763 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach enter kprobe: 'kprobe/__sys_sendmmsg', success!
[2024-09-05 16:37:53.898733 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach enter kprobe: 'kprobe/__sys_recvmsg', success!
[2024-09-05 16:37:53.963776 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach enter kprobe: 'kprobe/__sys_recvmmsg', success!
[2024-09-05 16:37:54.006744 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach enter kprobe: 'kprobe/do_writev', success!
[2024-09-05 16:37:54.067769 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach enter kprobe: 'kprobe/do_readv', success!
[2024-09-05 16:37:54.079680 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach tracepoint: 'tracepoint/syscalls/sys_enter_write', succeed!
[2024-09-05 16:37:54.091668 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach tracepoint: 'tracepoint/syscalls/sys_enter_read', succeed!
[2024-09-05 16:37:54.100658 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach tracepoint: 'tracepoint/syscalls/sys_enter_sendto', succeed!
[2024-09-05 16:37:54.115660 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach tracepoint: 'tracepoint/syscalls/sys_enter_recvfrom', succeed!
[2024-09-05 16:37:54.131668 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach tracepoint: 'tracepoint/syscalls/sys_enter_connect', succeed!
[2024-09-05 16:37:54.147670 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach tracepoint: 'tracepoint/syscalls/sys_exit_socket', succeed!
[2024-09-05 16:37:54.159657 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach tracepoint: 'tracepoint/syscalls/sys_exit_read', succeed!
[2024-09-05 16:37:54.167653 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach tracepoint: 'tracepoint/syscalls/sys_exit_write', succeed!
[2024-09-05 16:37:54.179654 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach tracepoint: 'tracepoint/syscalls/sys_exit_sendto', succeed!
[2024-09-05 16:37:54.192651 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach tracepoint: 'tracepoint/syscalls/sys_exit_recvfrom', succeed!
[2024-09-05 16:37:54.201656 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach tracepoint: 'tracepoint/syscalls/sys_exit_sendmsg', succeed!
[2024-09-05 16:37:54.210657 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach tracepoint: 'tracepoint/syscalls/sys_exit_sendmmsg', succeed!
[2024-09-05 16:37:54.218653 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach tracepoint: 'tracepoint/syscalls/sys_exit_recvmsg', succeed!
[2024-09-05 16:37:54.230656 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach tracepoint: 'tracepoint/syscalls/sys_exit_recvmmsg', succeed!
[2024-09-05 16:37:54.238658 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach tracepoint: 'tracepoint/syscalls/sys_exit_writev', succeed!
[2024-09-05 16:37:54.248647 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach tracepoint: 'tracepoint/syscalls/sys_exit_readv', succeed!
[2024-09-05 16:37:54.260652 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach tracepoint: 'tracepoint/syscalls/sys_exit_accept', succeed!
[2024-09-05 16:37:54.272664 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach tracepoint: 'tracepoint/syscalls/sys_exit_accept4', succeed!
[2024-09-05 16:37:54.284660 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach tracepoint: 'tracepoint/syscalls/sys_exit_fork', succeed!
[2024-09-05 16:37:54.292680 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach tracepoint: 'tracepoint/syscalls/sys_exit_clone', succeed!
[2024-09-05 16:37:54.313671 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach tracepoint: 'tracepoint/sched/sched_process_exec', succeed!
[2024-09-05 16:37:54.326665 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach tracepoint: 'tracepoint/sched/sched_process_exit', succeed!
[2024-09-05 16:37:54.350686 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach tracepoint: 'tracepoint/syscalls/sys_enter_close', succeed!
[2024-09-05 16:37:54.350761 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 0 close fd 235
[2024-09-05 16:37:54.350789 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 1 close fd 236
[2024-09-05 16:37:54.350816 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 2 close fd 237
[2024-09-05 16:37:54.350839 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 3 close fd 238
[2024-09-05 16:37:54.350864 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 4 close fd 239
[2024-09-05 16:37:54.350888 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 5 close fd 240
[2024-09-05 16:37:54.350914 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 6 close fd 241
[2024-09-05 16:37:54.350940 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 7 close fd 242
[2024-09-05 16:37:54.350962 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 8 close fd 243
[2024-09-05 16:37:54.350986 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 9 close fd 244
[2024-09-05 16:37:54.351009 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 10 close fd 245
[2024-09-05 16:37:54.351034 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 11 close fd 246
[2024-09-05 16:37:54.351057 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 12 close fd 247
[2024-09-05 16:37:54.351093 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 13 close fd 248
[2024-09-05 16:37:54.351111 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 14 close fd 249
[2024-09-05 16:37:54.351132 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 15 close fd 250
[2024-09-05 16:37:54.351157 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 16 close fd 251
[2024-09-05 16:37:54.351185 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 17 close fd 252
[2024-09-05 16:37:54.351210 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 18 close fd 253
[2024-09-05 16:37:54.351233 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 19 close fd 254
[2024-09-05 16:37:54.351250 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 20 close fd 255
[2024-09-05 16:37:54.351274 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 21 close fd 256
[2024-09-05 16:37:54.351300 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 22 close fd 257
[2024-09-05 16:37:54.351323 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 23 close fd 258
[2024-09-05 16:37:54.351341 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 24 close fd 259
[2024-09-05 16:37:54.351366 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 25 close fd 260
[2024-09-05 16:37:54.351394 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 26 close fd 261
[2024-09-05 16:37:54.351407 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 27 close fd 262
[2024-09-05 16:37:54.351429 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 28 close fd 263
[2024-09-05 16:37:54.351456 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 29 close fd 264
[2024-09-05 16:37:54.351477 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 30 close fd 265
[2024-09-05 16:37:54.351504 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO detach cpu 31 close fd 266
[2024-09-05 16:37:54.351698 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO tracer "socket-trace" detach perf event prog successful.
[2024-09-05 16:37:54.351706 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO Successfully completed detach.
[2024-09-05 16:37:54.351714 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO Set current state: TRACER_STOP.
[2024-09-05 16:37:54.352340 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO [reclaim_socket_map] curr_socket_count 0 sockets_reclaim_count :66
[2024-09-05 16:37:54.582949 +08:00] INFO [src/sender/uniform_sender.rs:184] 3-doc-to-collector-sender id 3 update agent id from 6 to 6, team id from 0 to 1, organization id from 0 to 1.
[2024-09-05 16:37:54.583103 +08:00] INFO [src/sender/uniform_sender.rs:184] 2-protolog-to-collector-sender id 4 update agent id from 6 to 6, team id from 0 to 1, organization id from 0 to 1.
[2024-09-05 16:37:54.583206 +08:00] INFO [src/sender/uniform_sender.rs:184] 3-flowlog-to-collector-sender id 2 update agent id from 6 to 6, team id from 0 to 1, organization id from 0 to 1.
[2024-09-05 16:37:54.583339 +08:00] INFO [src/sender/uniform_sender.rs:184] 2-packet-sequence-block-to-sender id 6 update agent id from 6 to 6, team id from 0 to 1, organization id from 0 to 1.
[2024-09-05 16:37:55.593001 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO Call socket_tracer_start()
[2024-09-05 16:37:55.593101 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:946] ebpf collector started
[2024-09-05 16:37:55.593226 +08:00] INFO [src/sender/uniform_sender.rs:275] 1-otel-to-sender uniform sender id: 10 started
[2024-09-05 16:37:55.593355 +08:00] INFO [src/sender/uniform_sender.rs:275] 1-compressed-otel-to-sender uniform sender id: 13 started
[2024-09-05 16:37:55.593458 +08:00] INFO [src/sender/uniform_sender.rs:275] 1-prometheus-to-sender uniform sender id: 11 started
[2024-09-05 16:37:55.593577 +08:00] INFO [src/sender/uniform_sender.rs:275] 1-telegraf-to-sender uniform sender id: 12 started
[2024-09-05 16:37:55.593678 +08:00] INFO [src/sender/uniform_sender.rs:275] 1-profile-to-sender uniform sender id: 8 started
[2024-09-05 16:37:55.593801 +08:00] INFO [src/sender/uniform_sender.rs:275] 1-proc-event-to-sender uniform sender id: 7 started
[2024-09-05 16:37:55.593945 +08:00] INFO [src/sender/uniform_sender.rs:275] 1-application-log-to-sender uniform sender id: 9 started
[2024-09-05 16:37:55.594012 +08:00] INFO [src/collector/l7_quadruple_generator.rs:531] new l7 quadruple_generator id: 2, second_delay: 9, minute_delay: 69, l7_metrics_enabled: true, vtap_flow_1s_enabled: true collector_enabled: true
[2024-09-05 16:37:55.594075 +08:00] INFO [src/integration_collector.rs:1032] integration collector starting
[2024-09-05 16:37:55.594244 +08:00] INFO [src/integration_collector.rs:1138] integration collector started
[2024-09-05 16:37:55.594258 +08:00] INFO [src/integration_collector.rs:1139] integration collector listening on http://[::]:38086
[2024-09-05 16:37:55.594526 +08:00] INFO [src/collector/l7_quadruple_generator.rs:473] l7 quadruple generator id: 2 started
[2024-09-05 16:37:55.594699 +08:00] INFO [src/collector/collector.rs:1436] second_collector id=(2) started
[2024-09-05 16:37:55.594841 +08:00] INFO [src/collector/collector.rs:1436] minute_collector id=(2) started
[2024-09-05 16:37:55.594948 +08:00] INFO [src/sender/uniform_sender.rs:275] 2-pcap-batch-to-sender uniform sender id: 5 started
[2024-09-05 16:37:55.594962 +08:00] INFO [src/trident.rs:2633] Started agent components.
[2024-09-05 16:37:55.594972 +08:00] INFO [src/utils/stats.rs:336] set stats hostname to "master-41874-1-V2"
[2024-09-05 16:37:55.594984 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:822] ebpf collector stopping ebpf-kernel.
[2024-09-05 16:37:55.594997 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO Call socket_tracer_stop()
[2024-09-05 16:37:55.595004 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:973] ebpf collector stopping thread.
[2024-09-05 16:37:56.087399 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:977] ebpf collector stopped.
[2024-09-05 16:37:56.087778 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO Call socket_tracer_start()
[2024-09-05 16:37:56.087804 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:946] ebpf collector started
[2024-09-05 16:37:56.087818 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO Received limit_size (2048), the final value is set to '2048'
[2024-09-05 16:37:56.154336 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO Set current state: TRACER_WAIT_START.
[2024-09-05 16:37:56.154425 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO tracer(socket-trace) attach ...
[2024-09-05 16:37:56.175136 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach enter kprobe: 'kprobe/__sys_sendmsg', success!
[2024-09-05 16:37:56.197563 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach enter kprobe: 'kprobe/__sys_sendmmsg', success!
[2024-09-05 16:37:56.252996 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach enter kprobe: 'kprobe/__sys_recvmsg', success!
[2024-09-05 16:37:56.271487 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach enter kprobe: 'kprobe/__sys_recvmmsg', success!
[2024-09-05 16:37:56.286871 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach enter kprobe: 'kprobe/do_writev', success!
[2024-09-05 16:37:56.353345 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach enter kprobe: 'kprobe/do_readv', success!
[2024-09-05 16:37:56.357056 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_enter_write', succeed!
[2024-09-05 16:37:56.357304 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_enter_read', succeed!
[2024-09-05 16:37:56.357389 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_enter_sendto', succeed!
[2024-09-05 16:37:56.357457 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_enter_recvfrom', succeed!
[2024-09-05 16:37:56.357557 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_enter_connect', succeed!
[2024-09-05 16:37:56.357829 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_socket', succeed!
[2024-09-05 16:37:56.357930 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_read', succeed!
[2024-09-05 16:37:56.358003 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_write', succeed!
[2024-09-05 16:37:56.358067 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_sendto', succeed!
[2024-09-05 16:37:56.358134 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_recvfrom', succeed!
[2024-09-05 16:37:56.358197 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_sendmsg', succeed!
[2024-09-05 16:37:56.358255 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_sendmmsg', succeed!
[2024-09-05 16:37:56.358315 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_recvmsg', succeed!
[2024-09-05 16:37:56.358382 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_recvmmsg', succeed!
[2024-09-05 16:37:56.358450 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_writev', succeed!
[2024-09-05 16:37:56.358512 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_readv', succeed!
[2024-09-05 16:37:56.358569 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_accept', succeed!
[2024-09-05 16:37:56.358657 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_accept4', succeed!
[2024-09-05 16:37:56.358806 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_fork', succeed!
[2024-09-05 16:37:56.358907 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_clone', succeed!
[2024-09-05 16:37:56.359186 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach tracepoint: 'tracepoint/sched/sched_process_exec', succeed!
[2024-09-05 16:37:56.359453 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach tracepoint: 'tracepoint/sched/sched_process_exit', succeed!
[2024-09-05 16:37:56.359537 +08:00] INFO [src/ebpf/mod.rs:707] [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_enter_close', succeed!
...省略日志