search

deepflowio / deepflow

eBPF Observability - Distributed Tracing and Profiling
https://deepflow.io
Apache License 2.0
2.99k stars 334 forks source link

[BUG] The deepflow agent ebpf collector needs to adapt to the new kernel #5012

Closed sangfei closed 10 months ago

sangfei commented 11 months ago

Search before asking

DeepFlow Component

Agent

What you expected to happen

[2023-12-08 13:20:43.672725 +08:00] ERROR [src/ebpf_dispatcher/ebpf_dispatcher.rs:609] [eBPF Kernel Adapt] The socket_tracer_start() error. Kernel offset adapt failed. Please ensure that BTF is enabled (kernel bu ilt with CONFIG_DEBUG_INFO_BTF=y option). If the current kernel version is low (<5.2), upgrading the Linux kernel to 5.2+ (kernel built with CONFIG_DEBUG_INFO_BTF=y option) can solve the problem. If it is not possible to upgrade the kernel, the kernel-de vel package can be provided for developers to adapt and solve the problem.

How to reproduce

make -C /lib/modules/4.19.117.business.1-amd64/build M=/root/show-kernel-struct-offset modules make[1]: Entering directory '/usr/src/linux-headers-4.19.117.business.1-amd64' CC [M] /root/show-kernel-struct-offset/skso.o Building modules, stage 2. MODPOST 1 modules CC /root/show-kernel-struct-offset/skso.mod.o LD [M] /root/show-kernel-struct-offset/skso.ko make[1]: Leaving directory '/usr/src/linux-headers-4.19.117.business.1-amd64' objdump -d skso.ko

skso.ko: file format elf64-x86-64

Disassembly of section .text:

0000000000000000 : 0: e8 00 00 00 00 callq 5 <STRUCT_TCP_SOCK_COPIED_SEQ_OFFSET+0x5> 5: b8 7c 05 00 00 mov $0x57c,%eax a: c3 retq b: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)

0000000000000010 : 10: e8 00 00 00 00 callq 15 <STRUCT_TCP_SOCK_WRITE_SEQ_OFFSET+0x5> 15: b8 ec 06 00 00 mov $0x6ec,%eax 1a: c3 retq 1b: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)

0000000000000020 : 20: e8 00 00 00 00 callq 25 <STRUCT_TASK_FILES_OFFSET+0x5> 25: b8 b8 06 00 00 mov $0x6b8,%eax 2a: c3 retq 2b: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)

0000000000000030 : 30: e8 00 00 00 00 callq 35 <STRUCT_SOCK_FLAGS_OFFSET+0x5> 35: b8 f8 01 00 00 mov $0x1f8,%eax 3a: c3 retq 3b: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)

0000000000000040 : 40: e8 00 00 00 00 callq 45 <STRUCT_FILES_PRIVATE_DATA_OFFSET+0x5> 45: b8 c8 00 00 00 mov $0xc8,%eax 4a: c3 retq 4b: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)

0000000000000050 : 50: e8 00 00 00 00 callq 55 <STRUCT_TASK_TLS_OFFSET+0x5> 55: b8 a8 0a 00 00 mov $0xaa8,%eax 5a: c3 retq 5b: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)

0000000000000060 : 60: e8 00 00 00 00 callq 65 <init_module+0x5> 65: 31 c0 xor %eax,%eax 67: c3 retq 68: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 6f: 00

0000000000000070 : 70: e8 00 00 00 00 callq 75 <cleanup_module+0x5> 75: f3 c3 repz retq

DeepFlow version

-003:kubectl exec -it -n deepflow deploy/deepflow-server -- deepflow-server -v 2023/12/21 19:41:35 ENV K8S_NODE_NAME_FOR_DEEPFLOW=xxxx; K8S_NODE_IP_FOR_DEEPFLOW=xxxx; K8S_POD_NAME_FOR_DEEPFLOW=deepflow-server-848b8c8758-5cw4h; K8S_POD_IP_FOR_DEEPFLOW=xxxx; K8S_NAMESPACE_FOR_DEEPFLOW=deepflow Name: deepflow-server community edition Branch: v6.4.3 CommitID: 8e168feb8335c5938959516ba62b34ea0143c237 RevCount: 9294 Compiler: go version go1.20.11 linux/amd64 CompileTime: 2023-11-24 02:25:13

-003:~# kubectl exec -it -n deepflow ds/deepflow-agent -- deepflow-agent -v Defaulted container "deepflow-agent" out of: deepflow-agent, configure-sysctl (init) 9294-8e168feb8335c5938959516ba62b34ea0143c237 Name: deepflow-agent community edition Branch: v6.4.3 CommitId: 8e168feb8335c5938959516ba62b34ea0143c237 RevCount: 9294 Compiler: rustc 1.73.0 (cc66ad468 2023-10-03) CompileTime: 2023-11-24 02:27:06

DeepFlow agent list

No response

Kubernetes CNI

No response

Operation-System/Kernel version

"Debian GNU/Linux 9 (stretch)" 4.19.117.business.1-amd64

Anything else

No response

Are you willing to submit a PR?

Code of Conduct

yinjiping commented 10 months ago

This issue has been fixed in the main and v6.4 branches. Thank you for submitting the issue. PR: https://github.com/deepflowio/deepflow/pull/5234

yinjiping commented 10 months ago

Execute the command 'uname - v' to see the return information @sangfei

sangfei commented 9 months ago

Execute the command 'uname - v' to see the return information @sangfei

uname -v

4.19.117.bsk.business.1 SMP Debian 4.19.117.bsk.business.1 Wed

sangfei commented 9 months ago

@yinjiping 这个问题没解决掉，现在启动报错如下，请再帮忙看一下如何解决 [2024-02-21 23:14:45.848210 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:650] ebpf collector init... [2024-02-21 23:14:45.848251 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:398] ebpf golang uprobe proc regexp is empty, skip set [2024-02-21 23:14:45.848274 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:414] ebpf openssl uprobe proc regexp is empty, skip set [2024-02-21 23:14:45.848291 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:437] ebpf golang symbol proc regexp is empty, skip set [2024-02-21 23:14:45.848310 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:442] l7 protocol Custom parse enabled [2024-02-21 23:14:45.848341 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:442] l7 protocol Http1 parse enabled [2024-02-21 23:14:45.848364 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:442] l7 protocol Http2 parse enabled [2024-02-21 23:14:45.848381 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:442] l7 protocol DNS parse enabled [2024-02-21 23:14:45.848399 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:442] l7 protocol SofaRPC parse enabled [2024-02-21 23:14:45.848415 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:442] l7 protocol MySQL parse enabled [2024-02-21 23:14:45.848431 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:442] l7 protocol Kafka parse enabled [2024-02-21 23:14:45.848447 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:442] l7 protocol Redis parse enabled [2024-02-21 23:14:45.848464 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:442] l7 protocol MongoDB parse enabled [2024-02-21 23:14:45.848481 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:442] l7 protocol PostgreSQL parse enabled [2024-02-21 23:14:45.848496 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:442] l7 protocol Dubbo parse enabled [2024-02-21 23:14:45.848511 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:442] l7 protocol FastCGI parse enabled [2024-02-21 23:14:45.848526 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:442] l7 protocol Oracle parse enabled [2024-02-21 23:14:45.848542 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:442] l7 protocol MQTT parse enabled [2024-02-21 23:14:45.848559 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:442] l7 protocol AMQP parse enabled [2024-02-21 23:14:45.848574 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:442] l7 protocol NATS parse enabled [2024-02-21 23:14:45.848589 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:442] l7 protocol TLS parse enabled [2024-02-21 23:14:45.848605 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:442] l7 protocol OpenWire parse enabled [2024-02-21 23:14:45.848662 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Currently "/proc/sys/net/core/bpf_jit_enable" value is 1, not need set. [2024-02-21 23:14:45.848756 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO log2_default_hugepage_sz 21 [2024-02-21 23:14:45.848788 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO linux version : 4.19.117.bsk.business.1-amd64 (version code : 4244587007) [2024-02-21 23:14:45.848808 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO RLIMIT_NOFILE cur:1048576, rlim_max:1048576 [2024-02-21 23:14:45.848857 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO sys_boot_time_ns : 1706791677807729691 [2024-02-21 23:14:45.848858 +08:00] INFO [src/sender/uniform_sender.rs:344] remote_logger sender update dst ip from 0.0.0.0 to 10.199.16.6 [2024-02-21 23:14:45.849060 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO [deepflow-ebpfctl] sockopt register succeed, type get: 404 - 404 set: 400 - 403 [2024-02-21 23:14:45.849152 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO register_period_event_op 'kick_kern' succeed. [2024-02-21 23:14:45.849169 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO register_period_event_op 'boot time update' succeed. [2024-02-21 23:14:45.849247 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Tracer 'socket-trace', Not Found. [2024-02-21 23:14:45.849263 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Tracer 'socket-trace', Not Found. [2024-02-21 23:14:45.849275 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Tracer 'socket-trace', Not Found. [2024-02-21 23:14:45.849426 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Set ports_bitmap[DNS] success, ports 53,5353 [2024-02-21 23:14:45.849516 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Set ports_bitmap[TLS] success, ports 443 [2024-02-21 23:14:45.849722 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Set ports_bitmap[SofaRPC] success, ports 1-65535 [2024-02-21 23:14:45.849935 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Set ports_bitmap[Kafka] success, ports 1-65535 [2024-02-21 23:14:45.850140 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Set ports_bitmap[MongoDB] success, ports 1-65535 [2024-02-21 23:14:45.850354 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Set ports_bitmap[OpenWire] success, ports 1-65535 [2024-02-21 23:14:45.850559 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Set ports_bitmap[Custom] success, ports 1-65535 [2024-02-21 23:14:45.850767 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Set ports_bitmap[Redis] success, ports 1-65535 [2024-02-21 23:14:45.850979 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Set ports_bitmap[Oracle] success, ports 1-65535 [2024-02-21 23:14:45.851187 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Set ports_bitmap[MQTT] success, ports 1-65535 [2024-02-21 23:14:45.851401 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Set ports_bitmap[NATS] success, ports 1-65535 [2024-02-21 23:14:45.851608 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Set ports_bitmap[AMQP] success, ports 1-65535 [2024-02-21 23:14:45.851813 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Set ports_bitmap[Dubbo] success, ports 1-65535 [2024-02-21 23:14:45.851962 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Set ports_bitmap[HTTP2] success, ports 1-65535 [2024-02-21 23:14:45.852190 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Set ports_bitmap[MySQL] success, ports 1-65535 [2024-02-21 23:14:45.852405 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Set ports_bitmap[PgSQL] success, ports 1-65535 [2024-02-21 23:14:45.852543 +08:00] INFO [src/sender/uniform_sender.rs:407] remote_logger sender tcp connection to 10.199.16.6:30033 succeed. [2024-02-21 23:14:45.852609 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Set ports_bitmap[FastCGI] success, ports 1-65535 [2024-02-21 23:14:45.852829 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Set ports_bitmap[HTTP1] success, ports 1-65535 [2024-02-21 23:14:45.880754 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Tracer 'socket-trace', Not Found. [2024-02-21 23:14:45.881071 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO license: GPL [2024-02-21 23:14:45.881711 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Update map ("socket_info_map"), set max_entries 524288 [2024-02-21 23:14:45.881753 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO Update map ("__trace_map"), set max_entries 524288 [2024-02-21 23:14:46.000994 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] WARNING: func load_objprogs() [user/load.c:418] bcc_prog_load() failed. name: runtime_execute, Invalid argument errno: 22 [2024-02-21 23:14:46.001083 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] WARNING: func ebpf_obj_load() [user/load.c:723] eBPF load programs failed. (errno 22) [2024-02-21 23:14:46.001104 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO release object ("socket-trace-bpf-linux-common") ... [2024-02-21 23:14:46.001202 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] INFO release object done [2024-02-21 23:14:46.001228 +08:00] INFO [src/ebpf/mod.rs:651] [eBPF] WARNING: func tracer_bpf_load() [user/tracer.c:523] bpf load "socket-trace-bpf-linux-common" failed, error:Invalid argument (22). [2024-02-21 23:14:46.001284 +08:00] ERROR [src/trident.rs:2318] ebpf collector error: EbpfRunningError