[eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.[BUG] #3421
[X] I had searched in the issues and found no similar feature requirement.
DeepFlow Component
Agent
What you expected to happen
2023-06-25 11:10:42 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
thread_num: 1,
perf_pages_count: 128,
ring_size: 65536,
max_socket_entries: 524288,
max_trace_entries: 524288,
socket_map_max_reclaim: 520000,
go_tracing_timeout: 0,
io_event_collect_mode: 1,
io_event_minimal_duration: 1ms,
},
} to EbpfConfig {
collector_enabled: true,
l7_metrics_enabled: true,
vtap_id: 3,
epc_id: 1,
l7_log_packet_size: 1024,
l7_log_session_timeout: 120s,
l7_protocol_inference_max_fail_count: 50,
l7_protocol_inference_ttl: 60,
l7_log_tap_types: [
(
0,
true,
),
],
ctrl_mac: 00:00:00:00:00:00,
l7_protocol_enabled_bitmap: [
Http1,
Http2,
DNS,
SofaRPC,
MySQL,
Kafka,
Redis,
PostgreSQL,
Dubbo,
MQTT,
],
l7_protocol_ports: {
"DNS": "53",
},
ebpf: EbpfYamlConfig {
disabled: false,
log_file: "",
kprobe_whitelist: EbpfKprobeWhitelist {
port_list: "",
},
uprobe_proc_regexp: UprobeProcRegExp {
golang_symbol: "",
golang: ".",
openssl: ".",
},
thread_num: 1,
perf_pages_count: 128,
ring_size: 65536,
max_socket_entries: 524288,
max_trace_entries: 524288,
socket_map_max_reclaim: 520000,
go_tracing_timeout: 0,
io_event_collect_mode: 1,
io_event_minimal_duration: 1ms,
},
}
[2023-06-25 11:10:42.151625 +08:00] INFO [src/collector/quadruple_generator.rs:659] quadruple generator update collector_enabled to true
[2023-06-25 11:10:42.151811 +08:00] INFO [src/collector/quadruple_generator.rs:659] quadruple generator update collector_enabled to true
[2023-06-25 11:10:42.290964 +08:00] INFO [src/sender/uniform_sender.rs:335] 1-telegraf-to-sender sender update dst ip from 0.0.0.0 to 10.244.210.19
[2023-06-25 11:10:42.291193 +08:00] INFO [src/sender/uniform_sender.rs:346] 1-telegraf-to-sender sender update dst port from 30033 to 20033
[2023-06-25 11:10:42.315300 +08:00] INFO [src/sender/uniform_sender.rs:335] 1-compressed-otel-to-sender sender update dst ip from 0.0.0.0 to 10.244.210.19
[2023-06-25 11:10:42.315554 +08:00] INFO [src/sender/uniform_sender.rs:346] 1-compressed-otel-to-sender sender update dst port from 30033 to 20033
[2023-06-25 11:10:42.325733 +08:00] INFO [src/sender/uniform_sender.rs:335] 1-profile-to-sender sender update dst ip from 0.0.0.0 to 10.244.210.19
[2023-06-25 11:10:42.326122 +08:00] INFO [src/sender/uniform_sender.rs:335] 1-otel-to-sender sender update dst ip from 0.0.0.0 to 10.244.210.19
[2023-06-25 11:10:42.326282 +08:00] INFO [src/sender/uniform_sender.rs:346] 1-profile-to-sender sender update dst port from 30033 to 20033
[2023-06-25 11:10:42.326349 +08:00] INFO [src/sender/uniform_sender.rs:346] 1-otel-to-sender sender update dst port from 30033 to 20033
[2023-06-25 11:10:42.349862 +08:00] INFO [src/sender/uniform_sender.rs:335] 1-proc-event-to-sender sender update dst ip from 0.0.0.0 to 10.244.210.19
[2023-06-25 11:10:42.350383 +08:00] INFO [src/sender/uniform_sender.rs:346] 1-proc-event-to-sender sender update dst port from 30033 to 20033
[2023-06-25 11:10:42.362520 +08:00] INFO [src/sender/uniform_sender.rs:335] 1-prometheus-to-sender sender update dst ip from 0.0.0.0 to 10.244.210.19
[2023-06-25 11:10:42.362905 +08:00] INFO [src/sender/uniform_sender.rs:346] 1-prometheus-to-sender sender update dst port from 30033 to 20033
2023-06-25 11:10:44 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
[2023-06-25 11:10:44.321140 +08:00] INFO [src/sender/uniform_sender.rs:335] 2-packet-sequence-block-to-sender sender update dst ip from 0.0.0.0 to 10.244.210.19
[2023-06-25 11:10:44.321694 +08:00] INFO [src/sender/uniform_sender.rs:346] 2-packet-sequence-block-to-sender sender update dst port from 30033 to 20033
2023-06-25 11:10:46 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:10:48 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:10:50 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:10:52 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:10:54 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:10:56 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:10:58 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:00 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:02 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:04 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:06 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:08 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:10 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:12 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
[2023-06-25 11:11:14.139469 +08:00] ERROR [src/sender/uniform_sender.rs:391] 3-flowlog-to-collector-sender sender tcp connection to 0.0.0.0:30033 failed
[2023-06-25 11:11:14.140164 +08:00] INFO [src/sender/uniform_sender.rs:335] 3-flowlog-to-collector-sender sender update dst ip from 0.0.0.0 to 10.244.210.19
[2023-06-25 11:11:14.140720 +08:00] INFO [src/sender/uniform_sender.rs:346] 3-flowlog-to-collector-sender sender update dst port from 30033 to 20033
2023-06-25 11:11:14 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:16 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:18 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
[2023-06-25 11:11:19.168826 +08:00] WARN [src/rpc/synchronizer.rs:950] ntp response empty
2023-06-25 11:11:20 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
[2023-06-25 11:11:21.642985 +08:00] INFO [src/rpc/synchronizer.rs:1298] ProxyController update to Some("10.244.210.19"):20035
2023-06-25 11:11:22 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:24 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:26 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:28 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:30 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:32 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:34 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:36 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:38 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:40 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
[2023-06-25 11:11:41.518922 +08:00] INFO [src/rpc/session.rs:501] rpc Proxy changed to proxy 10.244.210.19 20035 from proxy 10.197.212.207 30035
[2023-06-25 11:11:41.522222 +08:00] INFO [src/platform/platform_synchronizer/linux.rs:785] local version 1687655489, remote version 0
2023-06-25 11:11:42 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
[2023-06-25 11:11:42.811405 +08:00] ERROR [src/sender/uniform_sender.rs:391] 3-doc-to-collector-sender sender tcp connection to 0.0.0.0:30033 failed
[2023-06-25 11:11:42.811470 +08:00] ERROR [src/sender/uniform_sender.rs:391] stats sender tcp connection to 0.0.0.0:30033 failed
[2023-06-25 11:11:42.811405 +08:00] ERROR [src/sender/uniform_sender.rs:391] 2-protolog-to-collector-sender sender tcp connection to 0.0.0.0:30033 failed
[2023-06-25 11:11:42.811928 +08:00] INFO [src/sender/uniform_sender.rs:335] 2-protolog-to-collector-sender sender update dst ip from 0.0.0.0 to 10.244.210.19
[2023-06-25 11:11:42.811965 +08:00] INFO [src/sender/uniform_sender.rs:335] 3-doc-to-collector-sender sender update dst ip from 0.0.0.0 to 10.244.210.19
[2023-06-25 11:11:42.812132 +08:00] INFO [src/sender/uniform_sender.rs:346] 2-protolog-to-collector-sender sender update dst port from 30033 to 20033
[2023-06-25 11:11:42.812099 +08:00] INFO [src/sender/uniform_sender.rs:346] 3-doc-to-collector-sender sender update dst port from 30033 to 20033
[2023-06-25 11:11:42.812174 +08:00] INFO [src/sender/uniform_sender.rs:335] stats sender update dst ip from 0.0.0.0 to 10.244.210.19
[2023-06-25 11:11:42.812332 +08:00] INFO [src/sender/uniform_sender.rs:346] stats sender update dst port from 30033 to 20033
2023-06-25 11:11:44 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:46 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:48 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:50 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:52 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:54 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:56 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:11:58 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:12:00 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:12:02 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:12:04 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:12:06 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:12:08 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:12:10 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:12:12 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:12:14 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:12:16 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:12:18 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:12:20 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
[2023-06-25 11:12:20.181156 +08:00] WARN [src/rpc/synchronizer.rs:950] ntp response empty
[2023-06-25 11:12:22.168072 +08:00] ERROR [src/ebpf_dispatcher/ebpf_dispatcher.rs:467] The tracer_start() error. Kernel offset adapt failed. Provide the operating system name and the 'kernel-devel' package for developers to adapt.
[2023-06-25 11:12:22.168559 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:579] ebpf collector started
2023-06-25 11:12:22 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later.
2023-06-25 11:12:22 [eBPF] INFO Received limit_size (1024), the final value is set to '1024'
[2023-06-25 11:12:22.180429 +08:00] INFO [src/dispatcher/local_mode_dispatcher.rs:270] Stopped dispatcher (0)
[2023-06-25 11:12:22.182870 +08:00] INFO [src/dispatcher/base_dispatcher.rs:655] Decap tunnel type change to VXLAN IPIP
[2023-06-25 11:12:22.182977 +08:00] INFO [src/dispatcher/local_mode_dispatcher.rs:63] Start dispatcher (0)
[2023-06-25 11:12:22.186456 +08:00] INFO [src/dispatcher/base_dispatcher.rs:788] Dispatcher(0) Adding VMs: [00:00:00:00:00:00, fa:16:3e:d4:d3:38, f6:8f:f9:23:66:3d, 26:15:07:cf:59:38, 96:5c:5d:be:d3:2e, fa:1e:ec:d9:88:9b, 56:7b:6d:1d:0f:8e, f2:6b:c7:58:e1:13, d6:bd:1f:cc:09:67, 2a:36:b7:e9:38:fc, 7e:68:b5:9b:7b:28, 82:b4:61:2c:50:bd, be:e6:f4:77:fe:51, aa:46:75:6c:a7:24, 7a:03:ec:53:45:c8, 3a:c5:e5:df:5c:5b, 42:be:c2:19:c5:e7]
[2023-06-25 11:13:21.193344 +08:00] WARN [src/rpc/synchronizer.rs:950] ntp response empty
[2023-06-25 11:14:22.221218 +08:00] WARN [src/rpc/synchronizer.rs:950] ntp response empty
How to reproduce
[root@localhost show-kernel-struct-offset-master]# make
make -C /lib/modules/4.18.0-372.19.1.30.po1.x86_64/build M=/root/show-kernel-struct-offset-master modules
make[1]: 进入目录“/usr/src/kernels/4.18.0-372.19.1.30.po1.x86_64”
CC [M] /root/show-kernel-struct-offset-master/skso.o
Building modules, stage 2.
MODPOST 1 modules
CC /root/show-kernel-struct-offset-master/skso.mod.o
LD [M] /root/show-kernel-struct-offset-master/skso.ko
make[1]: 离开目录“/usr/src/kernels/4.18.0-372.19.1.30.po1.x86_64”
objdump -d skso.ko
VTAP_ID NAME TYPE CTRL_IP CTRL_MAC STATE GROUP EXCEPTIONS
1 node-2047f-1-V2 K8S_VM 10.65.182.193 fa:16:3e:87:4a:9f NORMAL default
2 node-8911c-0-V3 K8S_VM 10.65.182.192 fa:16:3e:e7:6b:02 NORMAL default
3 master-87b0e-0-V1 K8S_VM 10.65.182.190 fa:16:3e:d4:d3:38 NORMAL default
Search before asking
DeepFlow Component
Agent
What you expected to happen
2023-06-25 11:10:42 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. thread_num: 1, perf_pages_count: 128, ring_size: 65536, max_socket_entries: 524288, max_trace_entries: 524288, socket_map_max_reclaim: 520000, go_tracing_timeout: 0, io_event_collect_mode: 1, io_event_minimal_duration: 1ms, }, } to EbpfConfig { collector_enabled: true, l7_metrics_enabled: true, vtap_id: 3, epc_id: 1, l7_log_packet_size: 1024, l7_log_session_timeout: 120s, l7_protocol_inference_max_fail_count: 50, l7_protocol_inference_ttl: 60, l7_log_tap_types: [ ( 0, true, ), ], ctrl_mac: 00:00:00:00:00:00, l7_protocol_enabled_bitmap: [ Http1, Http2, DNS, SofaRPC, MySQL, Kafka, Redis, PostgreSQL, Dubbo, MQTT, ], l7_protocol_ports: { "DNS": "53", }, ebpf: EbpfYamlConfig { disabled: false, log_file: "", kprobe_whitelist: EbpfKprobeWhitelist { port_list: "", }, uprobe_proc_regexp: UprobeProcRegExp { golang_symbol: "", golang: ".", openssl: ".", }, thread_num: 1, perf_pages_count: 128, ring_size: 65536, max_socket_entries: 524288, max_trace_entries: 524288, socket_map_max_reclaim: 520000, go_tracing_timeout: 0, io_event_collect_mode: 1, io_event_minimal_duration: 1ms, }, } [2023-06-25 11:10:42.151625 +08:00] INFO [src/collector/quadruple_generator.rs:659] quadruple generator update collector_enabled to true [2023-06-25 11:10:42.151811 +08:00] INFO [src/collector/quadruple_generator.rs:659] quadruple generator update collector_enabled to true [2023-06-25 11:10:42.290964 +08:00] INFO [src/sender/uniform_sender.rs:335] 1-telegraf-to-sender sender update dst ip from 0.0.0.0 to 10.244.210.19 [2023-06-25 11:10:42.291193 +08:00] INFO [src/sender/uniform_sender.rs:346] 1-telegraf-to-sender sender update dst port from 30033 to 20033 [2023-06-25 11:10:42.315300 +08:00] INFO [src/sender/uniform_sender.rs:335] 1-compressed-otel-to-sender sender update dst ip from 0.0.0.0 to 10.244.210.19 [2023-06-25 11:10:42.315554 +08:00] INFO [src/sender/uniform_sender.rs:346] 1-compressed-otel-to-sender sender update dst port from 30033 to 20033 [2023-06-25 11:10:42.325733 +08:00] INFO [src/sender/uniform_sender.rs:335] 1-profile-to-sender sender update dst ip from 0.0.0.0 to 10.244.210.19 [2023-06-25 11:10:42.326122 +08:00] INFO [src/sender/uniform_sender.rs:335] 1-otel-to-sender sender update dst ip from 0.0.0.0 to 10.244.210.19 [2023-06-25 11:10:42.326282 +08:00] INFO [src/sender/uniform_sender.rs:346] 1-profile-to-sender sender update dst port from 30033 to 20033 [2023-06-25 11:10:42.326349 +08:00] INFO [src/sender/uniform_sender.rs:346] 1-otel-to-sender sender update dst port from 30033 to 20033 [2023-06-25 11:10:42.349862 +08:00] INFO [src/sender/uniform_sender.rs:335] 1-proc-event-to-sender sender update dst ip from 0.0.0.0 to 10.244.210.19 [2023-06-25 11:10:42.350383 +08:00] INFO [src/sender/uniform_sender.rs:346] 1-proc-event-to-sender sender update dst port from 30033 to 20033 [2023-06-25 11:10:42.362520 +08:00] INFO [src/sender/uniform_sender.rs:335] 1-prometheus-to-sender sender update dst ip from 0.0.0.0 to 10.244.210.19 [2023-06-25 11:10:42.362905 +08:00] INFO [src/sender/uniform_sender.rs:346] 1-prometheus-to-sender sender update dst port from 30033 to 20033 2023-06-25 11:10:44 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. [2023-06-25 11:10:44.321140 +08:00] INFO [src/sender/uniform_sender.rs:335] 2-packet-sequence-block-to-sender sender update dst ip from 0.0.0.0 to 10.244.210.19 [2023-06-25 11:10:44.321694 +08:00] INFO [src/sender/uniform_sender.rs:346] 2-packet-sequence-block-to-sender sender update dst port from 30033 to 20033 2023-06-25 11:10:46 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:10:48 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:10:50 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:10:52 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:10:54 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:10:56 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:10:58 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:00 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:02 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:04 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:06 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:08 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:10 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:12 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. [2023-06-25 11:11:14.139469 +08:00] ERROR [src/sender/uniform_sender.rs:391] 3-flowlog-to-collector-sender sender tcp connection to 0.0.0.0:30033 failed [2023-06-25 11:11:14.140164 +08:00] INFO [src/sender/uniform_sender.rs:335] 3-flowlog-to-collector-sender sender update dst ip from 0.0.0.0 to 10.244.210.19 [2023-06-25 11:11:14.140720 +08:00] INFO [src/sender/uniform_sender.rs:346] 3-flowlog-to-collector-sender sender update dst port from 30033 to 20033 2023-06-25 11:11:14 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:16 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:18 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. [2023-06-25 11:11:19.168826 +08:00] WARN [src/rpc/synchronizer.rs:950] ntp response empty 2023-06-25 11:11:20 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. [2023-06-25 11:11:21.642985 +08:00] INFO [src/rpc/synchronizer.rs:1298] ProxyController update to Some("10.244.210.19"):20035 2023-06-25 11:11:22 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:24 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:26 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:28 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:30 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:32 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:34 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:36 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:38 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:40 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. [2023-06-25 11:11:41.518922 +08:00] INFO [src/rpc/session.rs:501] rpc Proxy changed to proxy 10.244.210.19 20035 from proxy 10.197.212.207 30035 [2023-06-25 11:11:41.522222 +08:00] INFO [src/platform/platform_synchronizer/linux.rs:785] local version 1687655489, remote version 0 2023-06-25 11:11:42 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. [2023-06-25 11:11:42.811405 +08:00] ERROR [src/sender/uniform_sender.rs:391] 3-doc-to-collector-sender sender tcp connection to 0.0.0.0:30033 failed [2023-06-25 11:11:42.811470 +08:00] ERROR [src/sender/uniform_sender.rs:391] stats sender tcp connection to 0.0.0.0:30033 failed [2023-06-25 11:11:42.811405 +08:00] ERROR [src/sender/uniform_sender.rs:391] 2-protolog-to-collector-sender sender tcp connection to 0.0.0.0:30033 failed [2023-06-25 11:11:42.811928 +08:00] INFO [src/sender/uniform_sender.rs:335] 2-protolog-to-collector-sender sender update dst ip from 0.0.0.0 to 10.244.210.19 [2023-06-25 11:11:42.811965 +08:00] INFO [src/sender/uniform_sender.rs:335] 3-doc-to-collector-sender sender update dst ip from 0.0.0.0 to 10.244.210.19 [2023-06-25 11:11:42.812132 +08:00] INFO [src/sender/uniform_sender.rs:346] 2-protolog-to-collector-sender sender update dst port from 30033 to 20033 [2023-06-25 11:11:42.812099 +08:00] INFO [src/sender/uniform_sender.rs:346] 3-doc-to-collector-sender sender update dst port from 30033 to 20033 [2023-06-25 11:11:42.812174 +08:00] INFO [src/sender/uniform_sender.rs:335] stats sender update dst ip from 0.0.0.0 to 10.244.210.19 [2023-06-25 11:11:42.812332 +08:00] INFO [src/sender/uniform_sender.rs:346] stats sender update dst port from 30033 to 20033 2023-06-25 11:11:44 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:46 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:48 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:50 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:52 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:54 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:56 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:11:58 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:12:00 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:12:02 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:12:04 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:12:06 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:12:08 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:12:10 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:12:12 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:12:14 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:12:16 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:12:18 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:12:20 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. [2023-06-25 11:12:20.181156 +08:00] WARN [src/rpc/synchronizer.rs:950] ntp response empty [2023-06-25 11:12:22.168072 +08:00] ERROR [src/ebpf_dispatcher/ebpf_dispatcher.rs:467] The tracer_start() error. Kernel offset adapt failed. Provide the operating system name and the 'kernel-devel' package for developers to adapt. [2023-06-25 11:12:22.168559 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:579] ebpf collector started 2023-06-25 11:12:22 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1704] Adapting the linux kernel(4.18.0-372.19.1.30.po1.x86_64) is in progress, please try the start operation again later. 2023-06-25 11:12:22 [eBPF] INFO Received limit_size (1024), the final value is set to '1024' [2023-06-25 11:12:22.180429 +08:00] INFO [src/dispatcher/local_mode_dispatcher.rs:270] Stopped dispatcher (0) [2023-06-25 11:12:22.182870 +08:00] INFO [src/dispatcher/base_dispatcher.rs:655] Decap tunnel type change to VXLAN IPIP [2023-06-25 11:12:22.182977 +08:00] INFO [src/dispatcher/local_mode_dispatcher.rs:63] Start dispatcher (0) [2023-06-25 11:12:22.186456 +08:00] INFO [src/dispatcher/base_dispatcher.rs:788] Dispatcher(0) Adding VMs: [00:00:00:00:00:00, fa:16:3e:d4:d3:38, f6:8f:f9:23:66:3d, 26:15:07:cf:59:38, 96:5c:5d:be:d3:2e, fa:1e:ec:d9:88:9b, 56:7b:6d:1d:0f:8e, f2:6b:c7:58:e1:13, d6:bd:1f:cc:09:67, 2a:36:b7:e9:38:fc, 7e:68:b5:9b:7b:28, 82:b4:61:2c:50:bd, be:e6:f4:77:fe:51, aa:46:75:6c:a7:24, 7a:03:ec:53:45:c8, 3a:c5:e5:df:5c:5b, 42:be:c2:19:c5:e7] [2023-06-25 11:13:21.193344 +08:00] WARN [src/rpc/synchronizer.rs:950] ntp response empty [2023-06-25 11:14:22.221218 +08:00] WARN [src/rpc/synchronizer.rs:950] ntp response empty
How to reproduce
[root@localhost show-kernel-struct-offset-master]# make make -C /lib/modules/4.18.0-372.19.1.30.po1.x86_64/build M=/root/show-kernel-struct-offset-master modules make[1]: 进入目录“/usr/src/kernels/4.18.0-372.19.1.30.po1.x86_64” CC [M] /root/show-kernel-struct-offset-master/skso.o Building modules, stage 2. MODPOST 1 modules CC /root/show-kernel-struct-offset-master/skso.mod.o LD [M] /root/show-kernel-struct-offset-master/skso.ko make[1]: 离开目录“/usr/src/kernels/4.18.0-372.19.1.30.po1.x86_64” objdump -d skso.ko
skso.ko: 文件格式 elf64-x86-64
Disassembly of section .text:
0000000000000000:
0: e8 00 00 00 00 callq 5 <STRUCT_TCP_SOCK_COPIED_SEQ_OFFSET+0x5>
5: b8 5c 06 00 00 mov $0x65c,%eax
a: c3 retq
b: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
0000000000000010:
10: e8 00 00 00 00 callq 15 <STRUCT_TCP_SOCK_WRITE_SEQ_OFFSET+0x5>
15: b8 dc 07 00 00 mov $0x7dc,%eax
1a: c3 retq
1b: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
0000000000000020:
20: e8 00 00 00 00 callq 25 <STRUCT_TASK_FILES_OFFSET+0x5>
25: b8 20 0b 00 00 mov $0xb20,%eax
2a: c3 retq
2b: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
0000000000000030:
30: e8 00 00 00 00 callq 35 <STRUCT_SOCK_FLAGS_OFFSET+0x5>
35: b8 18 02 00 00 mov $0x218,%eax
3a: c3 retq
3b: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
0000000000000040:
40: e8 00 00 00 00 callq 45 <STRUCT_FILES_PRIVATE_DATA_OFFSET+0x5>
45: b8 c8 00 00 00 mov $0xc8,%eax
4a: c3 retq
4b: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
0000000000000050:
50: e8 00 00 00 00 callq 55 <STRUCT_TASK_TLS_OFFSET+0x5>
55: b8 a8 13 00 00 mov $0x13a8,%eax
5a: c3 retq
5b: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
0000000000000060:
60: e8 00 00 00 00 callq 65 <init_module+0x5>
65: 31 c0 xor %eax,%eax
67: c3 retq
68: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
6f: 00
0000000000000070:
70: e8 00 00 00 00 callq 75 <cleanup_module+0x5>
75: f3 c3 repz retq
[root@localhost show-kernel-struct-offset-master]#
DeepFlow version
[root@localhost show-kernel-struct-offset-master]# kubectl exec -it -n deepflow deploy/deepflow-server -- deepflow-server -v 2023/06/25 14:29:58 ENV K8S_NODE_NAME_FOR_DEEPFLOW=node-2047f-1; K8S_NODE_IP_FOR_DEEPFLOW=10.65.182.193; K8S_POD_NAME_FOR_DEEPFLOW=deepflow-server-7cddd864df-4v8p5; K8S_POD_IP_FOR_DEEPFLOW=10.244.210.19; K8S_NAMESPACE_FOR_DEEPFLOW=deepflow Name: deepflow-server community edition Branch: v6.2.6 CommitID: a994f079b4096b3c00f8ae8c0becb4b7a8bcbb68 RevCount: 8079 Compiler: go version go1.18.10 linux/amd64 CompileTime: 2023-04-07 07:03:29
[root@localhost show-kernel-struct-offset-master]# kubectl exec -it -n deepflow ds/deepflow-agent -- deepflow-agent -v Defaulted container "deepflow-agent" out of: deepflow-agent, configure-sysctl (init) 8079-a994f079b4096b3c00f8ae8c0becb4b7a8bcbb68 Name: deepflow-agent community edition Branch: v6.2.6 CommitId: a994f079b4096b3c00f8ae8c0becb4b7a8bcbb68 RevCount: 8079 Compiler: rustc 1.68.0 (2c8cc3432 2023-03-06) CompileTime: 2023-04-07 07:03:47
DeepFlow agent list
VTAP_ID NAME TYPE CTRL_IP CTRL_MAC STATE GROUP EXCEPTIONS 1 node-2047f-1-V2 K8S_VM 10.65.182.193 fa:16:3e:87:4a:9f NORMAL default 2 node-8911c-0-V3 K8S_VM 10.65.182.192 fa:16:3e:e7:6b:02 NORMAL default 3 master-87b0e-0-V1 K8S_VM 10.65.182.190 fa:16:3e:d4:d3:38 NORMAL default
Kubernetes CNI
calico
Operation-System/Kernel version
awk -F '=' '/PRETTY_NAME/ { print $2 }' /etc/os-release "PlatOS 1.3 (LTS)"
[root@localhost ~]# uname -r 4.18.0-372.19.1.30.po1.x86_64
自研系统
Anything else
No response
Are you willing to submit a PR?
Code of Conduct