search

deepflowio / deepflow

eBPF Observability - Distributed Tracing and Profiling
https://deepflow.io
Apache License 2.0
2.98k stars 333 forks source link

Distributed Tracing dashboard no data with eBPF demo [BUG] #2823

Open gamestart opened 1 year ago

gamestart commented 1 year ago

Search before asking

DeepFlow Component

Agent

What you expected to happen

I have installed deepflow v6.2.4 by helm , and deploy deepflow-otel-grpc-demo deepflow-ebpf-spring-demo(all running)，but grafana dashboards for Distributed Tracing have no data for deepflow-ebpf-spring-demo when it worked for deepflow-otel-grpc-demo.

How to reproduce

No response

DeepFlow version

kubectl exec -it -n deepflow deploy/deepflow-server -- deepflow-server -v 2023/04/04 16:18:44 ENV K8S_NODE_NAME_FOR_DEEPFLOW=smore; K8S_NODE_IP_FOR_DEEPFLOW=10.80.31.44; K8S_POD_NAME_FOR_DEEPFLOW=deepflow-server-7cf75b6ff7-bfl2s; K8S_POD_IP_FOR_DEEPFLOW=10.210.14.142; K8S_NAMESPACE_FOR_DEEPFLOW=deepflow Name: deepflow-server community edition Branch: v6.2.4 CommitID: c5f16a6bd88902f8b45d82ea9a6f8148d126700f RevCount: 7872 Compiler: go version go1.18.10 linux/amd64 CompileTime: 2023-03-08 02:18:32

DeepFlow agent list

No response

Kubernetes CNI

calico

Operation-System/Kernel version

5.4.0-109-generic

Anything else

here is the agent log

2023-04-04 15:36:01 ESC[0;35m[eBPF] WARNING: func load_obj__progs() [user/load.c:410] bcc_prog_load() failed. name: bpf_func_sys_exit_sendmmsg, errno: 22 ESC[0m2023-04-04 15:36:01 ESC[0;35m[eBPF] WARNING: func load_obj__progs() [user/load.c:414] The number of EBPF instructions (5778) exceeded the maximum limit (4096). ESC[0m2023-04-04 15:36:01 ESC[0;35m[eBPF] WARNING: func ebpf_obj_load() [user/load.c:715] eBPF load programs failed. (errno 22) ESC[0m2023-04-04 15:36:01 ESC[0;35m[eBPF] WARNING: func tracer_bpf_load() [user/tracer.c:231] bpf load "socket-trace-bpf-linux-5.2_plus" failed, error:Invalid argument (22).

Are you willing to submit a PR?

Code of Conduct

Nick-0314 commented 1 year ago

To try updating to 6.2.5, use the helm repo below

helm repo remove deepflow
helm repo add deepflow https://deepflow-ce.oss-cn-beijing.aliyuncs.com/chart/stable
Nick-0314 commented 1 year ago

If that doesn't work, try switching the deepflow-agent image tag to latest

gamestart commented 1 year ago

It worked, thanks. But another question is that there is no tracing data between a job pod and another pod managed by deployment/daemonset/statefulset.

Nick-0314 commented 1 year ago

We can try to search by using the IP of the client job pod. We have not learned the k8s job information

yantingqiu commented 1 year ago

I encountered the same problem, but the difference is that all my images are the latest version. How should I solve this problem?@dundun9 Server: image Agent: image

Nick-0314 commented 1 year ago

I encountered the same problem, but the difference is that all my images are the latest version. How should I solve this problem?@dundun9 Server: image Agent: image

Post the agent log

yantingqiu commented 1 year ago

I encountered the same problem, but the difference is that all my images are the latest version. How should I solve this problem?@dundun9 Server: image Agent: image

Post the agent log

2023-05-08 21:11:42 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1742] [eBPF Kernel Adapt] Adapting the linux kernel(4.18.0-372.9.1.15.po1.x86_64) is in progress, please try the start operation again later.
[2023-05-08 21:11:42.439569 +08:00] INFO [src/platform/kubernetes/active_poller.rs:213] kubernetes poller updated to version (2)
2023-05-08 21:11:44 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1742] [eBPF Kernel Adapt] Adapting the linux kernel(4.18.0-372.9.1.15.po1.x86_64) is in progress, please try the start operation again later.
2023-05-08 21:11:46 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1742] [eBPF Kernel Adapt] Adapting the linux kernel(4.18.0-372.9.1.15.po1.x86_64) is in progress, please try the start operation again later.
[2023-05-08 21:11:47.443754 +08:00] INFO [src/platform/kubernetes/api_watcher.rs:575] version updated to 1683551307
[2023-05-08 21:11:47.444751 +08:00] INFO [src/rpc/session.rs:564] rpc IP changed to controller 10.1.201.95 from unavailable proxy 10.244.0.46
2023-05-08 21:11:48 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1742] [eBPF Kernel Adapt] Adapting the linux kernel(4.18.0-372.9.1.15.po1.x86_64) is in progress, please try the start operation again later.
[2023-05-08 21:11:48.458915 +08:00] ERROR [src/rpc/session.rs:271] dial server(10.1.201.95) failed transport error
2023-05-08 21:11:50 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1742] [eBPF Kernel Adapt] Adapting the linux kernel(4.18.0-372.9.1.15.po1.x86_64) is in progress, please try the start operation again later.
2023-05-08 21:11:52 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1742] [eBPF Kernel Adapt] Adapting the linux kernel(4.18.0-372.9.1.15.po1.x86_64) is in progress, please try the start operation again later.
[2023-05-08 21:11:53.460980 +08:00] ERROR [src/platform/kubernetes/api_watcher.rs:637] kubernetes_api_sync grpc call failed: status: Unavailable, message: "error trying to connect: deadline has elapsed", details: [], metadata: MetadataMap { headers: {} }
2023-05-08 21:11:54 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1742] [eBPF Kernel Adapt] Adapting the linux kernel(4.18.0-372.9.1.15.po1.x86_64) is in progress, please try the start operation again later.
2023-05-08 21:11:56 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1742] [eBPF Kernel Adapt] Adapting the linux kernel(4.18.0-372.9.1.15.po1.x86_64) is in progress, please try the start operation again later.
2023-05-08 21:11:58 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1742] [eBPF Kernel Adapt] Adapting the linux kernel(4.18.0-372.9.1.15.po1.x86_64) is in progress, please try the start operation again later.
2023-05-08 21:12:00 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1742] [eBPF Kernel Adapt] Adapting the linux kernel(4.18.0-372.9.1.15.po1.x86_64) is in progress, please try the start operation again later.
[2023-05-08 21:12:02.428382 +08:00] ERROR [src/ebpf_dispatcher/ebpf_dispatcher.rs:467] [eBPF Kernel Adapt] The tracer_start() error. Kernel offset adapt failed. Please ensure that BTF is enabled (kernel built with CONFIG_DEBUG_INFO_BTF=y option). If the current kernel version is low (<5.3), upgrading the Linux kernel to 5.3+ (kernel built with CONFIG_DEBUG_INFO_BTF=y option) can solve the problem. If it is not possible to upgrade the kernel, the kernel-devel package can be provided for developers to adapt and solve the problem.
2023-05-08 21:12:02 [eBPF] WARNING: func socket_tracer_start() [user/socket.c:1742] [eBPF Kernel Adapt] Adapting the linux kernel(4.18.0-372.9.1.15.po1.x86_64) is in progress, please try the start operation again later.
2023-05-08 21:12:02 [eBPF] INFO Received limit_size (1024), the final value is set to '1024'
[2023-05-08 21:12:02.428734 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:585] ebpf collector started
[2023-05-08 21:12:02.435424 +08:00] INFO [src/dispatcher/base_dispatcher.rs:760] Dispatcher(0) Removing VMs: [46:e7:2f:92:95:0f, 7e:47:25:3f:0b:66, 5e:a7:c0:bc:d9:3
[2023-05-08 21:12:02.435636 +08:00] INFO [src/dispatcher/base_dispatcher.rs:793] Dispatcher(0) Adding VMs: [ca:90:5c:d3:81:6f, 06:8a:a6:0a:93:4d, ba:5a:3c:91:fb:d6, e2:ea:14:0b:0d:22, 6a:02:4f:8a:95:cb, a2:4c:9b:8d:27:28, f6:80:9e:cc:ad:dc, e2:b0:d9:cc:03:c0, 2a:f3:50:c7:11:16, 6a:0b:2d:7f:a8:13, d2:c1:21:cb:67:af, 8a:54:b5:01:74:01, c2:08:4d:ff:d7:88, 3a:5e:62:be:17:26, 52:78:34:32:1f:75, 32:73:84:86:13:83]
[2023-05-08 21:12:19.050775 +08:00] ERROR [src/sender/uniform_sender.rs:395] 3-flowlog-to-collector-sender sender tcp connection to 10.244.0.46:20033 failed
[2023-05-08 21:12:24.760044 +08:00] INFO [src/rpc/synchronizer.rs:1227] TapMode: Local, CtrlMac: 00:50:56:b2:41:b3, CtrlIp: 10.131.139.9, Hostname: master
[2023-05-08 21:12:24.760623 +08:00] INFO [src/rpc/session.rs:595] rpc IP changed to controller 10.1.201.95 30035 from unavailable controller 10.1.201.95 30035
[2023-05-08 21:12:24.775143 +08:00] INFO [src/platform/platform_synchronizer/linux.rs:501] Platform information changed to version 1683551307
[2023-05-08 21:12:24.775288 +08:00] INFO [src/platform/platform_synchronizer/linux.rs:746] local version changed to 1683551307
[2023-05-08 21:12:24.775396 +08:00] INFO [src/rpc/session.rs:595] rpc IP changed to controller 10.1.201.95 30035 from unavailable controller 10.1.201.95 30035
[2023-05-08 21:12:25.770875 +08:00] ERROR [src/rpc/session.rs:271] dial server(10.1.201.95) failed transport error
[2023-05-08 21:12:25.834856 +08:00] ERROR [src/rpc/session.rs:271] dial server(10.1.201.95) failed transport error
[2023-05-08 21:12:26.730915 +08:00] ERROR [src/sender/uniform_sender.rs:395] 2-protolog-to-collector-sender sender tcp connection to 10.244.0.46:20033 failed
[2023-05-08 21:12:26.731626 +08:00] ERROR [src/rpc/synchronizer.rs:1249] grpc sync error, server 10.1.201.95 30035 unavailable, status-code The service is currently unavailable, message: "error trying to connect: tcp connect error: No route to host (os error 113)"
[2023-05-08 21:12:27.812747 +08:00] INFO [src/rpc/session.rs:595] rpc IP changed to controller 10.1.201.95 30035 from unavailable controller 10.1.201.95 30035
[2023-05-08 21:12:28.843807 +08:00] ERROR [src/rpc/session.rs:271] dial server(10.1.201.95) failed transport error
[2023-05-08 21:12:29.803188 +08:00] ERROR [src/platform/platform_synchronizer/linux.rs:765] send platform information with genesis_sync grpc call failed: status: Unavailable, message: "error trying to connect: tcp connect error: No route to host (os error 113)", details: [], metadata: MetadataMap { headers: {} }
[2023-05-08 21:12:32.874997 +08:00] WARN [src/rpc/synchronizer.rs:951] ntp request failed with: Status { code: Unavailable, message: "error trying to connect: tcp connect error: No route to host (os error 113)", source: Some(tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 113, kind: HostUnreachable, message: "No route to host" })))) }
[2023-05-08 21:12:35.947044 +08:00] ERROR [src/sender/uniform_sender.rs:395] 3-flowlog-to-collector-sender sender tcp connection to 10.244.0.46:20033 failed
[2023-05-08 21:12:39.019848 +08:00] ERROR [src/sender/uniform_sender.rs:395] 2-protolog-to-collector-sender sender tcp connection to 10.244.0.46:20033 failed
[2023-05-08 21:12:40.451906 +08:00] INFO [src/rpc/session.rs:595] rpc IP changed to controller 10.1.201.95 30035 from unavailable controller 10.1.201.95 30035
[2023-05-08 21:12:41.514766 +08:00] ERROR [src/rpc/session.rs:271] dial server(10.1.201.95) failed transport error
[2023-05-08 21:12:42.092049 +08:00] ERROR [src/rpc/synchronizer.rs:805] rpc error Status { code: Unavailable, message: "error trying to connect: tcp connect error: No route to host (os error 113)", source: Some(tonic::transport::Error(Transport, hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 113, kind: HostUnreachable, message: "No route to host" })))) }
[2023-05-08 21:12:48.234805 +08:00] ERROR [src/sender/uniform_sender.rs:395] 3-doc-to-collector-sender sender tcp connection to 10.244.0.46:20033 failed
[2023-05-08 21:13:25.611030 +08:00] ERROR [src/sender/uniform_sender.rs:395] 2-protolog-to-collector-sender sender tcp connection to 10.244.0.46:20033 failed
[2023-05-08 21:13:26.732751 +08:00] INFO [src/rpc/synchronizer.rs:1227] TapMode: Local, CtrlMac: 00:50:56:b2:41:b3, CtrlIp: 10.131.139.9, Hostname: master
[2023-05-08 21:13:26.733474 +08:00] INFO [src/rpc/session.rs:595] rpc IP changed to controller 10.1.201.95 30035 from unavailable controller 10.1.201.95 30035
[2023-05-08 21:13:26.741774 +08:00] INFO [src/rpc/synchronizer.rs:190] Update PlatformData version 1688365452 to 1688558591.
[2023-05-08 21:13:26.742185 +08:00] INFO [src/rpc/synchronizer.rs:202] Update IpGroups version 1683362468 to 1683551555.
[2023-05-08 21:13:26.742237 +08:00] INFO [src/rpc/synchronizer.rs:212] Update FlowAcls version 2683362668 to 2683551755.
[2023-05-08 21:13:26.742260 +08:00] INFO [src/rpc/synchronizer.rs:735] Grpc version ip-groups: 1683551555, interfaces, peer-connections and cidrs: 1688558591, flow-acls: 2683551755
[2023-05-08 21:13:26.746453 +08:00] INFO [src/rpc/synchronizer.rs:751] Grpc finish update cost 4.189797ms on 3 listener, 1 ip-groups, 73 interfaces, 0 peer-connections, 4 cidrs, 0 flow-acls
[2023-05-08 21:13:26.749055 +08:00] INFO [src/config/handler.rs:1425] dispatcher config change from DispatcherConfig {
global_pps_threshold: 200000,
capture_packet_size: 65535,
l7_log_packet_size: 1024,
tunnel_type_bitmap: TunnelTypeBitmap(
6,
),
trident_type: TtVmPod,
vtap_id: 1,
capture_socket_type: Auto,
extra_netns_regex: "",
tap_interface_regex: "^(tap.*|cali.*|veth.*|eth.*|en[ospx].*|lxc.*|lo|[0-9a-f]+_h)$",
if_mac_source: IfMac,
analyzer_ip: "10.244.0.46",
analyzer_port: 20033,
proxy_controller_ip: "10.244.0.46",
proxy_controller_port: 20035,
capture_bpf: "",
max_memory: 805306368,
af_packet_blocks: 48,
af_packet_version: TpacketVersionHighestavailablet,
tap_mode: Local,
region_id: 1,
pod_cluster_id: 1,
enabled: true,
npb_dedup_enabled: true,
} to DispatcherConfig {
global_pps_threshold: 200000,
capture_packet_size: 65535,
l7_log_packet_size: 1024,
tunnel_type_bitmap: TunnelTypeBitmap(
6,
),
trident_type: TtVmPod,
vtap_id: 1,
capture_socket_type: Auto,
extra_netns_regex: "",
tap_interface_regex: "^(tap.*|cali.*|veth.*|eth.*|en[ospx].*|lxc.*|lo|[0-9a-f]+_h)$",
if_mac_source: IfMac,
analyzer_ip: "10.244.0.53",
analyzer_port: 20033,
proxy_controller_ip: "10.244.0.46",
proxy_controller_port: 20035,
capture_bpf: "",
max_memory: 805306368,
af_packet_blocks: 48,
af_packet_version: TpacketVersionHighestavailablet,
tap_mode: Local,
region_id: 1,
pod_cluster_id: 1,
enabled: true,
npb_dedup_enabled: true,
}
[2023-05-08 21:13:26.749229 +08:00] INFO [src/config/handler.rs:1488] Rsyslog client connect to 10.244.0.53 20033
[2023-05-08 21:13:26.749261 +08:00] INFO [src/config/handler.rs:1501] stats config change from StatsConfig {
interval: 10s,
host: "master-V1",
analyzer_ip: "10.244.0.46",
analyzer_port: 20033,
} to StatsConfig {
interval: 10s,
host: "master-V1",
analyzer_ip: "10.244.0.53",
analyzer_port: 20033,
}
[2023-05-08 21:13:26.749383 +08:00] INFO [src/config/handler.rs:1803] sender config change from SenderConfig {
mtu: 1500,
dest_ip: "10.244.0.46",
vtap_id: 1,
dest_port: 20033,
npb_port: 4789,
vxlan_flags: 255,
npb_enable_qos_bypass: false,
npb_vlan: 0,
npb_vlan_mode: None,
npb_dedup_enabled: true,
npb_bps_threshold: 1000000000,
npb_socket_type: RawUdp,
collector_socket_type: Tcp,
standalone_data_file_size: 200,
standalone_data_file_dir: "/var/log/deepflow-agent",
server_tx_bandwidth_threshold: 0,
bandwidth_probe_interval: 10s,
enabled: true,
} to SenderConfig {
mtu: 1500,
dest_ip: "10.244.0.53",
vtap_id: 1,
dest_port: 20033,
npb_port: 4789,
vxlan_flags: 255,
npb_enable_qos_bypass: false,
npb_vlan: 0,
npb_vlan_mode: None,
npb_dedup_enabled: true,
npb_bps_threshold: 1000000000,
npb_socket_type: RawUdp,
collector_socket_type: Tcp,
standalone_data_file_size: 200,
standalone_data_file_dir: "/var/log/deepflow-agent",
server_tx_bandwidth_threshold: 0,
bandwidth_probe_interval: 10s,
enabled: true,
}
[2023-05-08 21:13:27.408570 +08:00] INFO [src/sender/uniform_sender.rs:335] 2-packet-sequence-block-to-sender sender update dst ip from 10.244.0.46 to 10.244.0.53
[2023-05-08 21:13:28.419151 +08:00] INFO [src/sender/uniform_sender.rs:335] 1-profile-to-sender sender update dst ip from 10.244.0.46 to 10.244.0.53
[2023-05-08 21:13:28.419199 +08:00] INFO [src/sender/uniform_sender.rs:335] 1-otel-to-sender sender update dst ip from 10.244.0.46 to 10.244.0.53
[2023-05-08 21:13:28.419199 +08:00] INFO [src/sender/uniform_sender.rs:335] 1-compressed-otel-to-sender sender update dst ip from 10.244.0.46 to 10.244.0.53
[2023-05-08 21:13:28.419616 +08:00] INFO [src/sender/uniform_sender.rs:335] 1-proc-event-to-sender sender update dst ip from 10.244.0.46 to 10.244.0.53
[2023-05-08 21:13:28.419873 +08:00] INFO [src/sender/uniform_sender.rs:335] 1-prometheus-to-sender sender update dst ip from 10.244.0.46 to 10.244.0.53
[2023-05-08 21:13:28.420581 +08:00] INFO [src/sender/uniform_sender.rs:335] 1-telegraf-to-sender sender update dst ip from 10.244.0.46 to 10.244.0.53
[2023-05-08 21:13:28.683063 +08:00] ERROR [src/sender/uniform_sender.rs:395] 3-doc-to-collector-sender sender tcp connection to 10.244.0.46:20033 failed
[2023-05-08 21:13:28.683116 +08:00] ERROR [src/sender/uniform_sender.rs:395] 3-flowlog-to-collector-sender sender tcp connection to 10.244.0.46:20033 failed
[2023-05-08 21:13:29.809986 +08:00] INFO [src/platform/platform_synchronizer/linux.rs:746] local version changed to 1683551307
[2023-05-08 21:13:29.810559 +08:00] INFO [src/rpc/session.rs:614] rpc IP changed to proxy 10.244.0.46 20035 from controller 10.1.201.95 30035
[2023-05-08 21:13:31.755181 +08:00] ERROR [src/rpc/session.rs:271] dial server(10.244.0.46) failed transport error
[2023-05-08 21:13:32.876270 +08:00] INFO [src/rpc/session.rs:564] rpc IP changed to controller 10.1.201.95 from unavailable proxy 10.244.0.46
[2023-05-08 21:13:33.894884 +08:00] WARN [src/rpc/synchronizer.rs:957] ntp response empty
[2023-05-08 21:13:37.611638 +08:00] INFO [src/sender/uniform_sender.rs:335] 2-protolog-to-collector-sender sender update dst ip from 10.244.0.46 to 10.244.0.53
[2023-05-08 21:13:37.612187 +08:00] ERROR [src/sender/uniform_sender.rs:395] 2-protolog-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:13:39.683811 +08:00] INFO [src/sender/uniform_sender.rs:335] 3-doc-to-collector-sender sender update dst ip from 10.244.0.46 to 10.244.0.53
[2023-05-08 21:13:39.684213 +08:00] ERROR [src/sender/uniform_sender.rs:395] 3-doc-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:13:41.683731 +08:00] INFO [src/sender/uniform_sender.rs:335] 3-flowlog-to-collector-sender sender update dst ip from 10.244.0.46 to 10.244.0.53
[2023-05-08 21:13:41.684088 +08:00] ERROR [src/sender/uniform_sender.rs:395] 3-flowlog-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:13:42.094160 +08:00] INFO [src/rpc/session.rs:595] rpc IP changed to controller 10.1.201.95 30035 from unavailable controller 10.1.201.95 30035
[2023-05-08 21:13:42.100505 +08:00] INFO [src/config/handler.rs:1425] dispatcher config change from DispatcherConfig {
global_pps_threshold: 200000,
capture_packet_size: 65535,
l7_log_packet_size: 1024,
tunnel_type_bitmap: TunnelTypeBitmap(
6,
),
trident_type: TtVmPod,
vtap_id: 1,
capture_socket_type: Auto,
extra_netns_regex: "",
tap_interface_regex: "^(tap.*|cali.*|veth.*|eth.*|en[ospx].*|lxc.*|lo|[0-9a-f]+_h)$",
if_mac_source: IfMac,
analyzer_ip: "10.244.0.53",
analyzer_port: 20033,
proxy_controller_ip: "10.244.0.46",
proxy_controller_port: 20035,
capture_bpf: "",
max_memory: 805306368,
af_packet_blocks: 48,
af_packet_version: TpacketVersionHighestavailablet,
tap_mode: Local,
region_id: 1,
pod_cluster_id: 1,
enabled: true,
npb_dedup_enabled: true,
} to DispatcherConfig {
global_pps_threshold: 200000,
capture_packet_size: 65535,
l7_log_packet_size: 1024,
tunnel_type_bitmap: TunnelTypeBitmap(
6,
),
trident_type: TtVmPod,
vtap_id: 1,
capture_socket_type: Auto,
extra_netns_regex: "",
tap_interface_regex: "^(tap.*|cali.*|veth.*|eth.*|en[ospx].*|lxc.*|lo|[0-9a-f]+_h)$",
if_mac_source: IfMac,
analyzer_ip: "10.244.0.53",
analyzer_port: 20033,
proxy_controller_ip: "10.244.0.53",
proxy_controller_port: 20035,
capture_bpf: "",
max_memory: 805306368,
af_packet_blocks: 48,
af_packet_version: TpacketVersionHighestavailablet,
tap_mode: Local,
region_id: 1,
pod_cluster_id: 1,
enabled: true,
npb_dedup_enabled: true,
}
[2023-05-08 21:13:42.460462 +08:00] INFO [src/platform/kubernetes/active_poller.rs:213] kubernetes poller updated to version (3)
[2023-05-08 21:13:50.612755 +08:00] ERROR [src/sender/uniform_sender.rs:395] 2-protolog-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:13:51.684736 +08:00] ERROR [src/sender/uniform_sender.rs:395] 3-flowlog-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:13:53.575872 +08:00] INFO [src/platform/kubernetes/api_watcher.rs:575] version updated to 1683551309
[2023-05-08 21:13:53.576647 +08:00] INFO [src/rpc/session.rs:614] rpc IP changed to proxy 10.244.0.46 20035 from controller 10.1.201.95 30035
[2023-05-08 21:13:53.684793 +08:00] ERROR [src/sender/uniform_sender.rs:395] 3-doc-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:13:54.795985 +08:00] ERROR [src/rpc/session.rs:271] dial server(10.244.0.46) failed transport error
[2023-05-08 21:14:00.613430 +08:00] ERROR [src/sender/uniform_sender.rs:395] 2-protolog-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:14:02.685477 +08:00] ERROR [src/sender/uniform_sender.rs:395] 3-flowlog-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:14:07.685341 +08:00] ERROR [src/sender/uniform_sender.rs:395] 3-doc-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:14:14.614139 +08:00] ERROR [src/sender/uniform_sender.rs:395] 2-protolog-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:14:14.686189 +08:00] ERROR [src/sender/uniform_sender.rs:395] 3-flowlog-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:14:19.685872 +08:00] ERROR [src/sender/uniform_sender.rs:395] 3-doc-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:14:25.614841 +08:00] ERROR [src/sender/uniform_sender.rs:395] 2-protolog-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:14:26.686839 +08:00] ERROR [src/sender/uniform_sender.rs:395] 3-flowlog-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:14:26.749724 +08:00] INFO [src/rpc/synchronizer.rs:1227] TapMode: Local, CtrlMac: 00:50:56:b2:41:b3, CtrlIp: 10.131.139.9, Hostname: master
[2023-05-08 21:14:26.750375 +08:00] INFO [src/rpc/session.rs:564] rpc IP changed to controller 10.1.201.95 from unavailable proxy 10.244.0.46
[2023-05-08 21:14:26.752929 +08:00] INFO [src/rpc/synchronizer.rs:1305] ProxyController update to Some("10.244.0.53"):20035
[2023-05-08 21:14:26.760693 +08:00] INFO [src/dispatcher/base_dispatcher.rs:760] Dispatcher(0) Removing VMs: [c2:08:4d:ff:d7:88]
[2023-05-08 21:14:29.686418 +08:00] ERROR [src/sender/uniform_sender.rs:395] 3-doc-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:14:31.762899 +08:00] INFO [src/platform/platform_synchronizer/linux.rs:501] Platform information changed to version 1683551308
[2023-05-08 21:14:31.763155 +08:00] INFO [src/platform/platform_synchronizer/linux.rs:746] local version changed to 1683551308
[2023-05-08 21:14:31.763370 +08:00] INFO [src/rpc/session.rs:614] rpc IP changed to proxy 10.244.0.53 20035 from controller 10.1.201.95 30035
[2023-05-08 21:14:34.962051 +08:00] WARN [src/rpc/synchronizer.rs:957] ntp response empty
[2023-05-08 21:14:37.615795 +08:00] ERROR [src/sender/uniform_sender.rs:395] 2-protolog-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:14:40.687493 +08:00] ERROR [src/sender/uniform_sender.rs:395] 3-flowlog-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:14:43.686974 +08:00] ERROR [src/sender/uniform_sender.rs:395] 3-doc-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:14:51.616656 +08:00] ERROR [src/sender/uniform_sender.rs:395] 2-protolog-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:14:52.688208 +08:00] ERROR [src/sender/uniform_sender.rs:395] 3-flowlog-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:14:54.687758 +08:00] ERROR [src/sender/uniform_sender.rs:395] 3-doc-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:15:03.688973 +08:00] ERROR [src/sender/uniform_sender.rs:395] 3-flowlog-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:15:05.617478 +08:00] ERROR [src/sender/uniform_sender.rs:395] 2-protolog-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:15:08.688518 +08:00] ERROR [src/sender/uniform_sender.rs:395] 3-doc-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:15:13.689791 +08:00] ERROR [src/sender/uniform_sender.rs:395] 3-flowlog-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:15:16.618176 +08:00] ERROR [src/sender/uniform_sender.rs:395] 2-protolog-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:15:19.689180 +08:00] ERROR [src/sender/uniform_sender.rs:395] 3-doc-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:15:25.690590 +08:00] ERROR [src/sender/uniform_sender.rs:395] 3-flowlog-to-collector-sender sender tcp connection to 10.244.0.53:20033 failed
[2023-05-08 21:15:30.619120 +08:00] INFO [src/sender/uniform_sender.rs:384] 2-protolog-to-collector-sender sender tcp connection to 10.244.0.53:20033 succeed.
[2023-05-08 21:15:32.689812 +08:00] INFO [src/sender/uniform_sender.rs:384] 3-doc-to-collector-sender sender tcp connection to 10.244.0.53:20033 succeed.
[2023-05-08 21:15:35.345596 +08:00] INFO [src/rpc/synchronizer.rs:815] grpc server changed
[2023-05-08 21:15:35.348620 +08:00] INFO [src/rpc/synchronizer.rs:190] Update PlatformData version 1688558591 to 1688558592.
[2023-05-08 21:15:35.348849 +08:00] INFO [src/rpc/synchronizer.rs:735] Grpc version ip-groups: 1683551555, interfaces, peer-connections and cidrs: 1688558592, flow-acls: 2683551755
[2023-05-08 21:15:35.352166 +08:00] INFO [src/rpc/synchronizer.rs:751] Grpc finish update cost 3.311593ms on 3 listener, 1 ip-groups, 73 interfaces, 0 peer-connections, 4 cidrs, 0 flow-acls
[2023-05-08 21:15:35.987837 +08:00] WARN [src/rpc/synchronizer.rs:957] ntp response empty
[2023-05-08 21:15:37.691483 +08:00] INFO [src/sender/uniform_sender.rs:384] 3-flowlog-to-collector-sender sender tcp connection to 10.244.0.53:20033 succeed.
[2023-05-08 21:16:35.274577 +08:00] INFO [src/rpc/synchronizer.rs:190] Update PlatformData version 1688558592 to 1688558593.
[2023-05-08 21:16:35.274863 +08:00] INFO [src/rpc/synchronizer.rs:735] Grpc version ip-groups: 1683551555, interfaces, peer-connections and cidrs: 1688558593, flow-acls: 2683551755
[2023-05-08 21:16:35.278821 +08:00] INFO [src/rpc/synchronizer.rs:751] Grpc finish update cost 3.955574ms on 3 listener, 1 ip-groups, 73 interfaces, 0 peer-connections, 4 cidrs, 0 flow-acls
[2023-05-08 21:16:37.003808 +08:00] WARN [src/rpc/synchronizer.rs:957] ntp response empty
[2023-05-08 21:16:42.488678 +08:00] INFO [src/platform/kubernetes/active_poller.rs:213] kubernetes poller updated to version (4)
Nick-0314 commented 1 year ago

eBPF is not adapted successfully Install the kernel-kernel package with gcc and the same version of the kernel, pull the project, and make a try, be careful not to run it on an online machine

https://github.com/nrjatyunshan/show-kernel-struct-offset/tree/master

yantingqiu commented 1 year ago

eBPF is not adapted successfully Install the kernel-kernel package with gcc and the same version of the kernel, pull the project, and make a try, be careful not to run it on an online machine

https://github.com/nrjatyunshan/show-kernel-struct-offset/tree/master

Thank you! I have created a new issue. https://github.com/deepflowio/deepflow/issues/3093