search

deepflowio / deepflow

eBPF Observability - Distributed Tracing and Profiling
https://deepflow.io
Apache License 2.0
2.87k stars 317 forks source link

[BUG] #8308

Closed liuqian1990 closed 4 days ago

liuqian1990 commented 5 days ago

Search before asking

DeepFlow Component

Agent

What you expected to happen

root@deepflow-agent-4xrk7:/# cat /var/log/deepflow-agent/deepflow-agent.log | grep -i ebpf
[2024-10-15 20:01:19.272877 +08:00] INFO [src/config/handler.rs:2692] ebpf config change from EbpfConfig {
    ebpf: EbpfYamlConfig {
        global_ebpf_pps_threshold: 0,
        kprobe_whitelist: EbpfKprobePortlist {
        kprobe_blacklist: EbpfKprobePortlist {
} to EbpfConfig {
    ebpf: EbpfYamlConfig {
        global_ebpf_pps_threshold: 0,
        kprobe_whitelist: EbpfKprobePortlist {
        kprobe_blacklist: EbpfKprobePortlist {
[2024-10-15 20:01:19.332859 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:673] ebpf collector init...
[2024-10-15 20:01:19.332923 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:420] ebpf golang uprobe proc regexp is empty, skip set
[2024-10-15 20:01:19.332937 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:436] ebpf openssl uprobe proc regexp is empty, skip set
[2024-10-15 20:01:19.332948 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:459] ebpf golang symbol proc regexp is empty, skip set
[2024-10-15 20:01:19.332961 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:464] l7 protocol MySQL parse enabled
[2024-10-15 20:01:19.333041 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] WARN func fs_read() [user/common.c:407] Open debug file("/proc/sys/net/core/bpf_jit_enable") open failed.
[2024-10-15 20:01:19.333074 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] WARN func bpf_tracer_init() [user/tracer.c:1623] "/proc/sys/net/core/bpf_jit_enable value is invalid
[2024-10-15 20:01:19.333139 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO log2_default_hugepage_sz 21
[2024-10-15 20:01:19.333201 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] WARN func fetch_kernel_version() [user/common.c:629] release 6.1.58+ version #1 SMP PREEMPT_DYNAMIC Mon Jan 29 15:19:25 UTC 2024 (major 6 minor 1 rev 58 num 31183)
[2024-10-15 20:01:19.333233 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] WARN func fetch_kernel_version_code() [user/common.c:680] fetch_kernel_version error
[2024-10-15 20:01:19.333251 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO linux version : 6.1.58+ (version code : 0)
[2024-10-15 20:01:19.333274 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO RLIMIT_NOFILE cur:1048576, rlim_max:1048576
[2024-10-15 20:01:19.333351 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO sys_boot_time_ns : 1721307761189667978
[2024-10-15 20:01:19.333452 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO open(/sys/kernel/debug/tracing/kprobe_events): failed Read-only file system
[2024-10-15 20:01:19.333574 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO open(/sys/kernel/debug/tracing/uprobe_events): failed Read-only file system
[2024-10-15 20:01:19.333725 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO [deepflow-ebpfctl] sockopt register succeed, type get: 404 - 404 set: 400 - 403
[2024-10-15 20:01:19.333876 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO register_period_event_op 'kick_kern' succeed.
[2024-10-15 20:01:19.333960 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO register_period_event_op 'boot time update' succeed.
[2024-10-15 20:01:19.334116 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO Tracer 'socket-trace', Not Found.
[2024-10-15 20:01:19.334153 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO Tracer 'socket-trace', Not Found.
[2024-10-15 20:01:19.334176 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO Tracer 'socket-trace', Not Found.
[2024-10-15 20:01:19.334450 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO Set ports_bitmap[DNS] success, ports 53,5353
[2024-10-15 20:01:19.334636 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO Set ports_bitmap[TLS] success, ports 443,6443
[2024-10-15 20:01:19.334835 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO Set ports_bitmap[Oracle] success, ports 1521
[2024-10-15 20:01:19.335057 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO Set ports_bitmap[HTTP2] success, ports 1-65535
[2024-10-15 20:01:19.335297 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO Set ports_bitmap[Custom] success, ports 1-65535
[2024-10-15 20:01:19.335549 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO Set ports_bitmap[MySQL] success, ports 1-65535
[2024-10-15 20:01:19.335874 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO Set ports_bitmap[HTTP1] success, ports 1-65535
[2024-10-15 20:01:19.336182 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO Set ports_bitmap[Kafka] success, ports 1-65535
[2024-10-15 20:01:19.336694 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO Set ports_bitmap[Redis] success, ports 1-65535
[2024-10-15 20:01:19.336948 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO Set ports_bitmap[MongoDB] success, ports 1-65535
[2024-10-15 20:01:19.337224 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO Set ports_bitmap[PgSQL] success, ports 1-65535
[2024-10-15 20:01:19.337573 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO Set ports_bitmap[FastCGI] success, ports 1-65535
[2024-10-15 20:01:19.338106 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO Set ports_bitmap[Dubbo] success, ports 1-65535
[2024-10-15 20:01:19.338730 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO Set ports_bitmap[SofaRPC] success, ports 1-65535
[2024-10-15 20:01:19.339018 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] INFO Set ports_bitmap[MQTT] success, ports 1-65535
[2024-10-15 20:01:19.339209 +08:00] INFO [src/ebpf/mod.rs:634] [eBPF] WARN func fetch_kernel_version() [user/common.c:629] release 6.1.58+ version #1 SMP PREEMPT_DYNAMIC Mon Jan 29 15:19:25 UTC 2024 (major 6 minor 1 rev 58 num 0)
[2024-10-15 20:01:19.339300 +08:00] ERROR [src/trident.rs:2368] ebpf collector error: EbpfRunningError

How to reproduce

no

DeepFlow version

no

DeepFlow agent list

no

Kubernetes CNI

k8s cni cilium

Operation-System/Kernel version

root@deepflow-agent-4xrk7:/# uname -r 6.1.58+

Anything else

no

Are you willing to submit a PR?

Code of Conduct

Ghostbaby commented 5 days ago
1473371932 commented 4 days ago 
root@deepflow-agent-gctkn:/# deepflow-agent -v
10911-8e3cbb3b2ff975627906c6d02d9c75011ab56c4a
Name: deepflow-agent community edition
Branch: v6.5
CommitId: 8e3cbb3b2ff975627906c6d02d9c75011ab56c4a
RevCount: 10911
Compiler: rustc 1.77.1 (7cf61ebde 2024-03-27)
CompileTime: 2024-10-15 07:47:08
yinjiping commented 4 days ago

release 为6.1.58+这个版本的内核解析不了，无法解析内核版本后面无法选择加载哪一个ebpf字节码二进制文件。

我们适配一下

@Ghostbaby

liuqian1990 commented 4 days ago

使用的是gcp gke 

root@deepflow-agent-952cj:/# uname -a
Linux deepflow-agent-952cj 6.1.58+ #1 SMP PREEMPT_DYNAMIC Mon Jan 29 15:19:25 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
root@deepflow-agent-952cj:/# uname -r 
6.1.58+
root@deepflow-agent-952cj:/# uname -v
#1 SMP PREEMPT_DYNAMIC Mon Jan 29 15:19:25 UTC 2024
liuqian1990 commented 4 days ago

现像有部分数据已经采集到，有部分没有

liuqian1990 commented 4 days ago

现像有部分数据已经采集到，有部分没有

[2024-10-16 10:21:05.781150 +08:00] INFO [src/trident.rs:376] static_config Config {
    controller_ips: [
        "10.0.34.139",
    ],
    controller_port: 30035,
    controller_tls_port: 30135,
    controller_cert_file_prefix: "",
    log_file: "/var/log/deepflow-agent/deepflow-agent.log",
    kubernetes_cluster_id: "",
    kubernetes_cluster_name: None,
    vtap_group_id_request: "g-4834bb8687",
    controller_domain_name: [
        "woo-deepflow-server",
    ],
    agent_mode: Managed,
    override_os_hostname: None,
    async_worker_thread_number: 16,
    agent_unique_identifier: IpMac,
    pid_file: "",
    team_id: "",
    cgroups_disabled: false,
}
[2024-10-16 10:21:05.782505 +08:00] INFO [src/trident.rs:426] ==================== Launching DeepFlow-Agent ====================
[2024-10-16 10:21:05.786829 +08:00] INFO [src/trident.rs:427] Environment variables: "K8S_NODE_IP_FOR_DEEPFLOW=10.0.15.203 CTRL_NETWORK_INTERFACE= K8S_POD_IP_FOR_DEEPFLOW=10.4.42.234 IN_CONTAINER=yes ONLY_WATCH_K8S_RESOURCE= K8S_NAMESPACE_FOR_DEEPFLOW=woo"
[2024-10-16 10:21:05.786898 +08:00] INFO [src/trident.rs:430] use K8S_NODE_IP_FOR_DEEPFLOW env ip as destination_ip(10.4.42.234)
[2024-10-16 10:21:05.788161 +08:00] INFO [src/trident.rs:469] agent 10.0.15.203/42:01:0a:00:0f:cb running in Managed mode, ctrl_ip 10.4.42.234 ctrl_mac ca:ee:a6:56:92:f4
[2024-10-16 10:21:05.830678 +08:00] INFO [src/config/config.rs:224] set kubernetes_cluster_id to d-rws6NYfvDR
[2024-10-16 10:21:05.886455 +08:00] INFO [src/rpc/remote_exec.rs:322] Started remote executor
[2024-10-16 10:21:05.886540 +08:00] INFO [src/trident.rs:1204] Resolve controller domain name woo-deepflow-server 10.0.34.139
[2024-10-16 10:21:05.886722 +08:00] INFO [src/trident.rs:566] don't initialize cgroups controller, because agent is running in container
[2024-10-16 10:21:05.886928 +08:00] INFO [src/utils/guard.rs:472] guard started
[2024-10-16 10:21:05.888958 +08:00] INFO [src/rpc/remote_exec.rs:259] grpc server changed
[2024-10-16 10:21:05.891123 +08:00] INFO [src/rpc/synchronizer.rs:1426] hostname changed from "" to "gke-e2-highmem-8-4e116e47-pgj2"
[2024-10-16 10:21:05.899612 +08:00] INFO [src/rpc/synchronizer.rs:206] Update PlatformData version 0 to 1733916137.
[2024-10-16 10:21:05.900085 +08:00] INFO [src/rpc/synchronizer.rs:218] Update IpGroups version 0 to 1728903659.
[2024-10-16 10:21:05.900117 +08:00] INFO [src/rpc/synchronizer.rs:228] Update FlowAcls version 0 to 2728903859.
[2024-10-16 10:21:05.900126 +08:00] INFO [src/rpc/synchronizer.rs:811] Grpc version ip-groups: 1728903659, interfaces, peer-connections and cidrs: 1733916137, flow-acls: 2728903859
[2024-10-16 10:21:05.900135 +08:00] INFO [src/rpc/synchronizer.rs:828] Grpc finish update cost 9.335µs on 0 listener, 1 ip-groups, 1128 interfaces, 0 peer-connections, 3 cidrs, 0 flow-acls
[2024-10-16 10:21:05.900165 +08:00] INFO [src/rpc/synchronizer.rs:1531] ProxyController update to Some("10.4.45.190"):20035
[2024-10-16 10:21:05.903276 +08:00] INFO [src/monitor.rs:470] monitor started
[2024-10-16 10:21:05.903474 +08:00] INFO [src/platform/libvirt_xml_extractor.rs:104] libvirt_xml_extractor started
[2024-10-16 10:21:05.903744 +08:00] INFO [src/platform/synchronizer.rs:201] PlatformSynchronizer started
[2024-10-16 10:21:05.904242 +08:00] INFO [src/config/handler.rs:2052] CPU Affinity set to .
[2024-10-16 10:21:05.905850 +08:00] INFO [src/config/handler.rs:2327] dispatcher config change from DispatcherConfig {
    global_pps_threshold: 200000,
    capture_packet_size: 65535,
    l7_log_packet_size: 1024,
    tunnel_type_bitmap: TunnelTypeBitmap(
        0,
    ),
    tunnel_type_trim_bitmap: TunnelTypeBitmap(
        0,
    ),
    trident_type: TtUnknown,
    vtap_id: 0,
    capture_socket_type: Auto,
    extra_netns_regex: "",
    tap_interface_regex: "^(tap.*|cali.*|veth.*|eth.*|en[osipx].*|lxc.*|lo|[0-9a-f]+_h)$",
    if_mac_source: IfMac,
    analyzer_ip: "0.0.0.0",
    analyzer_port: 30033,
    proxy_controller_ip: "10.0.34.139",
    proxy_controller_port: 30035,
    capture_bpf: "",
    max_memory: 805306368,
    af_packet_blocks: 48,
    af_packet_version: TpacketVersionHighestavailablet,
    tap_mode: Local,
    region_id: 0,
    pod_cluster_id: 0,
    enabled: true,
    npb_dedup_enabled: true,
    dpdk_enabled: false,
    dispatcher_queue: false,
    bond_group: [],
    cpu_set: CpuSet {
        cpu_set: cpu_set_t {
            bits: [
                0,
                0,
                0,
                0,
                0,
                0,
                0,
                0,
                0,
                0,
                0,
                0,
                0,
                0,
                0,
                0,
            ],
        },
    },
} to DispatcherConfig {
    global_pps_threshold: 200000,
    capture_packet_size: 65535,
    l7_log_packet_size: 1024,
    tunnel_type_bitmap: TunnelTypeBitmap(
        6,
    ),
    tunnel_type_trim_bitmap: TunnelTypeBitmap(
        0,
    ),
    trident_type: TtVmPod,
    vtap_id: 3,
    capture_socket_type: Auto,
    extra_netns_regex: "",
    tap_interface_regex: "^(tap.*|gke.*|veth.*|eth.*|en[osipx].*|lxc.*|lo|[0-9a-f]+_h)$",
    if_mac_source: IfMac,
    analyzer_ip: "10.4.45.190",
    analyzer_port: 20033,
    proxy_controller_ip: "10.4.45.190",
    proxy_controller_port: 20035,
    capture_bpf: "",
    max_memory: 805306368,
    af_packet_blocks: 48,
    af_packet_version: TpacketVersionHighestavailablet,
    tap_mode: Local,
    region_id: 1,
    pod_cluster_id: 1,
    enabled: true,
    npb_dedup_enabled: true,
    dpdk_enabled: false,
    dispatcher_queue: false,
    bond_group: [],
    cpu_set: CpuSet {
        cpu_set: cpu_set_t {
            bits: [
                255,
                0,
                0,
                0,
                0,
                0,
                0,
                0,
                0,
                0,
                0,
                0,
                0,
                0,
                0,
                0,
            ],
        },
    },
}
[2024-10-16 10:21:05.906005 +08:00] INFO [src/config/handler.rs:2340] Enable rsyslog
[2024-10-16 10:21:05.906023 +08:00] INFO [src/config/handler.rs:2360] remote log hostname  -> gke-e2-highmem-8-4e116e47-pgj2-V25
[2024-10-16 10:21:05.906037 +08:00] INFO [src/config/handler.rs:2404] stats config change from StatsConfig {
    interval: 10s,
    host: "",
    analyzer_ip: "0.0.0.0",
    analyzer_port: 30033,
} to StatsConfig {
    interval: 10s,
    host: "gke-e2-highmem-8-4e116e47-pgj2-V25",
    analyzer_ip: "10.4.45.190",
    analyzer_port: 20033,
}
[2024-10-16 10:21:05.906070 +08:00] INFO [src/config/handler.rs:2419] debug config change from DebugConfig {
    vtap_id: 0,
    enabled: true,
    controller_ips: [
        10.0.34.139,
    ],
    controller_port: 30035,
    listen_port: 0,
    agent_mode: Managed,
} to DebugConfig {
    vtap_id: 3,
    enabled: true,
    controller_ips: [
        10.0.34.139,
    ],
    controller_port: 30035,
    listen_port: 0,
    agent_mode: Managed,
}
[2024-10-16 10:21:05.906111 +08:00] INFO [src/config/handler.rs:2590] flow_generator config change from FlowConfig {
    vtap_id: 0,
    trident_type: TtUnknown,
    cloud_gateway_traffic: false,
    collector_enabled: false,
    l7_log_tap_types: [],
    capacity: 65535,
    hash_slots: 131072,
    packet_delay: 1s,
    flush_interval: 1s,
    flow_timeout: FlowTimeout {
        opening: 5s,
        established: 300s,
        closing: 5s,
        established_rst: 35s,
        exception: 5s,
        closed_fin: 2s,
        single_direction: 5s,
        opening_rst: 1s,
        min: 1s,
        max: 300s,
    },
    ignore_tor_mac: false,
    ignore_l2_end: false,
    l7_metrics_enabled: true,
    app_proto_log_enabled: false,
    l4_performance_enabled: true,
    l7_log_packet_size: 1024,
    l7_protocol_inference_max_fail_count: 1000,
    l7_protocol_inference_ttl: 60,
    packet_sequence_flag: 0,
    packet_sequence_block_size: 256,
    l7_protocol_enabled_bitmap: [
        Custom,
        Http1,
        Http2,
        DNS,
        SofaRPC,
        MySQL,
        Kafka,
        Redis,
        MongoDB,
        PostgreSQL,
        Dubbo,
        FastCGI,
        Brpc,
        Oracle,
        MQTT,
        AMQP,
        NATS,
        Pulsar,
        TLS,
        OpenWire,
        ZMTP,
    ],
    plugins: PluginConfig {
        last_updated: 0,
        digest: 15130871412783076140,
        names: [],
    },
    server_ports: [],
} to FlowConfig {
    vtap_id: 3,
    trident_type: TtVmPod,
    cloud_gateway_traffic: false,
    collector_enabled: true,
    l7_log_tap_types: [
        (
            0,
            true,
        ),
    ],
    capacity: 65535,
    hash_slots: 131072,
    packet_delay: 1s,
    flush_interval: 1s,
    flow_timeout: FlowTimeout {
        opening: 5s,
        established: 300s,
        closing: 5s,
        established_rst: 35s,
        exception: 5s,
        closed_fin: 2s,
        single_direction: 5s,
        opening_rst: 1s,
        min: 1s,
        max: 300s,
    },
    ignore_tor_mac: false,
    ignore_l2_end: false,
    l7_metrics_enabled: true,
    app_proto_log_enabled: true,
    l4_performance_enabled: true,
    l7_log_packet_size: 1024,
    l7_protocol_inference_max_fail_count: 1000,
    l7_protocol_inference_ttl: 60,
    packet_sequence_flag: 0,
    packet_sequence_block_size: 256,
    l7_protocol_enabled_bitmap: [
        MySQL,
    ],
    plugins: PluginConfig {
        last_updated: 0,
        digest: 13646096770106105413,
        names: [],
    },
    server_ports: [],
}
[2024-10-16 10:21:05.906247 +08:00] INFO [src/config/handler.rs:2595] plugins changed, pulling 0 plugins from server
[2024-10-16 10:21:05.906266 +08:00] INFO [src/config/handler.rs:419] 0 wasm and 0 so plugins pulled from server
[2024-10-16 10:21:05.906282 +08:00] INFO [src/config/handler.rs:2611] collector config l4_log_store_tap_types change from [] to [(0, true)]
[2024-10-16 10:21:05.906291 +08:00] INFO [src/config/handler.rs:2686] quadruple generator update collector_enabled to true
[2024-10-16 10:21:05.906298 +08:00] INFO [src/config/handler.rs:2693] collector config change from CollectorConfig {
    enabled: true,
    inactive_server_port_enabled: true,
    inactive_ip_enabled: true,
    vtap_flow_1s_enabled: true,
    l4_log_store_tap_types: [],
    l4_log_ignore_tap_sides: [],
    l4_log_collect_nps_threshold: 10000,
    l7_metrics_enabled: true,
    trident_type: TtUnknown,
    vtap_id: 0,
    cloud_gateway_traffic: false,
    packet_delay: 1s,
} to CollectorConfig {
    enabled: true,
    inactive_server_port_enabled: true,
    inactive_ip_enabled: true,
    vtap_flow_1s_enabled: true,
    l4_log_store_tap_types: [
        (
            0,
            true,
        ),
    ],
    l4_log_ignore_tap_sides: [],
    l4_log_collect_nps_threshold: 10000,
    l7_metrics_enabled: true,
    trident_type: TtVmPod,
    vtap_id: 3,
    cloud_gateway_traffic: false,
    packet_delay: 1s,
}
[2024-10-16 10:21:05.906330 +08:00] INFO [src/config/handler.rs:2726] Kubernetes API enabled set to true
[2024-10-16 10:21:05.906361 +08:00] INFO [src/config/handler.rs:2746] platform config change from PlatformConfig {
    sync_interval: 10s,
    kubernetes_cluster_id: "",
    libvirt_xml_path: "/etc/libvirt/qemu",
    kubernetes_poller_type: Adaptive,
    vtap_id: 0,
    enabled: false,
    trident_type: TtUnknown,
    epc_id: 0,
    kubernetes_api_enabled: false,
    kubernetes_api_list_limit: 1000,
    kubernetes_api_list_interval: 600s,
    kubernetes_resources: [],
    max_memory: 805306368,
    namespace: None,
    thread_threshold: 500,
    tap_mode: Local,
    os_proc_scan_conf: OsProcScanConfig {
        os_proc_root: "/proc",
        os_proc_socket_sync_interval: 10,
        os_proc_socket_min_lifetime: 3,
        os_proc_regex: [
            ProcessName(
                .*,
                Accept,
                "",
            ),
            ProcessName(
                .*,
                Accept,
                "",
            ),
        ],
        os_app_tag_exec_user: "deepflow",
        os_app_tag_exec: [],
        os_proc_sync_enabled: false,
        os_proc_sync_tagged_only: false,
    },
    agent_enabled: true,
    extra_netns_regex: "",
} to PlatformConfig {
    sync_interval: 10s,
    kubernetes_cluster_id: "d-rws6NYfvDR",
    libvirt_xml_path: "/etc/libvirt/qemu/",
    kubernetes_poller_type: Adaptive,
    vtap_id: 3,
    enabled: false,
    trident_type: TtVmPod,
    epc_id: 1,
    kubernetes_api_enabled: true,
    kubernetes_api_list_limit: 1000,
    kubernetes_api_list_interval: 600s,
    kubernetes_resources: [],
    max_memory: 805306368,
    namespace: None,
    thread_threshold: 500,
    tap_mode: Local,
    os_proc_scan_conf: OsProcScanConfig {
        os_proc_root: "/proc",
        os_proc_socket_sync_interval: 10,
        os_proc_socket_min_lifetime: 3,
        os_proc_regex: [
            ProcessName(
                .*,
                Accept,
                "",
            ),
            ProcessName(
                .*,
                Accept,
                "",
            ),
        ],
        os_app_tag_exec_user: "deepflow",
        os_app_tag_exec: [],
        os_proc_sync_enabled: false,
        os_proc_sync_tagged_only: false,
    },
    agent_enabled: true,
    extra_netns_regex: "",
}
[2024-10-16 10:21:05.906480 +08:00] INFO [src/config/handler.rs:2827] sender config change from SenderConfig {
    mtu: 1500,
    dest_ip: "0.0.0.0",
    vtap_id: 0,
    team_id: 0,
    organize_id: 0,
    dest_port: 30033,
    npb_port: 4789,
    vxlan_flags: 255,
    npb_enable_qos_bypass: false,
    npb_vlan: 0,
    npb_vlan_mode: None,
    npb_dedup_enabled: true,
    npb_bps_threshold: 1000000000,
    npb_socket_type: RawUdp,
    collector_socket_type: Tcp,
    standalone_data_file_size: 200,
    standalone_data_file_dir: "/var/log/deepflow-agent",
    server_tx_bandwidth_threshold: 0,
    bandwidth_probe_interval: 10s,
    enabled: false,
} to SenderConfig {
    mtu: 1500,
    dest_ip: "10.4.45.190",
    vtap_id: 3,
    team_id: 1,
    organize_id: 1,
    dest_port: 20033,
    npb_port: 4789,
    vxlan_flags: 255,
    npb_enable_qos_bypass: false,
    npb_vlan: 0,
    npb_vlan_mode: None,
    npb_dedup_enabled: true,
    npb_bps_threshold: 1000000000,
    npb_socket_type: RawUdp,
    collector_socket_type: Tcp,
    standalone_data_file_size: 200,
    standalone_data_file_dir: "/var/log/deepflow-agent",
    server_tx_bandwidth_threshold: 0,
    bandwidth_probe_interval: 10s,
    enabled: true,
}
[2024-10-16 10:21:05.906510 +08:00] INFO [src/config/handler.rs:2840] handler config change from HandlerConfig {
    npb_dedup_enabled: true,
    trident_type: TtUnknown,
} to HandlerConfig {
    npb_dedup_enabled: true,
    trident_type: TtVmPod,
}
[2024-10-16 10:21:05.906529 +08:00] INFO [src/config/handler.rs:2848] log_parser config change from LogParserConfig {
    l7_log_collect_nps_threshold: 10000,
    l7_log_session_aggr_timeout: 120s,
    l7_log_session_slot_capacity: 1024,
    l7_log_dynamic: L7LogDynamicConfig {
        proxy_client: "x-forwarded-for",
        x_request_id: {
            "x-request-id",
        },
        trace_types: [
            TraceParent,
            Sw8,
        ],
        span_types: [
            TraceParent,
            Sw8,
        ],
        trace_set: {
            "traceparent",
            "sw8",
        },
        span_set: {
            "sw8",
            "traceparent",
        },
        expected_headers_set: {
            "host",
            ":path",
            "user-agent",
            ":authority",
            "sw8",
            "content-type",
            ":method",
            "x-request-id",
            "x-forwarded-for",
            "traceparent",
            "content-length",
            ":status",
            "referer",
            "grpc-status",
        },
        extra_log_fields: ExtraLogFields {
            http: [],
            http2: [],
        },
    },
    l7_log_ignore_tap_sides: [],
    l7_log_blacklist_trie: {},
    unconcerned_dns_nxdomain_trie: [],
} to LogParserConfig {
    l7_log_collect_nps_threshold: 10000,
    l7_log_session_aggr_timeout: 120s,
    l7_log_session_slot_capacity: 1024,
    l7_log_dynamic: L7LogDynamicConfig {
        proxy_client: "x-forwarded-for",
        x_request_id: {
            "x-request-id",
        },
        trace_types: [
            TraceParent,
            Sw8,
        ],
        span_types: [
            TraceParent,
            Sw8,
        ],
        trace_set: {
            "sw8",
            "traceparent",
        },
        span_set: {
            "sw8",
            "traceparent",
        },
        expected_headers_set: {
            "content-length",
            "referer",
            "content-type",
            ":status",
            "user-agent",
            ":authority",
            ":method",
            "host",
            "x-request-id",
            ":path",
            "x-forwarded-for",
            "traceparent",
            "grpc-status",
            "sw8",
        },
        extra_log_fields: ExtraLogFields {
            http: [],
            http2: [],
        },
    },
    l7_log_ignore_tap_sides: [],
    l7_log_blacklist_trie: {},
    unconcerned_dns_nxdomain_trie: [],
}
[2024-10-16 10:21:05.906641 +08:00] INFO [src/config/handler.rs:2913] ebpf config change from EbpfConfig {
    collector_enabled: false,
    l7_metrics_enabled: true,
    vtap_id: 0,
    epc_id: 0,
    l7_log_packet_size: 1024,
    l7_log_session_timeout: 120s,
    l7_protocol_inference_max_fail_count: 1000,
    l7_protocol_inference_ttl: 60,
    l7_log_tap_types: [],
    ctrl_mac: 00:00:00:00:00:00,
    l7_protocol_enabled_bitmap: [
        Custom,
        Http1,
        Http2,
        DNS,
        SofaRPC,
        MySQL,
        Kafka,
        Redis,
        MongoDB,
        PostgreSQL,
        Dubbo,
        FastCGI,
        Brpc,
        Oracle,
        MQTT,
        AMQP,
        NATS,
        Pulsar,
        TLS,
        OpenWire,
        ZMTP,
    ],
    queue_size: 65535,
    l7_protocol_ports: {
        "Oracle": "1521",
        "TLS": "443,6443",
        "DNS": "53,5353",
    },
    ebpf: EbpfYamlConfig {
        disabled: false,
        log_file: "",
        global_ebpf_pps_threshold: 0,
        kprobe_whitelist: EbpfKprobePortlist {
            port_list: "",
        },
        kprobe_blacklist: EbpfKprobePortlist {
            port_list: "",
        },
        uprobe_proc_regexp: UprobeProcRegExp {
            golang_symbol: "",
            golang: "",
            openssl: "",
        },
        thread_num: 1,
        perf_pages_count: 128,
        ring_size: 65536,
        max_socket_entries: 131072,
        max_trace_entries: 131072,
        socket_map_max_reclaim: 120000,
        go_tracing_timeout: 120,
        io_event_collect_mode: 1,
        io_event_minimal_duration: 1ms,
        java_symbol_file_refresh_defer_interval: 60s,
        on_cpu_profile: OnCpuProfile {
            disabled: false,
            frequency: 99,
            cpu: 0,
            regex: "^deepflow-.*",
        },
        off_cpu_profile: OffCpuProfile {
            disabled: false,
            regex: "^deepflow-.*",
            cpu: 0,
            min_block: 50µs,
        },
        syscall_out_of_order_cache_size: 16,
        syscall_out_of_order_reassembly: [],
        syscall_segmentation_reassembly: [],
    },
} to EbpfConfig {
    collector_enabled: true,
    l7_metrics_enabled: true,
    vtap_id: 3,
    epc_id: 1,
    l7_log_packet_size: 1024,
    l7_log_session_timeout: 120s,
    l7_protocol_inference_max_fail_count: 1000,
    l7_protocol_inference_ttl: 60,
    l7_log_tap_types: [
        (
            0,
            true,
        ),
    ],
    ctrl_mac: 00:00:00:00:00:00,
    l7_protocol_enabled_bitmap: [
        MySQL,
    ],
    queue_size: 65535,
    l7_protocol_ports: {
        "TLS": "443,6443",
        "DNS": "53,5353",
        "Oracle": "1521",
    },
    ebpf: EbpfYamlConfig {
        disabled: false,
        log_file: "",
        global_ebpf_pps_threshold: 0,
        kprobe_whitelist: EbpfKprobePortlist {
            port_list: "",
        },
        kprobe_blacklist: EbpfKprobePortlist {
            port_list: "",
        },
        uprobe_proc_regexp: UprobeProcRegExp {
            golang_symbol: "",
            golang: "",
            openssl: "",
        },
        thread_num: 1,
        perf_pages_count: 128,
        ring_size: 65536,
        max_socket_entries: 131072,
        max_trace_entries: 131072,
        socket_map_max_reclaim: 120000,
        go_tracing_timeout: 120,
        io_event_collect_mode: 1,
        io_event_minimal_duration: 1ms,
        java_symbol_file_refresh_defer_interval: 60s,
        on_cpu_profile: OnCpuProfile {
            disabled: false,
            frequency: 99,
            cpu: 0,
            regex: "^deepflow-.*",
        },
        off_cpu_profile: OffCpuProfile {
            disabled: false,
            regex: "^deepflow-.*",
            cpu: 0,
            min_block: 50µs,
        },
        syscall_out_of_order_cache_size: 16,
        syscall_out_of_order_reassembly: [],
        syscall_segmentation_reassembly: [],
    },
}
[2024-10-16 10:21:05.906802 +08:00] INFO [src/config/handler.rs:2929] trident_type change from TtUnknown to TtVmPod
[2024-10-16 10:21:05.907212 +08:00] INFO [src/sender/uniform_sender.rs:275] stats uniform sender id: 1 started
[2024-10-16 10:21:05.907317 +08:00] INFO [src/platform/kubernetes/api_watcher.rs:756] kubernetes api watcher starting
[2024-10-16 10:21:05.907476 +08:00] INFO [src/trident.rs:1749] Start check process...
[2024-10-16 10:21:05.907623 +08:00] INFO [src/platform/kubernetes/api_watcher.rs:510] api server url is: https://10.0.32.1/
[2024-10-16 10:21:05.922234 +08:00] INFO [src/trident.rs:1753] Start check core file...
[2024-10-16 10:21:05.922416 +08:00] INFO [src/utils/environment/linux.rs:135] Check core-files in dir: /core
[2024-10-16 10:21:05.922485 +08:00] WARN [src/utils/environment/linux.rs:139] Core file read dir /core error: No such file or directory (os error 2).
[2024-10-16 10:21:05.922529 +08:00] INFO [src/trident.rs:1756] Start check controller ip...
[2024-10-16 10:21:05.922555 +08:00] INFO [src/trident.rs:1758] Start check free space...
[2024-10-16 10:21:05.924380 +08:00] INFO [src/platform/kubernetes/api_watcher.rs:484] will query resource namespaces from core/v1
[2024-10-16 10:21:05.924461 +08:00] INFO [src/platform/kubernetes/api_watcher.rs:484] will query resource nodes from core/v1
[2024-10-16 10:21:05.924484 +08:00] INFO [src/platform/kubernetes/api_watcher.rs:484] will query resource pods from core/v1
[2024-10-16 10:21:05.924517 +08:00] INFO [src/platform/kubernetes/api_watcher.rs:484] will query resource replicationcontrollers from core/v1
[2024-10-16 10:21:05.924541 +08:00] INFO [src/platform/kubernetes/api_watcher.rs:484] will query resource services from core/v1
[2024-10-16 10:21:05.924561 +08:00] INFO [src/platform/kubernetes/api_watcher.rs:484] will query resource daemonsets from apps/v1
[2024-10-16 10:21:05.924587 +08:00] INFO [src/platform/kubernetes/api_watcher.rs:484] will query resource deployments from apps/v1
[2024-10-16 10:21:05.924607 +08:00] INFO [src/platform/kubernetes/api_watcher.rs:484] will query resource replicasets from apps/v1
[2024-10-16 10:21:05.924623 +08:00] INFO [src/platform/kubernetes/api_watcher.rs:484] will query resource statefulsets from apps/v1
[2024-10-16 10:21:05.924649 +08:00] INFO [src/platform/kubernetes/api_watcher.rs:484] will query resource ingresses from networking.k8s.io/v1
[2024-10-16 10:21:05.924771 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:512] networking.k8s.io/v1/ingresses watcher started
[2024-10-16 10:21:05.924809 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:512] core/v1/namespaces watcher started
[2024-10-16 10:21:05.924820 +08:00] INFO [src/trident.rs:1854] Agent run with feature-flags: NONE.
[2024-10-16 10:21:05.924857 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:512] apps/v1/daemonsets watcher started
[2024-10-16 10:21:05.924905 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:512] apps/v1/deployments watcher started
[2024-10-16 10:21:05.924954 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:512] core/v1/services watcher started
[2024-10-16 10:21:05.925002 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:512] core/v1/nodes watcher started
[2024-10-16 10:21:05.925063 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:512] core/v1/replicationcontrollers watcher started
[2024-10-16 10:21:05.925212 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:512] core/v1/pods watcher started
[2024-10-16 10:21:05.925390 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:512] apps/v1/replicasets watcher started
[2024-10-16 10:21:05.925621 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:512] apps/v1/statefulsets watcher started
[2024-10-16 10:21:05.925727 +08:00] INFO [src/platform/kubernetes/api_watcher.rs:811] kubernetes api watcher running
[2024-10-16 10:21:05.926382 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:687] list networking.k8s.io/v1/ingresses entries with limit 1000
[2024-10-16 10:21:05.927832 +08:00] INFO [src/platform/synchronizer.rs:235] Platform information changed to version 1729045266
[2024-10-16 10:21:05.927944 +08:00] INFO [src/platform/synchronizer.rs:260] local version is 1729045266, will send whole message
[2024-10-16 10:21:05.928034 +08:00] INFO [src/rpc/session.rs:606] rpc IP changed to proxy 10.4.45.190 20035 from controller 10.0.34.139 30035
[2024-10-16 10:21:05.934329 +08:00] INFO [src/rpc/synchronizer.rs:525] Reset version of acls, groups and platform_data.
[2024-10-16 10:21:05.934513 +08:00] INFO [src/platform/kubernetes/mod.rs:80] kubernetes poller privileges: set_ns=true read_link_ns=true
[2024-10-16 10:21:05.934553 +08:00] INFO [src/platform/kubernetes/mod.rs:90] platform monitoring no extra netns
[2024-10-16 10:21:05.934540 +08:00] INFO [src/sender/uniform_sender.rs:387] remote_logger sender update dst from 0.0.0.0:30033 to 10.4.45.190:20033
[2024-10-16 10:21:05.934574 +08:00] INFO [src/platform/synchronizer.rs:120] updating kubernetes poller
[2024-10-16 10:21:05.934666 +08:00] INFO [src/sender/uniform_sender.rs:184] remote_logger id 0 update agent id from 0 to 3, team id from 0 to 1, organization id from 0 to 1.
[2024-10-16 10:21:05.934816 +08:00] INFO [src/trident.rs:1950] static analyzer ip: '' actual analyzer ip '10.4.45.190'
[2024-10-16 10:21:05.935129 +08:00] INFO [src/sender/uniform_sender.rs:448] remote_logger sender tcp connection to 10.4.45.190:20033 succeed.
[2024-10-16 10:21:05.938851 +08:00] INFO [src/dispatcher/mod.rs:1281] Afpacket init with Options { frame_size: 65536, block_size: 1048576, num_blocks: 48, add_vlan_header: false, block_timeout: 64000000, poll_timeout: 100000000, version: TpacketVersionHighestavailablet, socket_type: SocketTypeRaw, iface: "", packet_fanout_mode: Some(0) }
[2024-10-16 10:21:05.983546 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:764] list networking.k8s.io/v1/ingresses returned 83 entries in 21576B
[2024-10-16 10:21:05.983766 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:639] networking.k8s.io/v1/ingresses watcher initial list ready
[2024-10-16 10:21:05.983874 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:643] networking.k8s.io/v1/ingresses watcher start watching
[2024-10-16 10:21:05.985480 +08:00] INFO [src/dispatcher/recv_engine/af_packet/tpacket.rs:220] kernel version is lower than 3.1, skip the packet fanout setting
[2024-10-16 10:21:05.987331 +08:00] INFO [src/dispatcher/base_dispatcher.rs:663] Decap tunnel type change to VXLAN IPIP
[2024-10-16 10:21:05.987548 +08:00] INFO [src/dispatcher/base_dispatcher.rs:731] Npb dedup change to true
[2024-10-16 10:21:05.987637 +08:00] INFO [src/dispatcher/base_dispatcher.rs:805] Dispatcher(0) Adding VMs: [00:00:00:00:00:00, 42:01:0a:00:0f:cb, ae:3f:30:be:da:f5, f2:9e:62:5e:9b:63, 1e:2f:6a:f5:76:67, a6:cb:b1:10:7e:e4, 3a:d5:d7:27:ab:e1, ba:16:e4:34:72:6f, d2:60:75:3a:9e:28, 76:29:a7:79:96:8b, e2:b0:c7:e7:1a:e0, 52:cb:4a:5e:24:ab, 7e:fd:00:a0:a1:a9, 46:6d:80:99:8a:53, ba:44:14:1b:a8:7e, 22:03:36:05:de:75, 4e:f2:3b:36:fe:9e, 46:8c:d3:8c:7e:f3, 02:36:83:d8:fb:88, 16:ca:5c:6d:4c:13, ae:46:c6:2a:4d:18, 3a:5c:da:da:c1:25, 3e:76:8f:23:60:a8, ea:5c:d2:b3:43:4c, 26:02:3e:00:bc:08, 8a:9f:97:ac:85:64, 5e:6f:d4:97:cd:8b, 52:ea:f6:22:33:c1, 12:05:11:cd:66:e7, ca:8f:c1:b1:3e:63, fe:e3:08:f4:de:e8, 8a:0a:c9:10:aa:83, fa:38:51:f1:1f:40, 8a:bc:3b:1d:39:e6] by [1, 2, 105, 125, 126, 127, 138, 155, 159, 320, 324, 325, 331, 371, 372, 374, 392, 393, 504, 540, 594, 651, 652, 719, 723, 724, 793, 837, 839, 840, 907, 916, 921, 932] + [00:00:00:00:00:00, 42:01:0a:00:0f:cb, ae:3f:30:be:da:f5, f2:9e:62:5e:9b:63, 1e:2f:6a:f5:76:67, a6:cb:b1:10:7e:e4, 3a:d5:d7:27:ab:e1, ba:16:e4:34:72:6f, d2:60:75:3a:9e:28, 76:29:a7:79:96:8b, e2:b0:c7:e7:1a:e0, 52:cb:4a:5e:24:ab, 7e:fd:00:a0:a1:a9, 46:6d:80:99:8a:53, ba:44:14:1b:a8:7e, 22:03:36:05:de:75, 4e:f2:3b:36:fe:9e, 46:8c:d3:8c:7e:f3, 02:36:83:d8:fb:88, 16:ca:5c:6d:4c:13, ae:46:c6:2a:4d:18, 3a:5c:da:da:c1:25, 3e:76:8f:23:60:a8, ea:5c:d2:b3:43:4c, 26:02:3e:00:bc:08, 8a:9f:97:ac:85:64, 5e:6f:d4:97:cd:8b, 52:ea:f6:22:33:c1, 12:05:11:cd:66:e7, ca:8f:c1:b1:3e:63, fe:e3:08:f4:de:e8, 8a:0a:c9:10:aa:83, fa:38:51:f1:1f:40, 8a:bc:3b:1d:39:e6]
[2024-10-16 10:21:05.987804 +08:00] INFO [src/rpc/synchronizer.rs:525] Reset version of acls, groups and platform_data.
[2024-10-16 10:21:05.988298 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:850] ebpf collector init...
[2024-10-16 10:21:05.988387 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:551] ebpf golang uprobe proc regexp is empty, skip set
[2024-10-16 10:21:05.988416 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:567] ebpf openssl uprobe proc regexp is empty, skip set
[2024-10-16 10:21:05.988437 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:590] ebpf golang symbol proc regexp is empty, skip set
[2024-10-16 10:21:05.988470 +08:00] INFO [src/ebpf_dispatcher/ebpf_dispatcher.rs:595] l7 protocol MySQL parse enabled
[2024-10-16 10:21:05.988584 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] WARN func fs_read() [user/common.c:410] Open debug file("/proc/sys/net/core/bpf_jit_enable") open failed.
[2024-10-16 10:21:05.988615 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] WARN func bpf_tracer_init() [user/tracer.c:1753] "/proc/sys/net/core/bpf_jit_enable value is invalid
[2024-10-16 10:21:05.990081 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO log2_default_hugepage_sz 21
[2024-10-16 10:21:05.990241 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] WARN func fetch_kernel_version() [user/common.c:651] release 6.1.58+ version #1 SMP PREEMPT_DYNAMIC Mon Jan 29 15:19:25 UTC 2024 (major 6 minor 1 rev 58 num -2147483648)
[2024-10-16 10:21:05.990279 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] WARN func fetch_kernel_version_code() [user/common.c:702] fetch_kernel_version error
[2024-10-16 10:21:05.990305 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO linux version : 6.1.58+ (version code : 0)
[2024-10-16 10:21:05.990334 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO RLIMIT_NOFILE cur:1048576, rlim_max:1048576
[2024-10-16 10:21:05.990419 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO sys_boot_time_ns : 1721282418480337657
[2024-10-16 10:21:05.990698 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO open(/sys/kernel/debug/tracing/kprobe_events): failed Read-only file system
[2024-10-16 10:21:05.990762 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO open(/sys/kernel/debug/tracing/uprobe_events): failed Read-only file system
[2024-10-16 10:21:05.990897 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO [deepflow-ebpfctl] sockopt register succeed, type get: 404 - 404 set: 400 - 403
[2024-10-16 10:21:05.991050 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO register_period_event_op 'trigger_kern_adapt' succeed.
[2024-10-16 10:21:05.991081 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO register_period_event_op 'boot time update' succeed.
[2024-10-16 10:21:05.991184 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO Tracer 'socket-trace', Not Found.
[2024-10-16 10:21:05.991212 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO Tracer 'socket-trace', Not Found.
[2024-10-16 10:21:05.991228 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO Tracer 'socket-trace', Not Found.
[2024-10-16 10:21:05.991541 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO Set ports_bitmap[TLS] success, ports 443,6443
[2024-10-16 10:21:05.991853 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO Set ports_bitmap[DNS] success, ports 53,5353
[2024-10-16 10:21:05.992049 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO Set ports_bitmap[Oracle] success, ports 1521
[2024-10-16 10:21:05.992291 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO Set ports_bitmap[ZMTP] success, ports 1-65535
[2024-10-16 10:21:05.992559 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO Set ports_bitmap[HTTP2] success, ports 1-65535
[2024-10-16 10:21:05.992813 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO Set ports_bitmap[Custom] success, ports 1-65535
[2024-10-16 10:21:05.993058 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO Set ports_bitmap[MySQL] success, ports 1-65535
[2024-10-16 10:21:05.993307 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO Set ports_bitmap[FastCGI] success, ports 1-65535
[2024-10-16 10:21:05.993597 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO Set ports_bitmap[HTTP1] success, ports 1-65535
[2024-10-16 10:21:05.993821 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO Set ports_bitmap[Redis] success, ports 1-65535
[2024-10-16 10:21:05.994038 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO Set ports_bitmap[SofaRPC] success, ports 1-65535
[2024-10-16 10:21:05.994282 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO Set ports_bitmap[MQTT] success, ports 1-65535
[2024-10-16 10:21:05.994509 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO Set ports_bitmap[AMQP] success, ports 1-65535
[2024-10-16 10:21:05.994734 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO Set ports_bitmap[Pulsar] success, ports 1-65535
[2024-10-16 10:21:05.994946 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO Set ports_bitmap[MongoDB] success, ports 1-65535
[2024-10-16 10:21:05.995160 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO Set ports_bitmap[PgSQL] success, ports 1-65535
[2024-10-16 10:21:05.995384 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO Set ports_bitmap[NATS] success, ports 1-65535
[2024-10-16 10:21:05.995687 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO Set ports_bitmap[Dubbo] success, ports 1-65535
[2024-10-16 10:21:05.995884 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO Set ports_bitmap[Kafka] success, ports 1-65535
[2024-10-16 10:21:05.996030 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO Set ports_bitmap[OpenWire] success, ports 1-65535
[2024-10-16 10:21:05.996236 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] INFO Set ports_bitmap[bRPC] success, ports 1-65535
[2024-10-16 10:21:05.996298 +08:00] INFO [src/ebpf/mod.rs:698] [eBPF] WARN func fetch_kernel_version() [user/common.c:651] release 6.1.58+ version #1 SMP PREEMPT_DYNAMIC Mon Jan 29 15:19:25 UTC 2024 (major 6 minor 1 rev 58 num 0)
[2024-10-16 10:21:05.996370 +08:00] ERROR [src/trident.rs:2345] ebpf collector error: EbpfRunningError
[2024-10-16 10:21:05.997383 +08:00] INFO [src/rpc/synchronizer.rs:525] Reset version of acls, groups and platform_data.
[2024-10-16 10:21:05.997482 +08:00] INFO [src/trident.rs:2580] Staring agent components.
[2024-10-16 10:21:05.997624 +08:00] INFO [src/platform/platform_synchronizer/linux.rs:148] socket info sync start
[2024-10-16 10:21:05.997674 +08:00] INFO [src/platform/kubernetes/active_poller.rs:153] starts kubernetes active poller
[2024-10-16 10:21:05.997877 +08:00] INFO [src/debug/debugger.rs:441] debugger started
[2024-10-16 10:21:05.998114 +08:00] INFO [src/sender/uniform_sender.rs:275] 3-doc-to-collector-sender uniform sender id: 3 started
[2024-10-16 10:21:05.998120 +08:00] INFO [src/debug/debugger.rs:129] debugger listening on: [::]:34249
[2024-10-16 10:21:05.998267 +08:00] INFO [src/sender/uniform_sender.rs:275] 2-protolog-to-collector-sender uniform sender id: 4 started
[2024-10-16 10:21:05.998398 +08:00] INFO [src/sender/uniform_sender.rs:275] 3-flowlog-to-collector-sender uniform sender id: 2 started
[2024-10-16 10:21:05.998823 +08:00] INFO [src/sender/uniform_sender.rs:275] 2-packet-sequence-block-to-sender uniform sender id: 6 started
[2024-10-16 10:21:05.999136 +08:00] INFO [src/dispatcher/local_mode_dispatcher.rs:68] Start dispatcher (0)
[2024-10-16 10:21:05.999154 +08:00] INFO [src/flow_generator/protocol_logs/parser.rs:883] app protocol logs parser (id=0) started
[2024-10-16 10:21:05.999237 +08:00] INFO [src/collector/quadruple_generator.rs:739] new quadruple_generator id: 0, second_delay: 9, minute_delay: 69, l7_metrics_enabled: true, vtap_flow_1s_enabled: true collector_enabled: true
[2024-10-16 10:21:06.003025 +08:00] INFO [src/collector/quadruple_generator.rs:663] quadruple generator id: 0 started
[2024-10-16 10:21:06.004422 +08:00] INFO [src/collector/flow_aggr.rs:125] l4 flow aggr id: 0 started
[2024-10-16 10:21:06.004758 +08:00] INFO [src/collector/collector.rs:1295] second_collector id=(0) started
[2024-10-16 10:21:06.005903 +08:00] INFO [src/collector/collector.rs:1295] minute_collector id=(0) started
[2024-10-16 10:21:06.005990 +08:00] INFO [src/collector/l7_quadruple_generator.rs:531] new l7 quadruple_generator id: 0, second_delay: 9, minute_delay: 69, l7_metrics_enabled: true, vtap_flow_1s_enabled: true collector_enabled: true
[2024-10-16 10:21:06.007133 +08:00] INFO [src/collector/l7_quadruple_generator.rs:473] l7 quadruple generator id: 0 started
[2024-10-16 10:21:06.007238 +08:00] INFO [src/collector/collector.rs:1436] second_collector id=(0) started
[2024-10-16 10:21:06.007559 +08:00] INFO [src/collector/collector.rs:1436] minute_collector id=(0) started
[2024-10-16 10:21:06.008017 +08:00] INFO [src/flow_generator/packet_sequence/parser.rs:88] packet sequence parser (id=0) started
[2024-10-16 10:21:06.008331 +08:00] INFO [src/handler/npb.rs:338] Start npb packet sender 0.
[2024-10-16 10:21:06.008903 +08:00] INFO [src/sender/uniform_sender.rs:275] 1-otel-to-sender uniform sender id: 10 started
[2024-10-16 10:21:06.009548 +08:00] INFO [src/sender/uniform_sender.rs:275] 1-compressed-otel-to-sender uniform sender id: 13 started
[2024-10-16 10:21:06.010272 +08:00] INFO [src/sender/uniform_sender.rs:275] 1-prometheus-to-sender uniform sender id: 11 started
[2024-10-16 10:21:06.010436 +08:00] INFO [src/sender/uniform_sender.rs:275] 1-telegraf-to-sender uniform sender id: 12 started
[2024-10-16 10:21:06.010701 +08:00] INFO [src/sender/uniform_sender.rs:275] 1-profile-to-sender uniform sender id: 8 started
[2024-10-16 10:21:06.010896 +08:00] INFO [src/sender/uniform_sender.rs:275] 1-proc-event-to-sender uniform sender id: 7 started
[2024-10-16 10:21:06.011070 +08:00] INFO [src/sender/uniform_sender.rs:275] 1-application-log-to-sender uniform sender id: 9 started
[2024-10-16 10:21:06.011245 +08:00] INFO [src/integration_collector.rs:1032] integration collector starting
[2024-10-16 10:21:06.011624 +08:00] INFO [src/integration_collector.rs:1138] integration collector started
[2024-10-16 10:21:06.011664 +08:00] INFO [src/integration_collector.rs:1139] integration collector listening on http://[::]:38086
[2024-10-16 10:21:06.011742 +08:00] INFO [src/collector/l7_quadruple_generator.rs:531] new l7 quadruple_generator id: 2, second_delay: 9, minute_delay: 69, l7_metrics_enabled: true, vtap_flow_1s_enabled: true collector_enabled: true
[2024-10-16 10:21:06.015138 +08:00] INFO [src/collector/l7_quadruple_generator.rs:473] l7 quadruple generator id: 2 started
[2024-10-16 10:21:06.015510 +08:00] INFO [src/collector/collector.rs:1436] second_collector id=(2) started
[2024-10-16 10:21:06.015842 +08:00] INFO [src/collector/collector.rs:1436] minute_collector id=(2) started
[2024-10-16 10:21:06.016166 +08:00] INFO [src/sender/uniform_sender.rs:275] 2-pcap-batch-to-sender uniform sender id: 5 started
[2024-10-16 10:21:06.016307 +08:00] INFO [src/trident.rs:2639] Started agent components.
[2024-10-16 10:21:06.016449 +08:00] INFO [src/utils/stats.rs:336] set stats hostname to "gke-e2-highmem-8-4e116e47-pgj2-V25"
[2024-10-16 10:21:06.016852 +08:00] INFO [src/platform/kubernetes/active_poller.rs:86] kubernetes poller updated to version (1)
[2024-10-16 10:21:06.027505 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:687] list core/v1/namespaces entries with limit 1000
[2024-10-16 10:21:06.041004 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:764] list core/v1/namespaces returned 52 entries in 5796B
[2024-10-16 10:21:06.041127 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:639] core/v1/namespaces watcher initial list ready
[2024-10-16 10:21:06.041297 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:643] core/v1/namespaces watcher start watching
[2024-10-16 10:21:06.130107 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:687] list apps/v1/statefulsets entries with limit 1000
[2024-10-16 10:21:06.431278 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:764] list apps/v1/statefulsets returned 169 entries in 262571B
[2024-10-16 10:21:06.431449 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:639] apps/v1/statefulsets watcher initial list ready
[2024-10-16 10:21:06.431486 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:643] apps/v1/statefulsets watcher start watching
[2024-10-16 10:21:06.433834 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:687] list core/v1/services entries with limit 1000
[2024-10-16 10:21:06.535459 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:764] list core/v1/services returned 460 entries in 235988B
[2024-10-16 10:21:06.535560 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:639] core/v1/services watcher initial list ready
[2024-10-16 10:21:06.535574 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:643] core/v1/services watcher start watching
[2024-10-16 10:21:06.536831 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:687] list core/v1/nodes entries with limit 1000
[2024-10-16 10:21:06.588706 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:764] list core/v1/nodes returned 40 entries in 44780B
[2024-10-16 10:21:06.588820 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:639] core/v1/nodes watcher initial list ready
[2024-10-16 10:21:06.588845 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:643] core/v1/nodes watcher start watching
[2024-10-16 10:21:06.635595 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:687] list apps/v1/deployments entries with limit 1000
[2024-10-16 10:21:06.852253 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:764] list apps/v1/deployments returned 283 entries in 386071B
[2024-10-16 10:21:06.852380 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:639] apps/v1/deployments watcher initial list ready
[2024-10-16 10:21:06.852406 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:643] apps/v1/deployments watcher start watching
[2024-10-16 10:21:06.941040 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:687] list core/v1/replicationcontrollers entries with limit 1000
[2024-10-16 10:21:06.948105 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:764] list core/v1/replicationcontrollers returned 0 entries in 0B
[2024-10-16 10:21:06.948221 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:639] core/v1/replicationcontrollers watcher initial list ready
[2024-10-16 10:21:06.948241 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:643] core/v1/replicationcontrollers watcher start watching
[2024-10-16 10:21:07.042284 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:687] list apps/v1/daemonsets entries with limit 1000
[2024-10-16 10:21:07.085190 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:764] list apps/v1/daemonsets returned 42 entries in 78117B
[2024-10-16 10:21:07.085301 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:639] apps/v1/daemonsets watcher initial list ready
[2024-10-16 10:21:07.085324 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:643] apps/v1/daemonsets watcher start watching
[2024-10-16 10:21:07.143635 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:687] list core/v1/pods entries with limit 1000
[2024-10-16 10:21:07.908020 +08:00] INFO [src/rpc/synchronizer.rs:907] grpc server changed
[2024-10-16 10:21:08.265141 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:764] list core/v1/pods returned 1191 entries in 1207449B
[2024-10-16 10:21:08.265386 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:639] core/v1/pods watcher initial list ready
[2024-10-16 10:21:08.265480 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:643] core/v1/pods watcher start watching
[2024-10-16 10:21:08.358007 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:687] list apps/v1/replicasets entries with limit 1000
[2024-10-16 10:21:08.907485 +08:00] INFO [src/sender/uniform_sender.rs:184] stats id 1 update agent id from 3 to 3, team id from 0 to 1, organization id from 0 to 1.
[2024-10-16 10:21:08.908129 +08:00] INFO [src/sender/uniform_sender.rs:448] stats sender tcp connection to 10.4.45.190:20033 succeed.
[2024-10-16 10:21:08.998626 +08:00] INFO [src/sender/uniform_sender.rs:184] 2-protolog-to-collector-sender id 4 update agent id from 3 to 3, team id from 0 to 1, organization id from 0 to 1.
[2024-10-16 10:21:08.998876 +08:00] INFO [src/sender/uniform_sender.rs:184] 3-flowlog-to-collector-sender id 2 update agent id from 3 to 3, team id from 0 to 1, organization id from 0 to 1.
[2024-10-16 10:21:09.000113 +08:00] INFO [src/sender/uniform_sender.rs:184] 2-packet-sequence-block-to-sender id 6 update agent id from 3 to 3, team id from 0 to 1, organization id from 0 to 1.
[2024-10-16 10:21:09.000594 +08:00] INFO [src/sender/uniform_sender.rs:184] 3-doc-to-collector-sender id 3 update agent id from 3 to 3, team id from 0 to 1, organization id from 0 to 1.
[2024-10-16 10:21:09.009154 +08:00] INFO [src/sender/uniform_sender.rs:184] 1-otel-to-sender id 10 update agent id from 3 to 3, team id from 0 to 1, organization id from 0 to 1.
[2024-10-16 10:21:09.010177 +08:00] INFO [src/sender/uniform_sender.rs:184] 1-compressed-otel-to-sender id 13 update agent id from 3 to 3, team id from 0 to 1, organization id from 0 to 1.
[2024-10-16 10:21:09.010881 +08:00] INFO [src/sender/uniform_sender.rs:184] 1-profile-to-sender id 8 update agent id from 3 to 3, team id from 0 to 1, organization id from 0 to 1.
[2024-10-16 10:21:09.011286 +08:00] INFO [src/sender/uniform_sender.rs:184] 1-telegraf-to-sender id 12 update agent id from 3 to 3, team id from 0 to 1, organization id from 0 to 1.
[2024-10-16 10:21:09.011485 +08:00] INFO [src/sender/uniform_sender.rs:184] 1-prometheus-to-sender id 11 update agent id from 3 to 3, team id from 0 to 1, organization id from 0 to 1.
[2024-10-16 10:21:09.012333 +08:00] INFO [src/sender/uniform_sender.rs:184] 1-application-log-to-sender id 9 update agent id from 3 to 3, team id from 0 to 1, organization id from 0 to 1.
[2024-10-16 10:21:09.017390 +08:00] INFO [src/sender/uniform_sender.rs:184] 2-pcap-batch-to-sender id 5 update agent id from 3 to 3, team id from 0 to 1, organization id from 0 to 1.
[2024-10-16 10:21:09.074954 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:764] list apps/v1/replicasets returned 1249 entries in 414823B
[2024-10-16 10:21:09.075066 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:639] apps/v1/replicasets watcher initial list ready
[2024-10-16 10:21:09.075082 +08:00] INFO [src/platform/kubernetes/resource_watcher.rs:643] apps/v1/replicasets watcher start watching
[2024-10-16 10:21:10.925890 +08:00] INFO [src/platform/kubernetes/api_watcher.rs:621] version updated to 1729045266 (core/namespaces: v0 -> v1; core/pods: v0 -> v2; apps/replicasets: v0 -> v1; apps/deployments: v0 -> v1; apps/daemonsets: v0 -> v1; apps/statefulsets: v0 -> v1; networking.k8s.io/ingresses: v0 -> v1; core/services: v0 -> v1; core/nodes: v0 -> v3)
[2024-10-16 10:21:15.930771 +08:00] INFO [src/platform/synchronizer.rs:226] updated kubernetes poller
[2024-10-16 10:21:15.972985 +08:00] INFO [src/platform/synchronizer.rs:235] Platform information changed to version 1729045267
[2024-10-16 10:21:15.973113 +08:00] INFO [src/platform/synchronizer.rs:260] local version is 1729045267, will send whole message
[2024-10-16 10:21:20.971162 +08:00] INFO [src/platform/kubernetes/api_watcher.rs:621] version updated to 1729045267 (core/pods: v2 -> v10; core/nodes: v3 -> v5)
[2024-10-16 10:21:30.990752 +08:00] INFO [src/platform/kubernetes/api_watcher.rs:621] version updated to 1729045268 (core/pods: v10 -> v16; core/nodes: v5 -> v6)
[2024-10-16 10:21:35.891108 +08:00] INFO [src/rpc/remote_exec.rs:259] grpc server changed
[2024-10-16 10:21:41.012933 +08:00] INFO [src/platform/kubernetes/api_watcher.rs:621] version updated to 1729045269 (core/pods: v16 -> v18; core/nodes: v6 -> v9)
[2024-10-16 10:21:51.031865 +08:00] INFO [src/platform/kubernetes/api_watcher.rs:621] version updated to 1729045270 (core/pods: v18 -> v27; core/nodes: v9 -> v10)