search

deepflowio / deepflow

eBPF Observability - Distributed Tracing and Profiling
https://deepflow.io
Apache License 2.0
2.95k stars 333 forks source link

[deepflow-agent]agent service exit #1276

Closed ljyfree closed 2 years ago

ljyfree commented 2 years ago

Expected Behavior

agent serive goes well

Actual Behavior

agent service exit

Steps to Reproduce the Problem

  1. KVM node(Ubuntu20.04)
  2. Install deb
  3. check agent serice 
    
root@hpc-1:~/mpich# systemctl status deepflow-agent
● deepflow-agent.service
 Loaded: loaded (/etc/systemd/system/deepflow-agent.service; enabled; vendor preset: enabled)
 Active: activating (auto-restart) (Result: exit-code) since Mon 2022-10-31 11:09:30 +08; 3s ago
Process: 309205 ExecStart=/usr/sbin/deepflow-agent (code=exited, status=1/FAILURE)
Main PID: 309205 (code=exited, status=1/FAILURE)

10-р сар 31 11:09:30 hpc-1 systemd[1]: deepflow-agent.service: Main process exited, code=exited, status=1/FAILURE 10-р сар 31 11:09:30 hpc-1 systemd[1]: deepflow-agent.service: Failed with result 'exit-code'. root@hpc-1:~/mpich# 


# Additional Info

-  deepflow version:

root@hpc-2:~/mpi# deepflow-agent -v 6875-0a58be53c8c81a8c9f27a9bf75b69f564c485985 Name: deepflow-agent community edition Branch: v6.1.5 CommitId: 0a58be53c8c81a8c9f27a9bf75b69f564c485985 RevCount: 6875 Compiler: rustc 1.64.0 (a55dd71d5 2022-09-19) CompileTime: 2022-10-27 03:22:03 root@hpc-2:~/mpi# 


- log

[2022-10-31 10:43:05.963145 +08:00] INFO [src/sender/uniform_sender.rs:219] uniform sender id: 100 started [2022-10-31 10:43:05.976293 +08:00] INFO [src/rpc/synchronizer.rs:466] Reset version of acls, groups and platform_data. [2022-10-31 10:43:05.976351 +08:00] INFO [src/platform/platform_synchronizer.rs:112] kubernetes poller privileges: set_ns=true read_link_ns=true [2022-10-31 10:43:05.976360 +08:00] INFO [src/platform/kubernetes/active_poller.rs:156] poller monitoring netns: [] [2022-10-31 10:43:05.978072 +08:00] INFO [src/trident.rs:936] static analyzer ip: actual analyzer ip 0.0.0.0 [2022-10-31 10:43:05.978392 +08:00] INFO [src/dispatcher/mod.rs:692] Afpacket init with Options { frame_size: 65536, block_size: 1048576, num_blocks: 48, add_vlan_header: false, block_timeout: 64000000, poll_timeout: 100000000, version: TpacketVersionHighestavailablet, socket_type: SocketTypeRaw, iface: "" } [2022-10-31 10:43:06.013719 +08:00] INFO [src/dispatcher/base_dispatcher.rs:641] Decap tunnel type change to VXLAN IPIP [2022-10-31 10:43:06.014040 +08:00] INFO [src/handler/npb.rs:201] Build with npb packet handler with id: 0 if_index: 1 mac: 00:00:00:00:00:00 [2022-10-31 10:43:06.014060 +08:00] INFO [src/handler/npb.rs:201] Build with npb packet handler with id: 0 if_index: 2 mac: 52:54:00:5b:1d:a2 [2022-10-31 10:43:06.014072 +08:00] INFO [src/handler/npb.rs:201] Build with npb packet handler with id: 0 if_index: 15 mac: 52:54:00:46:16:81 [2022-10-31 10:43:06.014094 +08:00] INFO [src/dispatcher/base_dispatcher.rs:751] Adding VMs: [00:00:00:00:00:00, 52:54:00:5b:1d:a2, 52:54:00:46:16:81] [2022-10-31 10:43:06.014165 +08:00] INFO [src/ebpf_collector/ebpf_collector.rs:1052] ebpf collector init... [2022-10-31 10:43:06.014187 +08:00] INFO [src/ebpf_collector/ebpf_collector.rs:929] ebpf set golang uprobe proc regexp: . [2022-10-31 10:43:06.014229 +08:00] INFO [src/ebpf_collector/ebpf_collector.rs:945] ebpf set openssl uprobe proc regexp: . [2022-10-31 10:43:06.014263 +08:00] INFO [src/ebpf_collector/ebpf_collector.rs:973] ebpf golang symbol proc regexp is empty, skip set [2022-10-31 10:43:06.014280 +08:00] INFO [src/ebpf_collector/ebpf_collector.rs:978] l7 protocol Http1 parse enabled [2022-10-31 10:43:06.014288 +08:00] INFO [src/ebpf_collector/ebpf_collector.rs:978] l7 protocol Http2 parse enabled [2022-10-31 10:43:06.014293 +08:00] INFO [src/ebpf_collector/ebpf_collector.rs:978] l7 protocol DNS parse enabled [2022-10-31 10:43:06.014298 +08:00] INFO [src/ebpf_collector/ebpf_collector.rs:978] l7 protocol MySQL parse enabled [2022-10-31 10:43:06.014302 +08:00] INFO [src/ebpf_collector/ebpf_collector.rs:978] l7 protocol Kafka parse enabled [2022-10-31 10:43:06.014307 +08:00] INFO [src/ebpf_collector/ebpf_collector.rs:978] l7 protocol Redis parse enabled [2022-10-31 10:43:06.014312 +08:00] INFO [src/ebpf_collector/ebpf_collector.rs:978] l7 protocol PostgreSQL parse enabled [2022-10-31 10:43:06.014316 +08:00] INFO [src/ebpf_collector/ebpf_collector.rs:978] l7 protocol Dubbo parse enabled [2022-10-31 10:43:06.014321 +08:00] INFO [src/ebpf_collector/ebpf_collector.rs:978] l7 protocol MQTT parse enabled [2022-10-31 10:43:08.544399 +08:00] INFO [src/ebpf_collector/ebpf_collector.rs:1062] ebpf collector initialized. [2022-10-31 10:43:08.544684 +08:00] INFO [src/trident.rs:719] Staring components. [2022-10-31 10:43:08.545051 +08:00] INFO [src/pcap/manager.rs:128] started WorkerManager [2022-10-31 10:43:08.545126 +08:00] INFO [src/platform/platform_synchronizer.rs:232] PlatformSynchronizer started [2022-10-31 10:43:08.545165 +08:00] INFO [src/platform/kubernetes/api_watcher.rs:192] ApiWatcher failed to start because kubernetes-cluster-id is empty [2022-10-31 10:43:08.545240 +08:00] INFO [src/debug/debugger.rs:355] debugger started [2022-10-31 10:43:08.545502 +08:00] INFO [src/sender/uniform_sender.rs:219] uniform sender id: 1 started [2022-10-31 10:43:08.545589 +08:00] INFO [src/sender/uniform_sender.rs:219] uniform sender id: 2 started [2022-10-31 10:43:08.545666 +08:00] INFO [src/sender/uniform_sender.rs:219] uniform sender id: 0 started [2022-10-31 10:43:08.545782 +08:00] INFO [src/sender/uniform_sender.rs:219] uniform sender id: 6 started [2022-10-31 10:43:08.545863 +08:00] INFO [src/flow_generator/packet_sequence/parser.rs:84] packet sequence parser (id=0) started [2022-10-31 10:43:08.546438 +08:00] INFO [src/flow_generator/protocol_logs/parser.rs:538] app protocol logs parser (id=0) started [2022-10-31 10:43:08.546502 +08:00] INFO [src/collector/quadruple_generator.rs:721] new quadruple_generator id: 0, second_delay: 8, minute_delay: 68, l7_metrics_enabled: true, vtap_flow_1s_enabled: true collector_enabled: true [2022-10-31 10:43:08.548024 +08:00] INFO [src/collector/quadruple_generator.rs:661] quadruple generator id: 0 started [2022-10-31 10:43:08.548142 +08:00] INFO [src/collector/flow_aggr.rs:99] l4 flow aggr id: 0 started [2022-10-31 10:43:08.548246 +08:00] INFO [src/collector/collector.rs:925] second_collector id=(0) started [2022-10-31 10:43:08.548350 +08:00] INFO [src/collector/collector.rs:925] minute_collector id=(0) started

[2022-10-31 10:43:08.549607 +08:00] INFO [src/dispatcher/local_mode_dispatcher.rs:62] Start dispatcher 0 [2022-10-31 10:43:08.550724 +08:00] ERROR [src/common/flow.rs:1308] invalid trident type, trident will stop ^C root@hpc-2:~/mpi# root@hpc-2:~/mpi# root@hpc-2:~/mpi# root@hpc-2:~/mpi# root@hpc-2:~/mpi# root@hpc-2:~/mpi# tail -f /var/log/deepflow-agent/deepflow-agent.log [2022-10-31 10:57:52.821750 +08:00] INFO [src/platform/kubernetes/api_watcher.rs:192] ApiWatcher failed to start because kubernetes-cluster-id is empty [2022-10-31 10:57:52.821875 +08:00] INFO [src/debug/debugger.rs:355] debugger started [2022-10-31 10:57:52.821990 +08:00] INFO [src/sender/uniform_sender.rs:219] uniform sender id: 1 started [2022-10-31 10:57:52.822055 +08:00] INFO [src/sender/uniform_sender.rs:219] uniform sender id: 2 started [2022-10-31 10:57:52.822113 +08:00] INFO [src/sender/uniform_sender.rs:219] uniform sender id: 0 started

[2022-10-31 10:57:52.822201 +08:00] INFO [src/sender/uniform_sender.rs:219] uniform sender id: 6 started [2022-10-31 10:57:52.823011 +08:00] INFO [src/flow_generator/packet_sequence/parser.rs:84] packet sequence parser (id=0) started [2022-10-31 10:57:52.823725 +08:00] INFO [src/dispatcher/local_mode_dispatcher.rs:62] Start dispatcher 0 [2022-10-31 10:57:52.824246 +08:00] ERROR [src/common/flow.rs:1308] invalid trident type, trident will stop ^C root@hpc-2:~/mpi# root@hpc-2:~/mpi# root@hpc-2:~/mpi# cat /etc/deepflow-agent.yaml

controller ip

controller-ips:

controller listen port

controller-port: 30035

controller security authenticate port

controller-tls-port: 30135

controller certificate file prefix, contain path

if certificate file exists, do certificate; or no

certificate file naming rule is prefix.controller-ip

example

controller-cert-file-prefix is /etc/trident_controller.cert

controller ip is: 10.10.10.10

so certificate file name is trident_controller.cert.10.10.10.10 in /etc/

controller-cert-file-prefix: ""

logfile path

log-file: /var/log/deepflow-agent/deepflow-agent.log

kubernetes-cluster-id，在对接kubernetes/openshift时需要配置

kubernetes-cluster-id:

支持采集器自动加入组

vtap-group-id-request: "g-1bJYxAEEKI" root@hpc-2:~/mpi# root@hpc-2:~/mpi# ip add | grep 192.168 inet 192.168.122.117/24 brd 192.168.122.255 scope global dynamic noprefixroute ens3 root@hpc-2:~/mpi# root@hpc-2:~/mpi# root@hpc-2:~/mpi# cat /etc/deepflow-agent.yaml

controller ip

controller-ips:

controller listen port

controller-port: 30035

controller security authenticate port

controller-tls-port: 30135

controller certificate file prefix, contain path

if certificate file exists, do certificate; or no

certificate file naming rule is prefix.controller-ip

example

controller-cert-file-prefix is /etc/trident_controller.cert

controller ip is: 10.10.10.10

so certificate file name is trident_controller.cert.10.10.10.10 in /etc/

controller-cert-file-prefix: ""

logfile path

log-file: /var/log/deepflow-agent/deepflow-agent.log

kubernetes-cluster-id，在对接kubernetes/openshift时需要配置

kubernetes-cluster-id:

支持采集器自动加入组

vtap-group-id-request: "g-1bJYxAEEKI" root@hpc-2:~/mpi# root@hpc-2:~/mpi# root@hpc-2:~/mpi# root@hpc-2:~/mpi# root@hpc-2:~/mpi# journalctl -xeu deepflow-agent 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO attach enter kprobe: 'kprobe/do_writev', success! 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO attach enter kprobe: 'kprobe/do_readv', success! 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_enter_write', succeed! 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_enter_read', succeed! 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_enter_sendto', succeed! 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_enter_recvfrom', succeed! 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_socket', succeed! 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_read', succeed! 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_write', succeed! 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_sendto', succeed! 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_recvfrom', succeed! 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_sendmsg', succeed! 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_sendmmsg', succeed! 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_recvmsg', succeed! 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_recvmmsg', succeed! 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_writev', succeed! 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_exit_readv', succeed! 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO attach tracepoint: 'tracepoint/sched/sched_process_fork', succeed! 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_enter_getppid', succeed! 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO attach tracepoint: 'tracepoint/syscalls/sys_enter_close', succeed! 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO attach tracepoint: 'tracepoint/sched/sched_process_exit', succeed! 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO register_extra_waiting_op 'offset-infer-server' succeed. 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO register_extra_waiting_op 'offset-infer-client' succeed. 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO register_period_event_op 'check-map-exceeded' succeed. 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO register_period_event_op 'check-kern-adapt' succeed. 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO sockopt register succeed, type -> 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: get: 504 - 504 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: set: 500 - 503 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO All tracers finish!!! 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO socket_tracer state is TRACER_INIT, not permit start. 10-р сар 31 11:04:37 hpc-2 deepflow-agent[289131]: 2022-10-31 11:4:37 [eBPF] INFO ctrl_main begin !!! 10-р сар 31 11:04:39 hpc-2 systemd[1]: deepflow-agent.service: Main process exited, code=exited, status=1/FAILURE -- Subject: Unit process exited -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- An ExecStart= process belonging to unit deepflow-agent.service has exited. -- -- The process' exit code is 'exited' and its exit status is 1. 10-р сар 31 11:04:39 hpc-2 systemd[1]: deepflow-agent.service: Failed with result 'exit-code'. -- Subject: Unit failed -- Defined-By: systemd -- Support: http://www.ubuntu.com/support -- -- The unit deepflow-agent.service has entered the 'failed' state with result 'exit-code'. root@hpc-2:~/mpi#

ljyfree commented 2 years ago

问题原因是values-custom.yaml缺少配置

configmap:
  server.yaml:
    controller:
      genesis:
        local_ip_ranges:
        - 10.0.0.0/8
        - 172.16.0.0/12
        - 192.168.0.0/16
        - 169.254.0.0/15
        - 224.0.0.0-240.255.255.255
        # - 7.8.0.0/16  # FIXME: your host all network cidr
      trisolaris:
        trident-type-for-unkonw-vtap: 3  # required

添加后agent可以连上

~# deepflow-ctl agent list
NAME     TYPE       CTRL_IP          CTRL_MAC           STATE    EXCEPTIONS GROUP
mypc-V1  K8S_VM     192.168.122.217  52:54:00:66:df:f2  RUNNING             default
hpc-1-W2 CHOST_VM   192.168.122.169  52:54:00:9b:92:aa  RUNNING             legacy-host
hpc-2-W3 CHOST_VM   192.168.122.117  52:54:00:5b:1d:a2  RUNNING             legacy-host

issue close