coroot / coroot-node-agent

A Prometheus exporter based on eBPF that gathers comprehensive container metrics
https://coroot.com/docs/metrics/node-agent
Apache License 2.0
333 stars 61 forks source link

Failed to start in docker agent version: 1.19.8 #95

Open donge opened 6 months ago

donge commented 6 months ago

kernel: Linux localhost 4.19.90-52.22.v2207.ky10.aarch64 #1 SMP Tue Mar 14 11:52:45 CST 2023 aarch64 aarch64 aarch64 GNU/Linux coroot agent version: 1.19.8 docker version: Client: Version: 23.0.6 API version: 1.42 Go version: go1.19.9 Git commit: ef23cbc Built: Fri May 5 21:16:16 2023 OS/Arch: linux/arm64 Context: default

Server: Docker Engine - Community Engine: Version: 23.0.6 API version: 1.42 (minimum version 1.12) Go version: go1.19.9 Git commit: 9dbdbd4 Built: Fri May 5 21:17:31 2023 OS/Arch: linux/arm64 Experimental: false containerd: Version: v1.6.21 GitCommit: 3dce8eb055cbb6872793272b4f20ed16117344f8 runc: Version: 1.1.7 GitCommit: v1.1.7-0-g860f061 docker-init: Version: 0.19.0 GitCommit: de40ad0

failed to start, and logs

I0604 04:14:47.431304 1274943 main.go:111] agent version: 1.19.8
I0604 04:14:47.431370 1274943 main.go:117] hostname: localhost
I0604 04:14:47.431377 1274943 main.go:118] kernel version: 4.19.90-52.22.v2207.ky10.aarch64
I0604 04:14:47.434007 1274943 main.go:75] machine-id:  1d7b4ad8f42243509ec578c69b2b0c9d
I0604 04:14:47.434066 1274943 tracing.go:37] OpenTelemetry traces collector endpoint:
I0604 04:14:47.434118 1274943 otel.go:29] OpenTelemetry logs collector endpoint:
I0604 04:14:47.434292 1274943 metadata.go:67] cloud provider:
I0604 04:14:47.434303 1274943 collector.go:157] instance metadata: <nil>
I0604 04:14:47.434435 1274943 profiling.go:52] profiles endpoint:
W0604 04:14:52.105964 1274943 registry.go:82] couldn't connect to containerd through the following UNIX sockets [/var/snap/microk8s/common/run/containerd.sock,/run/k0s/containerd.sock,/run/k3s/containerd/containerd.sock,/run/containerd/containerd.sock]: failed to dial "/proc/1/root/run/containerd/containerd.sock": context deadline exceeded
W0604 04:14:52.105964 1274943 registry.go:82] couldn't connect to containerd through the following UNIX sockets [/var/snap/microk8s/common/run/containerd.sock,/run/k0s/containerd.sock,/run/k3s/containerd/containerd.sock,/run/containerd/containerd.sock]: failed to dial "/proc/1/root/run/containerd/containerd.sock": context deadline exceeded
W0604 04:14:52.106002 1274943 registry.go:85] stat /proc/1/root/var/run/crio/crio.sock: no such file or directory
W0604 04:14:52.106002 1274943 registry.go:85] stat /proc/1/root/var/run/crio/crio.sock: no such file or directory
I0604 04:14:52.107340 1274943 tracer.go:79] L7 tracing is disabled
F0604 04:14:52.534310 1274943 main.go:149] failed to link program: reading file "/sys/kernel/debug/tracing/events/syscalls/sys_enter_connect/id": open /sys/kernel/debug/tracing/events/syscalls/sys_enter_connect/id: no such file or directory
F0604 04:14:52.534310 1274943 main.go:149] failed to link program: reading file "/sys/kernel/debug/tracing/events/syscalls/sys_enter_connect/id": open /sys/kernel/debug/tracing/events/syscalls/sys_enter_connect/id: no such file or directory
F0604 04:14:52.534310 1274943 main.go:149] failed to link program: reading file "/sys/kernel/debug/tracing/events/syscalls/sys_enter_connect/id": open /sys/kernel/debug/tracing/events/syscalls/sys_enter_connect/id: no such file or directory
F0604 04:14:52.534310 1274943 main.go:149] failed to link program: reading file "/sys/kernel/debug/tracing/events/syscalls/sys_enter_connect/id": open /sys/kernel/debug/tracing/events/syscalls/sys_enter_connect/id: no such file or directory
F0604 04:14:52.534310 1274943 main.go:149] failed to link program: reading file "/sys/kernel/debug/tracing/events/syscalls/sys_enter_connect/id": open /sys/kernel/debug/tracing/events/syscalls/sys_enter_connect/id: no such file or directory
[sw@localhost docker_compose]$ ls /sys/kernel/debug/tracing/events/
ls: cannot access '/sys/kernel/debug/tracing/events/': Permission denied
[sw@localhost docker_compose]$ sudo ls /sys/kernel/debug/tracing/events/
alarmtimer  cma           drm   filemap       hns3     io_uring  libata   net         percpu        rcu      sched   spi      thermal  workqueue
block       compaction        enable    fs_dax        huge_memory  ipi       mdio     nvme        power     regmap   scsi    sunrpc   timer    writeback
bpf_trace   context_tracking  ext4  ftrace        i2c      irq       migrate  oom         printk        rpcrdma  signal  swiotlb  ucsi     xdp
bridge      cpuhp         fib   gpio          ib_mad       jbd2      module   page_isolation  qdisc     rpm      skb     target   udp      xfs
cgroup      devlink       fib6  header_event  initcall     kmem      napi     pagemap         ras       rseq     smbus   task     vmscan   xhci-hcd
clk     dma_fence         filelock  header_page   iommu    kvm       neigh    page_pool       raw_syscalls  rtc      sock    tcp      wbt
[sw@localhost docker_compose]$ sudo ls /sys/kernel/debug/tracing/events/syscalls

I guess maybe some feature on tracing is not supported in this kernel , I disabled trace with "--disable-l7-tracing" it still can not start.