search

parca-dev / parca-agent

eBPF based always-on profiler auto-discovering targets in Kubernetes and systemd, zero code changes or restarts needed!
https://parca.dev/
Apache License 2.0
543 stars 67 forks source link

Fail to start up with: Failed to probe tracepoint #2976

Open jpmeijers opened 1 month ago

jpmeijers commented 1 month ago

Running parca-agent in Docker on Unraid 7.0.0-beta.2.

# uname -a
Linux Magda 6.8.12-Unraid #3 SMP PREEMPT_DYNAMIC Tue Jun 18 07:52:57 PDT 2024 x86_64 Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz GenuineIntel GNU/Linux

The container exits soon after startup due to it not finding /sys/kernel/debug/tracing/events/syscalls/sys_enter_mmap/id.

parca-agent-1  | time="2024-09-03T07:36:39Z" level=error msg="Failed to probe tracepoint: failed to get id for tracepoint: failed to read tracepoint ID for sys_enter_mmap: open /sys/kernel/debug/tracing/events/syscalls/sys_enter_mmap/id: no such file or directory"
parca-agent-1  | time="2024-09-03T07:36:39Z" level=error msg="======================= unexpected error ======================="
parca-agent-1  | time="2024-09-03T07:36:39Z" level=error msg="time=\"2024-09-03T07:36:39Z\" level=error msg=\"Failed to probe tracepoint: failed to get id for tracepoint: failed to read tracepoint ID for sys_enter_mmap: open /sys/kernel/debug/tracing/events/syscalls/sys_enter_mmap/id: no such file or directory\"\n"
parca-agent-1  | time="2024-09-03T07:36:39Z" level=error msg="================================================================"
parca-agent-1  | time="2024-09-03T07:36:39Z" level=error msg="about to report error to server"
parca-agent-1  | time="2024-09-03T07:36:39Z" level=info msg="report sent successfully"
parca-agent-1 exited with code 2

Looking at the host, the kernel debugfs is mounted, but the syscalls directory does not exist.

# ls /sys/kernel/debug/tracing/events/
9p/            cgroup/      devlink/       filelock/     huge_memory/  iommu/        kvmmmu/      migrate/    netlink/         percpu/        rpm/     sock/     tlb/        writeback/
alarmtimer/    clk/         dma_fence/     filemap/      hwmon/        ipi/          kyber/       mmap/       nmi/             power/         rseq/    swiotlb/  tls/        x86_fpu/
amd_cpu/       cma/         drm/           fs_dax/       hyperv/       irq/          libata/      mmap_lock/  notifier/        printk/        rtc/     target/   udp/        xdp/
block/         compaction/  enable         ftrace/       i2c/          irq_matrix/   lock/        module/     nvme/            qdisc/         sched/   task/     vmalloc/    xfs/
bpf_test_run/  cpuhp/       error_report/  gpio/         initcall/     irq_vectors/  maple_tree/  msr/        oom/             ras/           scsi/    tcp/      vmscan/     xhci-hcd/
bpf_trace/     csd/         exceptions/    handshake/    intel_iommu/  kmem/         mce/         napi/       page_isolation/  raw_syscalls/  signal/  thermal/  vsyscall/
bridge/        dev/         fib/           header_event  io_uring/     ksm/          mdio/        neigh/      page_pool/       rcu/           skb/     thp/      wbt/
btrfs/         devfreq/     fib6/          header_page   iomap/        kvm/          mei/         net/        pagemap/         regmap/        smbus/   timer/    workqueue/
umanwizard commented 2 weeks ago

Hi @jpmeijers ,

Can you please check if your kernel was compiled with CONFIG_HAVE_SYSCALL_TRACEPOINTS=y ?

umanwizard commented 2 weeks ago

Same question for CONFIG_FTRACE_SYSCALLS

umanwizard commented 2 weeks ago

Possibly, this is fixed by https://github.com/parca-dev/parca-agent/pull/2990

umanwizard commented 2 weeks ago

Presumably related: https://github.com/evilsocket/opensnitch/issues/774

umanwizard commented 2 weeks ago

@jpmeijers , can you please try again with the most recent image? ghcr.io/parca-dev/parca-agent:main-48f7888d

jpmeijers commented 15 hours ago

I tried the new container version on two different hosts. A physical Dell server running Unraid, and a VM running Ubuntu 20.04. Both fail with the same error:

Failed to load eBPF tracer: failed to read kernel symbols: unable to open /proc/kallsyms: open /proc/kallsyms: no such file or directory

Unraid:

# grep FTRACE /boot/config-$(uname -r)
grep: /boot/config-6.8.12-Unraid: No such file or directory

VM + Ubuntu:

# grep FTRACE /boot/config-$(uname -r)
CONFIG_HAVE_KPROBES_ON_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_DYNAMIC_FTRACE_WITH_REGS=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_FTRACE=y
# CONFIG_FTRACE_SYSCALLS is not set
jpmeijers commented 15 hours ago

The tracepoint error has however been replaced by a warning, so this issue can be closed. I'll have to open a new one about eBPF.

parca-agent-1  | 2024/10/25 13:36:42 INFO memory is not limited, skipping package=github.com/KimMachineGun/automemlimit/memlimit
parca-agent-1  | 2024/10/25 13:36:42 INFO memory is not limited, skipping package=github.com/KimMachineGun/automemlimit/memlimit
parca-agent-1  | ooooooooo.                                                  .o.                                            .
parca-agent-1  | `888   `Y88.                                               .888.                                         .o8
parca-agent-1  |  888   .d88'  .oooo.   oooo d8b  .ooooo.   .oooo.         .8"888.      .oooooooo  .ooooo.  ooo. .oo.   .o888oo
parca-agent-1  |  888ooo88P'  `P  )88b  `888""8P d88' `"Y8 `P  )88b       .8' `888.    888' `88b  d88' `88b `888P"Y88b    888
parca-agent-1  |  888          .oP"888   888     888        .oP"888      .88ooo8888.   888   888  888ooo888  888   888    888
parca-agent-1  |  888         d8(  888   888     888   .o8 d8(  888     .8'     `888.  `88bod8P'  888    .o  888   888    888 .
parca-agent-1  | o888o        `Y888""8o d888b    `Y8bod8P' `Y888""8o   o88o     o8888o `8oooooo.  `Y8bod8P' o888o o888o   "888"
parca-agent-1  |                                                                       d"     YD
parca-agent-1  |                                                                       "Y88888P'
parca-agent-1  | 
parca-agent-1  | time="2024-10-25T13:36:42Z" level=warning msg="Failed to probe tracepoint: failed to get id for tracepoint: failed to read tracepoint ID for sys_enter_mmap: open /sys/kernel/debug/tracing/events/syscalls/sys_enter_mmap/id: no such file or directory. Parca-agent may fail to run on some kernel versions."
parca-agent-1  | time="2024-10-25T13:36:42Z" level=info msg="External labels: "
parca-agent-1  | time="2024-10-25T13:36:42Z" level=info msg="Interpreter tracers: perl,php,python,hotspot,ruby,v8,dotnet"
parca-agent-1  | time="2024-10-25T13:36:42Z" level=info msg="no config file provided, using default config"
parca-agent-1  | time="2024-10-25T13:36:42Z" level=info msg="Can't connect Docker client to [/var/run/docker.sock]"
parca-agent-1  | time="2024-10-25T13:36:42Z" level=info msg="Environment variable KUBERNETES_SERVICE_HOST not set"
parca-agent-1  | time="2024-10-25T13:36:42Z" level=error msg="Failed to load eBPF tracer: failed to read kernel symbols: unable to open /proc/kallsyms: open /proc/kallsyms: no such file or directory"
parca-agent-1  | time="2024-10-25T13:36:42Z" level=error msg="======================= unexpected error ======================="
parca-agent-1  | time="2024-10-25T13:36:42Z" level=error msg="2024/10/25 13:36:42 INFO memory is not limited, skipping package=github.com/KimMachineGun/automemlimit/memlimit\ntime=\"2024-10-25T13:36:42Z\" level=warning msg=\"Failed to probe tracepoint: failed to get id for tracepoint: failed to read tracepoint ID for sys_enter_mmap: open /sys/kernel/debug/tracing/events/syscalls/sys_enter_mmap/id: no such file or directory. Parca-agent may fail to run on some kernel versions.\"\ntime=\"2024-10-25T13:36:42Z\" level=info msg=\"External labels: \"\ntime=\"2024-10-25T13:36:42Z\" level=info msg=\"Interpreter tracers: perl,php,python,hotspot,ruby,v8,dotnet\"\ntime=\"2024-10-25T13:36:42Z\" level=info msg=\"no config file provided, using default config\"\ntime=\"2024-10-25T13:36:42Z\" level=info msg=\"Can't connect Docker client to [/var/run/docker.sock]\"\ntime=\"2024-10-25T13:36:42Z\" level=info msg=\"Environment variable KUBERNETES_SERVICE_HOST not set\"\ntime=\"2024-10-25T13:36:42Z\" level=error msg=\"Failed to load eBPF tracer: failed to read kernel symbols: unable to open /proc/kallsyms: open /proc/kallsyms: no such file or directory\"\n"
parca-agent-1  | time="2024-10-25T13:36:42Z" level=error msg="================================================================"
parca-agent-1  | time="2024-10-25T13:36:42Z" level=error msg="about to report error to server"
parca-agent-1  | time="2024-10-25T13:36:43Z" level=warning msg="finished callprotocolgrpcgrpc.componentclientgrpc.serviceparca.telemetry.v1alpha1.TelemetryServicegrpc.methodReportPanicgrpc.method_typeunarygrpc.start_time2024-10-25T13:36:42Zgrpc.request.deadline2024-10-25T13:38:42Zgrpc.codeUnimplementedgrpc.errorrpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type \"text/plain; charset=utf-8\"grpc.time_ms152.119"
parca-agent-1  | time="2024-10-25T13:36:43Z" level=error msg="failed to call ReportPanic(): rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type \"text/plain; charset=utf-8\""
parca-agent-1 exited with code 1