yahoo / kubectl-flame

Kubectl plugin for effortless profiling on kubernetes
Apache License 2.0
796 stars 96 forks source link

no data? #73

Open clcc2019 opened 2 years ago

clcc2019 commented 2 years ago
image image
benjaminxie commented 2 years ago

@clcc2019 Did you ever figure out what was going wrong?

benjaminxie commented 1 year ago

I ran into this issue as well and I have some suspicions as to what may be going wrong.

Here's some relevant details

Since I'm trying to profile a Java application, the profiler that kubectl-flame will put on my container is async-profiler. According to their docs, I should be seeing output in my application logs similar to the following

--- Execution profile ---
Total samples:           687
Unknown (native):        1 (0.15%)

--- 6790000000 (98.84%) ns, 679 samples
  [ 0] Primes.isPrime
  [ 1] Primes.primesThread
  [ 2] Primes.access$000
  [ 3] Primes$1.run
  [ 4] java.lang.Thread.run

... a lot of output omitted for brevity ...

          ns  percent  samples  top
  ----------  -------  -------  ---
  6790000000   98.84%      679  Primes.isPrime
    40000000    0.58%        4  __do_softirq

... more output omitted ...

Instead, I see something like

WARNING: Kernel symbols are unavailable due to restrictions. Try
  echo 0 > /proc/sys/kernel/kptr_restrict
  echo 1 > /proc/sys/kernel/perf_event_paranoid
perf_event_open failed: Permission denied
perf_event_open failed: Permission denied
perf_event_open failed: Permission denied
perf_event_open failed: Permission denied
perf_event_open failed: Permission denied

If I try either one of echo 0 > /proc/sys/kernel/kptr_restrict or echo 1 > /proc/sys/kernel/perf_event_paranoid, I get sysctl: error setting key '<kernel.kptr_restrict or perf_even_paranoid>': Read-only file system

I think getting my flamegraph means either finding a way to set these two runtime variables or finding a workaround for this.

pvorb commented 1 year ago

@benjaminxie Does the sysctl command suggested in this comment work?

benjaminxie commented 1 year ago

@pvorb Not for me :( .