grafana / pyroscope

Continuous Profiling Platform. Debug performance issues down to a single line of code
https://grafana.com/oss/pyroscope/
GNU Affero General Public License v3.0
9.98k stars 598 forks source link

ebpf(python): does not work with memory swapping enabled nicely #3267

Open korniltsev opened 5 months ago

korniltsev commented 5 months ago

When a page is not physically in memory and we try to read it with bpf_probe_read_user, we get EFAULT We can try doing couple of things:

  1. If we hit an issue - try to continue unwinding and reading symbols, replace failed frame with some sort of "Failed" string
  2. Notify userspace about unavailable memory region, and userspace can try to read from the region by using /proc/pid/mem. This works OK during tests under no swap pressure, not sure how it will work under pressure, need tests.

This should only be enabled for dev/debug environments, we should not mess with page swapping in prod.

A python test to force a page to swap ```python3 #!/usr/bin/env python3 import time import logging import os import threading l = logging.getLogger() l.setLevel(logging.DEBUG) class Foo: def __init__(self): self.bar = 42 def work(self, n): i = 0 while i < n: i += 1 # heavy cpu random # i += os.urandom(1000)[0] def fast_function(self, ): self.work(2000000) def slow_function(self, ): self.work(8000000) def toplevel0(self, hui): self.toplevel1__л(hui) def toplevel1__л(self, hui): self.toplevel2(hui) def toplevel2(self, hui): while True: print(self, hui) self.fast_function() self.slow_function() time.sleep(5) def toplevel0(self, hui): Foo().toplevel0(hui) if __name__ == "__main__": import ctypes libc = ctypes.CDLL(None) t1 = threading.Thread(target=toplevel0, args=('hui', 'hui')) t1.start() voidp = ctypes.c_voidp while True: addr = input("Enter address: ").strip() print(addr) addr = int(addr, 16) addr = addr & 0xFFFFFFFFFFFFF000 # define MADV_PAGEOUT 21 /* Reclaim these pages. */ print(hex(addr)) addr = ctypes.cast(addr, voidp) print(addr) res = libc.madvise(addr, 0x1000, 21) print(f' res = {res} ') ```