search

parca-dev / parca-agent

eBPF based always-on profiler auto-discovering targets in Kubernetes and systemd, zero code changes or restarts needed!
https://parca.dev/
Apache License 2.0
532 stars 66 forks source link

symbolizer/user: Investigate bogus addresses #710

Open javierhonduco opened 2 years ago

javierhonduco commented 2 years ago

On current main:

image
javierhonduco commented 2 years ago

Haven't checked in detail yet, but I wonder if this could be when the kernel stack failed, but we still have zeroed in the merged stack.

If this is the case, we should signify this somehow so when it fails, we don't store the zeroed stacktrace. What I am not sure about is why we didn't notice this before?

brancz commented 2 years ago

Just in case this is not already known, we recently introduced on parca server side to throw away any address that can never be symbolized, eg has no mapping. We needed to do this as each unique address causes a key in the metastore to be created.

javierhonduco commented 2 years ago

Haven't checked in detail yet, but I wonder if this could be when the kernel stack failed, but we still have zeroed in the merged stack.

This is not correct, we skip zeroed frames.

https://github.com/parca-dev/parca-agent/blob/0d4f3102f590577d9145a0718f6cd409f92d22b4/pkg/profiler/cpu/cpu.go#L461

And yes, it seemed to be what you mention, Frederic, the issue lies in unmapped sections, so we should delve into the root cause on Parca Agent's side. Have some hunches but need some more time to test the hypothesis

javierhonduco commented 8 months ago

Should be fixed now @Sylfrena @kakkoyun