search

KDAB / hotspot

The Linux perf GUI for performance analysis.
4.16k stars 257 forks source link

Mysterious symbols showed up in HotSpot but not in perf-report, nor Tracy. #480

Closed mcourteaux closed 1 year ago

mcourteaux commented 1 year ago

I have little idea of what is going on, but I wrote down my journey here on Superuser.com:

https://superuser.com/questions/1778059/nft-pipapo-avx2-scratch-index-shows-up-on-all-my-perf-recordings/1778060#1778060

I think this could be due to a bug in either perf or hotspot, or in the way they communicate the information over the perf.data file.

milianw commented 1 year ago

Without a way to reproduce this issue is not actionable. Please provide that information and then reopen this ticket.

There are multiple ways that things might fail, e.g. no/missing kallsym mapping information leading to random associations or just a wrong kallsym being used. Or wrong dsos being used or ...

As is this ticket is worthless as it's not actionable

jasonk000 commented 1 year ago

This seems related to kallsyms as @milianw already suggested. I am able to reproduce this recording from remote machine and bringing back to local:

Remote:

$ uname -r
6.2.0-1009-aws

$ sudo perf record -e cycles -g -F max du -hs /tmp
info: Using a maximum frequency rate of 250 Hz
7.4M    /tmp
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.010 MB perf.data (2 samples) ]

$ sudo perf script | grep nft
<no output>

Local:

$ uname -r
6.2.0-32-generic

$ apt list --installed | grep 6.2.0.*dbgsym
...
linux-image-6.2.0-1009-aws-dbgsym/lunar-updates,now 6.2.0-1009.9 amd64 [installed]
linux-image-6.2.0-32-generic-dbgsym/lunar-updates,now 6.2.0-32.32 amd64 [installed]
linux-image-unsigned-6.2.0-1009-aws-dbgsym/lunar-updates,now 6.2.0-1009.9 amd64 [installed,automatic]
linux-image-unsigned-6.2.0-32-generic-dbgsym/lunar-updates,now 6.2.0-32.32 amd64 [installed,automatic]

$ hotspot perf.data

Opening in hotspot: image

Now, when I record with --kcore, and repeat the same process, providing hotspot with the --kallsyms location, I get the following:

image

And, minimally, copying kallsysms only, and without perf record --kcore, I get a correct working result as well.

So, I think the question mark here is, what should be displayed in hotspot if kallsyms is not matching. I didn't check with perf but probably it inspects some of the MMAP_ records in the perf.data file to see if everything lines up before reading with kallsyms.