search

CompArchCam / optiwise

OptiWISE profiling tool for granular CPI analysis.
MIT License
5 stars 0 forks source link

Analyze fails in the presence of other `.text` sections #20

Open tobast opened 1 week ago

tobast commented 1 week ago

Hello,

Upon a fresh git clone (at commit d815add9, current HEAD) and make, I tried running the example from the readme file:

optiwise run -- /usr/bin/echo hello

This however yields during the analysis phase

[snip]
Info: program exited; processing...
Info: merge blocks
Info: output cfg
Info: exit
Info: Reading disassebly from /dev/fd/4...
Error: Uncaught exception:
/dev/fd/4:2980 error: duplicated instruction in objdump?

Further investigating, the disassembly phase disassembles all of my kernel modules (eg /lib/modules/…-amd64/kernel/kernel/arch/x86/crypto/chacha-x86_64.ko).

Opening optiwise_result/disassemble/result.txt.gz at line 2980 points me to the disassembly of /lib/modules/5.10.0-27-amd64/kernel/net/bridge/br_netfilter.ko, with the following disassembly:

[…]
Disassembly of section .text.unlikely:

0000000000000000 <br_nf_pre_routing.cold>:
br_nf_pre_routing.cold():
   0:   mov    $0x0,%rdi                                               ## ====== LINE 2980 HERE
   7:   movb   $0x1,0x0(%rip)        # e <br_nf_pre_routing.cold+0xe>
   e:   callq  13 <br_nf_pre_routing.cold+0x13>
  13:   xor    %eax,%eax
  15:   jmpq   1a <.LC3+0xf>

Disassembly of section .init.text:

0000000000000000 <init_module>:
br_netfilter_init():
   0:   callq  5 <init_module+0x5>
   5:   push   %r12
   7:   mov    $0x0,%rdi

where, indeed, a second instruction for address 0 is disassembled (the first one being at the beginning of the .text section).

Looking around the exception thrown in the Optiwise code (src/analyzer/io.cpp:705 at commit d815add9), it indeed seems that the objdump_result C++ map is indexed by (current_module, addr), not accounting for the possible presence of multiple assembly sections in analyzed ELFs.

This prompts me with two questions.

Chadderz121 commented 1 week ago

Hello,

Thanks for the detailed report and analysis!

I think there are two problems here:

  1. optiwise sample is sampling kernel modules (e.g. chacha-x86_64.ko) when it really shouldn't be.
  2. optiwise analyze is choking on said kernel modules

1 sample bug

We're (probably) only interested in user space. On my system, optiwise sample only samples the process of interest. I'd be very interested to know what OS you're using, as maybe the defaults are different? optiwise sample is ultimately a wrapper for perf record and I can't seem to see an appropriate perf flag to avoid this.

To follow up: what is the output for you for perf --buildid-dir optiwise_result/binaries buildid-list -i optiwise_result/sample/result.perf.data and perf --buildid-dir optiwise_result/binaries script -i optiwise_result/sample/result.perf.data? The former command lists all the modules perf encountered during sampling, and we use this to generate the list for optiwise disassemble. The second command shows the raw samples that perf recorded: I notice that even on my system the first command overreports modules compared to the second; perhaps that's the true problem here: that the module list is bigger than the set of modules actually sampled.

2 analyze bug

Those kernel modules it encounters seem to cause an issue because (on my system too) they seem to lack a program header. I think this causes the overlapping addresses, as each section's addresses are relative to their start. I'm sure we can arrange some quick and dirty fix to prevent the crash here, but fundamentally optiwise wouldn't work correctly if there were samples on such a module, as you quite rightly report that we use (module, addr) pairs to identify code ubiquitously. I think in the short term the best fix is to throw an error for such modules only if they are sampled.

Chadderz121 commented 1 week ago

Further to this, if you try the patch in #21, you should (hopefully) be able to run without issues. This fixes the second problem I mentioned.

tobast commented 1 week ago

Hello,

Thanks for your quick reply!

For now I've "mitigated" the issue on my local version with the following patch:

diff --git a/scripts/share/optiwise/bin/optiwise-disassemble b/scripts/share/optiwise/bin/optiwise-disassemble
index 84fbb2a..3429256 100755
--- a/scripts/share/optiwise/bin/optiwise-disassemble
+++ b/scripts/share/optiwise/bin/optiwise-disassemble
@@ -199,9 +199,12 @@ for line in $with_buildid; do
     )
   fi

+  basename="$(basename "$file")"
   if [ "$file" = "[kernel.kallsyms]" ]; then
     # TODO support for kernel debugging
     continue
+  elif [ "${basename#*.}" = "ko" ] ; then  # kernel module: ignore
+    continue
   elif [ "$file" = "[vdso]" ]; then
     elf="$buildid_dir/.build-id/$buildid/vdso"
   else

This basically ignores any *.ko file in optiwise-disassemble, which both speeds up optiwise and lets it complete without error.

System used

I've met the issue on a Debian Bullseye

$ uname -a 
Linux … 5.10.0-27-amd64 #1 SMP Debian 5.10.205-2 (2023-12-31) x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye

I can try to reproduce whenever I have time on an up-to-date Archlinux.

$ uname -a
Linux … 6.6.48-1-lts #1 SMP PREEMPT_DYNAMIC Thu, 29 Aug 2024 17:56:14 +0000 x86_64 GNU/Linux

Perf output

For a run with /usr/bin/echo, buildid-list yields (somehow) the whole list of kernel modules:

$ perf --buildid-dir optiwise_result/binaries buildid-list -i optiwise_result/sample/result.perf.data
de1c5813daf08d4691ae0ec37c848b35728008f4 [kernel.kallsyms]
87f9b1000f910e403361c45dd43cbe4e24999e34 /lib/modules/5.10.0-27-amd64/kernel/fs/btrfs/btrfs.ko
7a79bf7fd2cf1b270017ac7209024fa77df9b475 /lib/modules/5.10.0-27-amd64/kernel/crypto/blake2b_generic.ko
42f4fcd1bada63f976ca793c8e6acf67fd59d64b /lib/modules/5.10.0-27-amd64/kernel/crypto/xor.ko
892bf2c4fb0871d21993b9ad28a365a90ee8d73d /lib/modules/5.10.0-27-amd64/kernel/lib/raid6/raid6_pq.ko
e75e275f09cc7f06baac2f99fcd1ce780123689b /lib/modules/5.10.0-27-amd64/kernel/fs/ufs/ufs.ko
840f7bbc929dac65ef0ba202b289ebde7dfa7d71 /lib/modules/5.10.0-27-amd64/kernel/fs/qnx4/qnx4.ko
57eae870f054b710d78f1571d9f2cd60182a9efe /lib/modules/5.10.0-27-amd64/kernel/fs/hfsplus/hfsplus.ko
77e4cbc621a7b2ffc8127ba4c7ac0614f9b6d347 /lib/modules/5.10.0-27-amd64/kernel/fs/hfs/hfs.ko
379c049b673d9fdcc544da5ecb635faf9e97a494 /lib/modules/5.10.0-27-amd64/kernel/drivers/cdrom/cdrom.ko
0c15df7e536479c6f66073cb0ade6fb3a8d8407b /lib/modules/5.10.0-27-amd64/kernel/fs/minix/minix.ko
235b61e3b67c08715911c4264301137194870d18 /lib/modules/5.10.0-27-amd64/kernel/fs/fat/msdos.ko
82c03a24384f8603da320cb136f770011136eda2 /lib/modules/5.10.0-27-amd64/kernel/fs/jfs/jfs.ko
8ae3a345d089af716e7802a27e998c20f5f08e40 /lib/modules/5.10.0-27-amd64/kernel/fs/xfs/xfs.ko
7e425472c8ca0d89a67786f9f92b0a65ac9da8f4 /lib/modules/5.10.0-27-amd64/kernel/drivers/net/tun.ko
27e374394655d599196e3483dabd9b76fd411ad6 /lib/modules/5.10.0-27-amd64/kernel/net/bluetooth/bluetooth.ko
9a2d7c55a0c6c78be513ea934004e969386aa1ef /lib/modules/5.10.0-27-amd64/kernel/crypto/jitterentropy_rng.ko
8198abcdcd59f869fdc83725d69e4eb5590c58a3 /lib/modules/5.10.0-27-amd64/kernel/crypto/drbg.ko
c53cfc1859af9f33c178800fac895a19e877c945 /lib/modules/5.10.0-27-amd64/kernel/crypto/ansi_cprng.ko
86d863dc883d543251e4aa1bc9b7c6106957e81c /lib/modules/5.10.0-27-amd64/kernel/crypto/ecdh_generic.ko
1cafc1c4952df9a8bb6bfe5cecb157fb913417cc /lib/modules/5.10.0-27-amd64/kernel/net/rfkill/rfkill.ko
b38f00aef287990aa69504dba649c968617fe54e /lib/modules/5.10.0-27-amd64/kernel/crypto/ecc.ko
6b0384c3ded7eaa251cedb5653326f3880c2e148 /lib/modules/5.10.0-27-amd64/kernel/drivers/net/veth.ko
9266624879af80b02c1f46fe59375472de661102 /lib/modules/5.10.0-27-amd64/kernel/arch/x86/kernel/msr.ko
a93521bbc33e74e095255db45266d8e80a7d4845 /lib/modules/5.10.0-27-amd64/misc/vboxnetadp.ko
33e4db7fec4969ace49503a6400dedb393ed11d4 /lib/modules/5.10.0-27-amd64/misc/vboxnetflt.ko
4cab69a01b86762e260c3308581e4c5102fa62ee /lib/modules/5.10.0-27-amd64/misc/vboxdrv.ko
505b5e378a94a885d1424a2b0c7c7b9d44fb8425 /lib/modules/5.10.0-27-amd64/kernel/net/netfilter/xt_MASQUERADE.ko
426d2c44071e16843529fbc4091c2bebefe84173 /lib/modules/5.10.0-27-amd64/kernel/net/netfilter/nf_conntrack_netlink.ko
ed578c6197234a041eb2e65727d0aa9a76c7f6c0 /lib/modules/5.10.0-27-amd64/kernel/net/xfrm/xfrm_user.ko
60e15d4749183e817d148f73e5d9a81e77b7b9ec /lib/modules/5.10.0-27-amd64/kernel/net/xfrm/xfrm_algo.ko
08afbdb42d0199f9dfb56c20ea728824e74aacf4 /lib/modules/5.10.0-27-amd64/kernel/net/netfilter/xt_addrtype.ko
01c02181cd3d113f7044d5834600c140ead0c5a2 /lib/modules/5.10.0-27-amd64/kernel/net/bridge/br_netfilter.ko
ef3a088605795c94892d247ad8bdd82bae07ecad /lib/modules/5.10.0-27-amd64/kernel/net/bridge/bridge.ko
389a07097dd6f570950e69f6f515254063015b57 /lib/modules/5.10.0-27-amd64/kernel/net/802/stp.ko
7608279fc3082fcfd3c8c6ca109cf42b8b09d5a8 /lib/modules/5.10.0-27-amd64/kernel/net/llc/llc.ko
8ee37cf37ed0e1a8009a2cc2d8a7c9a502845c8f /lib/modules/5.10.0-27-amd64/kernel/fs/overlayfs/overlay.ko
6c745c4a24df2d3c79880ef83c1943cd59e97e62 /lib/modules/5.10.0-27-amd64/kernel/drivers/net/wireguard/wireguard.ko
37d162022e00cf78ec780c3cba74a298d325211f /lib/modules/5.10.0-27-amd64/kernel/arch/x86/crypto/curve25519-x86_64.ko
faf16381afdf1cd1dea516f0044c9064058fcbc3 /lib/modules/5.10.0-27-amd64/kernel/lib/crypto/libchacha20poly1305.ko
f3e38798c7ec4e6c5fff743bbda1b8a8a9cb471a /lib/modules/5.10.0-27-amd64/kernel/arch/x86/crypto/chacha-x86_64.ko
06db8af48aaf40a03209101340d91e09468eb3f2 /lib/modules/5.10.0-27-amd64/kernel/arch/x86/crypto/poly1305-x86_64.ko
5323e060db9b8c7cc78ab80e704ebf3e886b8b97 /lib/modules/5.10.0-27-amd64/kernel/net/ipv6/ip6_udp_tunnel.ko
4be3a33abe6f087f47ffeb6bb9efd6d19a457dc1 /lib/modules/5.10.0-27-amd64/kernel/net/ipv4/udp_tunnel.ko
4e902d7f1f5a2d605d5447f59df2b38dc3fa9987 /lib/modules/5.10.0-27-amd64/kernel/lib/crypto/libcurve25519-generic.ko
d3e0370f883b8c9c19195a463d497f4aa4493d03 /lib/modules/5.10.0-27-amd64/kernel/lib/crypto/libchacha.ko
d3e0f7c4452b952d3a1089a9df0751c8b70d4d91 /lib/modules/5.10.0-27-amd64/kernel/net/netfilter/nft_chain_nat.ko
467e3da10f9bc600d07a4e9af2df436bba981373 /lib/modules/5.10.0-27-amd64/kernel/net/netfilter/nf_nat.ko
a62ee2166f738d20e18abd2dfda075a9840cd260 /lib/modules/5.10.0-27-amd64/kernel/net/netfilter/nft_counter.ko
ba28bf980f6d3c0385f50d2aa63ea5d4dbe43743 /lib/modules/5.10.0-27-amd64/kernel/net/netfilter/xt_tcpudp.ko
039cd58dbf98c93a1ea3bb2f029980f0f5006b9b /lib/modules/5.10.0-27-amd64/kernel/net/netfilter/xt_state.ko
ce41dbd6413c791807b3d171bac2296bbc869b67 /lib/modules/5.10.0-27-amd64/kernel/net/netfilter/xt_conntrack.ko
0fee209655e243c4561ad68e8d8633bfa32ea46b /lib/modules/5.10.0-27-amd64/kernel/net/netfilter/nf_conntrack.ko
ccec5286e619a206ceaf9c4a4fdb0b6baeeff207 /lib/modules/5.10.0-27-amd64/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko
1c86e2e995913426a3b085b12d4dd23252ae4e00 /lib/modules/5.10.0-27-amd64/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko
e9682240f7569cc0ef86af1374d93cd92e9c4e92 /lib/modules/5.10.0-27-amd64/kernel/net/ipv4/netfilter/ipt_REJECT.ko
3b702eefd8b5ac5239e982e95ac32b80603dd7d5 /lib/modules/5.10.0-27-amd64/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko
2ed7566c6f167876ee5090e8c0d0ab61656f1a46 /lib/modules/5.10.0-27-amd64/kernel/net/netfilter/nft_compat.ko
dc92490fe49d0aaa2b2383036f088f810abb15ce /lib/modules/5.10.0-27-amd64/kernel/net/netfilter/nf_tables.ko
da6c218d715306ac6ed4990c37cb6b69578f8846 /lib/modules/5.10.0-27-amd64/kernel/lib/libcrc32c.ko
d90a31faecad169127dac8eb46d5b7b0108042e4 /lib/modules/5.10.0-27-amd64/kernel/net/netfilter/nfnetlink.ko
b57a827bc1558ccbd71793993fb54001ed562bbe /lib/modules/5.10.0-27-amd64/kernel/drivers/powercap/intel_rapl_msr.ko
ada2c3a46e1582c0f293e194bfd4ad0187c371ce /lib/modules/5.10.0-27-amd64/kernel/drivers/powercap/intel_rapl_common.ko
add619491237e19cf7c831383986d44c15c5c2ee /lib/modules/5.10.0-27-amd64/kernel/drivers/edac/skx_edac.ko
3a50be55c565e20e048036dec84d9500b06b1683 /lib/modules/5.10.0-27-amd64/kernel/drivers/acpi/nfit/nfit.ko
3ef9d50f8bb5c6be67293b7b46a37f808cc31256 /lib/modules/5.10.0-27-amd64/kernel/drivers/nvdimm/libnvdimm.ko
ea359019482dcde8ea35344143d9776fdc774cee /lib/modules/5.10.0-27-amd64/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko
17277696f8318db03560a5ce7fe597cd3ffeedf6 /lib/modules/5.10.0-27-amd64/kernel/drivers/thermal/intel/intel_powerclamp.ko
4791142c69e366921438459e9c06ee73f21f180d /lib/modules/5.10.0-27-amd64/kernel/drivers/hwmon/coretemp.ko
89ff84575c0f13a23ab644f530dea820b0004560 /lib/modules/5.10.0-27-amd64/kernel/arch/x86/kvm/kvm-intel.ko
e71fe6bb78db2e116bc5549ac6307354853d1757 /lib/modules/5.10.0-27-amd64/kernel/drivers/char/ipmi/ipmi_ssif.ko
ed92d13854c2842846c8863d40b14fa883078917 /lib/modules/5.10.0-27-amd64/kernel/arch/x86/kvm/kvm.ko
7cae5ad628e04793bb3e671df5606f7d3eeeeb1a /lib/modules/5.10.0-27-amd64/kernel/virt/lib/irqbypass.ko
adbd91e2563fef5e299b80924b8803d083cced76 /lib/modules/5.10.0-27-amd64/kernel/arch/x86/events/rapl.ko
a219c8aae031546298ffdc641e7acac76a35dc8d /lib/modules/5.10.0-27-amd64/updates/dkms/zfs.ko
222bb0877b008ffdb3b457681786538c68f35124 /lib/modules/5.10.0-27-amd64/updates/dkms/zunicode.ko
dbef2131f117633a3cf95512de69c0aa3b673634 /lib/modules/5.10.0-27-amd64/updates/dkms/zzstd.ko
a721468a812f896fea56e4836091774c8b9d70f3 /lib/modules/5.10.0-27-amd64/updates/dkms/zlua.ko
eaadae7741f2a1a81cf5f1949e16071b3913ebfd /lib/modules/5.10.0-27-amd64/updates/dkms/zavl.ko
1803ca6d5876a6bcdac9e50f4e9a1dc99b474d4a /lib/modules/5.10.0-27-amd64/updates/dkms/icp.ko
bfe3f5fbe271f716c51477b406ae25ef5f076874 /lib/modules/5.10.0-27-amd64/kernel/fs/nls/nls_ascii.ko
1c157ffe181e804d8c52a804fdbcf22aee309fec /lib/modules/5.10.0-27-amd64/kernel/fs/nls/nls_cp437.ko
eb3cb320fe33c3ee1c146b73ea26435140ccfb09 /lib/modules/5.10.0-27-amd64/kernel/fs/fat/vfat.ko
8b6003ee74b252e32d8e6cea0f2d94bdd5e2b9ea /lib/modules/5.10.0-27-amd64/kernel/arch/x86/events/intel/intel-cstate.ko
334a99eceeec30db355268814379ba1a4d6c659c /lib/modules/5.10.0-27-amd64/kernel/fs/fat/fat.ko
d574f36af816f20ba05364b90a4905f553838b5f /lib/modules/5.10.0-27-amd64/updates/dkms/zcommon.ko
751179734723c4ca7c229931ab919340701c3548 /lib/modules/5.10.0-27-amd64/updates/dkms/znvpair.ko
3bb0ae1e4015e0fa8b780deef9522ae5b76fb468 /lib/modules/5.10.0-27-amd64/kernel/drivers/gpu/drm/mgag200/mgag200.ko
43a0fe88818078512272143b600ee21201ca594d /lib/modules/5.10.0-27-amd64/updates/dkms/spl.ko
1068cf5380b39db949b9c102e136e2044f5243a5 /lib/modules/5.10.0-27-amd64/kernel/drivers/gpu/drm/drm_kms_helper.ko
66c9267ad7bfccfe29ab7bedfd96f709f03ce987 /lib/modules/5.10.0-27-amd64/kernel/drivers/platform/x86/dell-smbios.ko
75ccf5d745247a6323f3c70e55e61938f677acca /lib/modules/5.10.0-27-amd64/kernel/drivers/watchdog/iTCO_wdt.ko
12bb83501c33988a01f000984186cc48d25fbd2c /lib/modules/5.10.0-27-amd64/kernel/drivers/platform/x86/dcdbas.ko
30cc09b563d6298cf382962b65c64b98c29ba47a /lib/modules/5.10.0-27-amd64/kernel/drivers/input/joydev.ko
d6e3d1dea98e2790beda58dad732da40f052de31 /lib/modules/5.10.0-27-amd64/kernel/drivers/media/cec/core/cec.ko
4d3e98744a55cb6c07cee552933fab2b9508036a /lib/modules/5.10.0-27-amd64/kernel/drivers/mfd/intel_pmc_bxt.ko
7d429d9212657229ff8c8bd508565b8057cc2cc7 /lib/modules/5.10.0-27-amd64/kernel/arch/x86/events/intel/intel-uncore.ko
c26d80f8b6223c1bf003cfafc509eea873c65d1e /lib/modules/5.10.0-27-amd64/kernel/drivers/platform/x86/dell-wmi-descriptor.ko
43c72394fa452545b9592faf977df8f1a2c848ee /lib/modules/5.10.0-27-amd64/kernel/drivers/platform/x86/wmi-bmof.ko
c24d4fc86abf87026c8ff96b8ab35d0d3ee6218c /lib/modules/5.10.0-27-amd64/kernel/drivers/watchdog/iTCO_vendor_support.ko
d979e0e0a02ae68e4f93da92ea7c7e27d29d10fa /lib/modules/5.10.0-27-amd64/kernel/drivers/firmware/efi/efi-pstore.ko
8a055e5ec03f2f58b31c6a94f8ab4f14e254c390 /lib/modules/5.10.0-27-amd64/kernel/drivers/input/misc/pcspkr.ko
b189fc9a22bc15eb9bed4bd1b0ce096496c41cf0 /lib/modules/5.10.0-27-amd64/kernel/drivers/scsi/sg.ko
738229263db681a6c0b299d0b96bcee54b8449f0 /lib/modules/5.10.0-27-amd64/kernel/drivers/i2c/algos/i2c-algo-bit.ko
06a4374b22eff42cb7462d421e625f7e569280d7 /lib/modules/5.10.0-27-amd64/kernel/drivers/misc/mei/mei-me.ko
29910359b290997036b067cbdd9622397a769155 /lib/modules/5.10.0-27-amd64/kernel/drivers/watchdog/watchdog.ko
bf2db9752d186c6391dce8385e94c018ce6554d1 /lib/modules/5.10.0-27-amd64/kernel/drivers/misc/mei/mei.ko
4e8c6a1675e395cd0df9366d37672b23ebe5aedd /lib/modules/5.10.0-27-amd64/kernel/drivers/acpi/acpi_ipmi.ko
b991bd14d9d4f6b5b470cc2cff37dc28f57c0704 /lib/modules/5.10.0-27-amd64/kernel/drivers/input/evdev.ko
7b3c3cddbe1bc1a9da3a8b3e0e8dfa5af89284ae /lib/modules/5.10.0-27-amd64/kernel/drivers/char/ipmi/ipmi_si.ko
88fe17c91ebeff3aff782a8d6f0ec9bb0dcb6f26 /lib/modules/5.10.0-27-amd64/kernel/drivers/char/ipmi/ipmi_devintf.ko
679ee633292630886c20196080161582003cb040 /lib/modules/5.10.0-27-amd64/kernel/drivers/char/ipmi/ipmi_msghandler.ko
1abddecf6aa08c0fae63160103f697dd918ea035 /lib/modules/5.10.0-27-amd64/kernel/drivers/hwmon/acpi_power_meter.ko
0334a8acc2dcab99f983b0b3e95e009bbea33053 /lib/modules/5.10.0-27-amd64/kernel/drivers/acpi/button.ko
79ed6aab1960b7358b64b5236150b6e837108510 /lib/modules/5.10.0-27-amd64/kernel/fs/binfmt_misc.ko
609ee617678a4cb5976311b44f2711f41f9207bb /lib/modules/5.10.0-27-amd64/kernel/drivers/gpu/drm/drm.ko
9f82602f8a2f9b6fbcf7330528e0f469da465d41 /lib/modules/5.10.0-27-amd64/kernel/fs/fuse/fuse.ko
a0d117d8c8e53bcdbdafccce8fc5527dd6a2d356 /lib/modules/5.10.0-27-amd64/kernel/fs/configfs/configfs.ko
170f09921be5c300d294672b663456b4b723e238 /lib/modules/5.10.0-27-amd64/kernel/fs/efivarfs/efivarfs.ko
afe47c8fcb0494508df52b54c39006e7cfce8988 /lib/modules/5.10.0-27-amd64/kernel/net/ipv4/netfilter/ip_tables.ko
99f92bec4fd883c765c1c7bf19d311e541b41b69 /lib/modules/5.10.0-27-amd64/kernel/net/netfilter/x_tables.ko
9f638458765db7438b557f247388027b155132c7 /lib/modules/5.10.0-27-amd64/kernel/fs/autofs/autofs4.ko
b7f292f67be83e2830b6ebab10c93e8887e53938 /lib/modules/5.10.0-27-amd64/kernel/fs/ext4/ext4.ko
2a1a5bcabc00602dcdc12b5dd3f4c33f24eb1219 /lib/modules/5.10.0-27-amd64/kernel/lib/crc16.ko
f28fa58b0d536b1f902f3273c330065e14ac8e09 /lib/modules/5.10.0-27-amd64/kernel/fs/mbcache.ko
03a4f6d931c0468a176ba609c8b1ed1947697919 /lib/modules/5.10.0-27-amd64/kernel/fs/jbd2/jbd2.ko
9999cf071b845869d8efee824a814860596858e0 /lib/modules/5.10.0-27-amd64/kernel/crypto/crc32c_generic.ko
c3590b9e56ced6d9efb06f7b09422ccc04fb5420 /lib/modules/5.10.0-27-amd64/kernel/drivers/md/dm-crypt.ko
9af8f51d0ee82d8e310a0adc1621d83e74978877 /lib/modules/5.10.0-27-amd64/kernel/drivers/md/dm-mod.ko
962fb2a4ec7117924e2120dadc54832e58aa8003 /lib/modules/5.10.0-27-amd64/kernel/drivers/hid/hid-generic.ko
1e19617280e92bad8b648e4aec4eb793a6121ead /lib/modules/5.10.0-27-amd64/kernel/drivers/scsi/sd_mod.ko
dbdb3b617db543415e07022273fff52dbbfe5500 /lib/modules/5.10.0-27-amd64/kernel/drivers/hid/usbhid/usbhid.ko
f3beaa6ee20fbd201990cee7fd8824cb84157c36 /lib/modules/5.10.0-27-amd64/kernel/drivers/hid/hid.ko
8383080a91a3b08e96d0aff49cf2ab8827678081 /lib/modules/5.10.0-27-amd64/kernel/arch/x86/crypto/crc32-pclmul.ko
6a2f76f7f9f7e8ebbbba08cbbec8ed618cec4772 /lib/modules/5.10.0-27-amd64/kernel/arch/x86/crypto/crc32c-intel.ko
87db99c8cd6bfa2dc3b47c53d80abf55ee2c2467 /lib/modules/5.10.0-27-amd64/kernel/drivers/ata/ahci.ko
cc88cd82bb187734eba7357487f657bbff211dce /lib/modules/5.10.0-27-amd64/kernel/drivers/usb/host/xhci-pci.ko
d99c951e8b5ef01dc032f1832bfb3d56c82388fe /lib/modules/5.10.0-27-amd64/kernel/arch/x86/crypto/ghash-clmulni-intel.ko
0c9d5cebad91c1f52bf4b6cb149cd50388666274 /lib/modules/5.10.0-27-amd64/kernel/drivers/ata/libahci.ko
1dd3f298e85dc7d969a0b97c624a23cf0bf767bd /lib/modules/5.10.0-27-amd64/kernel/drivers/usb/host/xhci-hcd.ko
16480679c192e940260835fb1d0dab432a786eeb /lib/modules/5.10.0-27-amd64/kernel/drivers/net/ethernet/broadcom/tg3.ko
3ba034ac993c3993100b02e42e949f1e40b20e8e /lib/modules/5.10.0-27-amd64/kernel/drivers/scsi/megaraid/megaraid_sas.ko
1e49834d8c0dc6ce6134db25d57724942e2c97bb /lib/modules/5.10.0-27-amd64/kernel/drivers/nvme/host/nvme.ko
bbe8bca83e2e9a411011da6912cef7eaca375f84 /lib/modules/5.10.0-27-amd64/kernel/drivers/ata/libata.ko
28391411f734b5be9105684dcd9df199e8efdf2d /lib/modules/5.10.0-27-amd64/kernel/arch/x86/crypto/aesni-intel.ko
e49726471b3e3c2580f5309b934395fb953ccd9e /lib/modules/5.10.0-27-amd64/kernel/drivers/nvme/host/nvme-core.ko
1a8a31cd54f3c102b0d9f1ad594fcd5ee8234204 /lib/modules/5.10.0-27-amd64/kernel/drivers/net/phy/libphy.ko
178f48cf5df2f516a9b7b28c1bfecaf386b287d7 /lib/modules/5.10.0-27-amd64/kernel/lib/crypto/libaes.ko
99d25075b8d9444f94fedcb78f305db68ec1dce2 /lib/modules/5.10.0-27-amd64/kernel/crypto/crypto_simd.ko
baf2d3f945791e059512dba9dadf96a167c5e437 /lib/modules/5.10.0-27-amd64/kernel/block/t10-pi.ko
751c1efcca84474595539f67befbd2a6dbf70a7e /lib/modules/5.10.0-27-amd64/kernel/drivers/usb/core/usbcore.ko
f6e5adbcd506c37a4b6e48974cce425223d5a45c /lib/modules/5.10.0-27-amd64/kernel/crypto/cryptd.ko
0f2a12182a2934be20762df4b85b32f8d93588b0 /lib/modules/5.10.0-27-amd64/kernel/lib/crc-t10dif.ko
d01d7ef596efe5aec0fca98ac5fe49167b9e455d /lib/modules/5.10.0-27-amd64/kernel/arch/x86/crypto/glue_helper.ko
1be67833745b42eb45c0b3ca2e0fba8d6db46676 /lib/modules/5.10.0-27-amd64/kernel/drivers/net/ethernet/broadcom/bnxt/bnxt_en.ko
f1dddb856c863ed6c2e133a0788933bacd0c3c27 /lib/modules/5.10.0-27-amd64/kernel/crypto/crct10dif_generic.ko
c64407a36d736caba2a457f7af261f2f37db47c4 /lib/modules/5.10.0-27-amd64/kernel/drivers/ptp/ptp.ko
fffaf1648fbfde02a39e4c77afcf4d1b06d9e2f3 /lib/modules/5.10.0-27-amd64/kernel/arch/x86/crypto/crct10dif-pclmul.ko
0efecb07714551f4997269da7da459acad5d1967 /lib/modules/5.10.0-27-amd64/kernel/drivers/scsi/scsi_mod.ko
b2aff45e5c3e08695fa80213ee9d46c399f4997e /lib/modules/5.10.0-27-amd64/kernel/crypto/crct10dif_common.ko
dfd0073cf20be24d3d9cad290a791a13de437700 /lib/modules/5.10.0-27-amd64/kernel/drivers/pps/pps_core.ko
ca07eed0fab2a76321e3919fadadfe99c2f8506e /lib/modules/5.10.0-27-amd64/kernel/drivers/i2c/busses/i2c-i801.ko
42455cdcacbfebeca3d3515dbbf3916f914be69e /lib/modules/5.10.0-27-amd64/kernel/drivers/mfd/lpc_ich.ko
8d9eaef07af597fbb68662bf356179b1d34ab53c /lib/modules/5.10.0-27-amd64/kernel/drivers/i2c/i2c-smbus.ko
222fc7621b84bb6d8f8d19c12b26ff175af1a936 /lib/modules/5.10.0-27-amd64/kernel/drivers/platform/x86/wmi.ko
10fc226a1df0bfe82fed9791a6c75e9604357a3b /lib/modules/5.10.0-27-amd64/kernel/drivers/usb/common/usb-common.ko
82370d917cbeafeafdfb22160b0a477f3bd3ef6f /usr/bin/echo
1b3277a419c3fa42b199e5a170ea215b32689793 /usr/lib/x86_64-linux-gnu/ld-2.31.so
6903d030e8a7b714470e7a07e480b0039132ebb0 [vdso]
2b86a1968781038c0766b17c1ea11a2a71d7d907 /usr/lib/x86_64-linux-gnu/libc-2.31.so

while script is way more reasonable:

$ perf --buildid-dir optiwise_result/binaries script -i optiwise_result/sample/result.perf.data
echo 697827 23951235.318423:          1 cpu-cycles:uP: 
            7f49fffa2090 _start+0x0 (/usr/lib/x86_64-linux-gnu/ld-2.31.so)

echo 697827 23951235.318433:          1 cpu-cycles:uP: 
            7f49fffa2090 _start+0x0 (/usr/lib/x86_64-linux-gnu/ld-2.31.so)

echo 697827 23951235.318436:         15 cpu-cycles:uP: 
            7f49fffa2090 _start+0x0 (/usr/lib/x86_64-linux-gnu/ld-2.31.so)

echo 697827 23951235.318440:        518 cpu-cycles:uP: 
            7f49fffa2090 _start+0x0 (/usr/lib/x86_64-linux-gnu/ld-2.31.so)

echo 697827 23951235.318448:      20117 cpu-cycles:uP: 
            7f49fffa2090 _start+0x0 (/usr/lib/x86_64-linux-gnu/ld-2.31.so)

echo 697827 23951235.318473:     319878 cpu-cycles:uP: 
            7f49fffba2ea get_common_indices.constprop.0+0x1a (/usr/lib/x86_64-linux-gnu/ld-2.31.so)
        4c56415741e58948 [unknown] ([unknown])

As this is a systems research lab machine, people before me may have tweaked the /proc/… to some extent, which could explain odd perf behaviour.

I'll try #21 whenever my current run ends and let you know if it fixes the crash.

tobast commented 6 days ago

Hello,

I can confirm #21 allows the analysis to complete without error on a fresh clone of the repository on the machine from above. Thanks for the quick fix!