search

llvm / llvm-project

The LLVM Project is a collection of modular and reusable compiler and toolchain technologies.
http://llvm.org
Other
29.13k stars 12.01k forks source link

perf2bolt: cannot process perf.data - unknown error #62316

Open zamazan4ik opened 1 year ago

zamazan4ik commented 1 year ago

During the BOLT evaluation, I met an error with perf2bolt. I collected perf profile with perf record -b -p my_pid -o perf.data -- sleep 630, got a perf.data for 2.2 Gib. Then, I transform it to BOLT format with perf2bolt -nl -strict=0 -p perf.data -o perf.fdata my_binary. During the transformation process I get the following log:

PERF2BOLT: Starting data aggregation job for perf.data
PERF2BOLT: spawning perf job to read events without LBR
PERF2BOLT: spawning perf job to read mem events
PERF2BOLT: spawning perf job to read process events
PERF2BOLT: spawning perf job to read task events
BOLT-INFO: Target architecture: x86_64
BOLT-INFO: BOLT version: <unknown>
BOLT-INFO: first alloc address is 0x400000
BOLT-INFO: creating new program header table at address 0x1d400000, offset 0x1d000000
BOLT-INFO: enabling relocation mode
BOLT-INFO: enabling lite mode
BOLT-WARNING: sizes differ for function chacha20_poly1305_seal/1. FDE : 31404; symbol table : 8870. Using max size.
BOLT-INFO: using chacha20_poly1305_open_avx2/1 as another entry to function chacha20_poly1305_seal/1
BOLT-WARNING: sizes differ for function aes_hw_set_encrypt_key/1. FDE : 1221; symbol table : 1422. Using max size.
BOLT-WARNING: Failed to analyze 456 relocations
BOLT-INFO: pre-processing profile using perf data aggregator
BOLT-INFO: binary build-id is:     8c2803a9131f394fc252a5082787ed1669bac558
PERF2BOLT: spawning perf job to read buildid list
PERF2BOLT: matched build-id and file name
PERF2BOLT: waiting for perf mmap events collection to finish...
PERF-ERROR: return code 1
unknown write failure on standard output

And perf2bolt fails to proceed with my perf report. Is it a known issue? How could I fix it?

My setup:

  1. OS: Fedora 38.
  2. Linux kernel: 6.2.11-300 (mainline Fedora 38 kernel right now)
  3. CPU: Ryzen 9 5900X 12 core
  4. RAM: 48 Gib
  5. SSD: Samsung 980 Pro 2 Tib
  6. llvm-bolt, perf2bolt versions: from Fedora 38 repositories - 16.0.0
zamazan4ik commented 1 year ago

When I reduce perf profile size (to approximately 400 Mib) - the issue does not appear. I guess it's something with processing huge profiles but I am not sure.

llvmbot commented 1 year ago

@llvm/issue-subscribers-bolt

nmdis1999 commented 1 year ago

Try to run df -h (or if you want to see what's happening htop in parallel tab) in a parallel tab when you run perf2bolt, from what I know perf2bolt runs perf script on your perf.data file and since the file is huge, it ends up consuming a lot of disk space. I am not sure what might be the solution other than increasing the disk space. Something which might help is removing /tmp/perf.script-* files.

llongint commented 1 year ago

If there is insufficient memory during linking, try：

export TMPDIR=/path/to/your/tmp