[BOLT][AArch64] 64k page support in DataAggregator (no-LBR perf samples)

[aaupov]

Host: Ubuntu 22.04, Ampere Altra A1 on Oracle Cloud Sampled clang-17 bootstrapped binary built from recent trunk using aaupov/llvm-project/nolbr and aaupov/llvm-devmtg-2022/altra.

perf2bolt fails with

PERF2BOLT-WARNING: unable to find base address of the binary when memory mapped at 0xaaaaac8c4000 using file offset 0x1e14000. Ignoring profile data for this mapping
PERF2BOLT-ERROR: could not find a profile matching binary "/tmp/tmp.E1ksjF8aZ3/BOLT_BASELINE/tools/clang/stage2-bins/bin/clang-17". Profile for the following binary name(s) is available:
  clang-17
...

The warning hints to the source of the issue: https://github.com/llvm/llvm-project/blob/b884f4ef0a2de3d0f24111411dff663fd68c2eb0/bolt/lib/Profile/DataAggregator.cpp#L2055-L2063

I sprinkled printf debug statements to look at the calculation done in getBaseAddressForMapping: https://github.com/llvm/llvm-project/blob/b884f4ef0a2de3d0f24111411dff663fd68c2eb0/bolt/lib/Core/BinaryContext.cpp#L1871-L1880

0x1e14000 is the FileOffset
the following lines show SegInfo.FileOffset, SegInfo.Alignment, and alignDown(SegInfo.FileOffset, SegInfo.Alignment)
offset 0x0, align 0x10000 => 0x0 == 0x1e14000
offset 0x1e14240, align 0x10000 => 0x1e10000 == 0x1e14000
offset 0x5877c60, align 0x10000 => 0x5870000 == 0x1e14000
offset 0x5c59ad8, align 0x10000 => 0x5c50000 == 0x1e14000

Checking ELF segments:

Program Headers:
  Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
  LOAD           0x1e14240 0x0000000001e24240 0x0000000001e24240 0x3a63a20 0x3a63a20 R E 0x10000

and perf script --show-mmap-events

        clang-17 179425 160317.159883: PERF_RECORD_MMAP2 179425/179425: [0xaaaaac8c4000(0x3a64000) @ 0x1e14000 08:01 2411305 3556563589]: r-xp /tmp/tmp.E1ksjF8aZ3/BOLT_BASELINE/tools/clang/stage2-bins/bin/clang-17

Clearly, we use an incorrect calculation for finding the base address, since Align doesn't necessarily imply an alignment of mmapped address. ELF spec doesn't mandate that:

[p_align] integral power of 2, and p_vaddr should equal p_offset, modulo p_align.

[llvmbot]

@llvm/issue-subscribers-bolt

[aaupov]

Should be resolved by https://reviews.llvm.org/D144588

UPD: it's not.

[onroadmuwl]

I happen to solve this problem by rebuilding the Linux kernel, which lets the perf2bolt work well on the Ampere processor and ubuntu20.04 system. I am not sure if this solution applies to everyone, but you can check the dmesg information during machine reboot. Before rebuilding the kernel, I got some information like "firmware bug kernel image not aligned on 64k boundary".

[aaupov]

Quick update: turns out that the host has 4k pages, which explains the mmapped address:

0xaaaaac8c4000(0x3a64000) @ 0x1e14000

(aligned by 0x1000 which is 4k).

So the proper solution is to align down not by segment alignment, which must be a multiple of page size, but rather page size itself. When aligned by 4k, this particular issue is resolved.

The question is how to extract the page size used while perf sampling: 1) Can we assume that perf2bolt works on the same host as was used to collect perf file? In this case getpagesize() would work. 2) If not, perf can capture page size of executed ip (PERF_SAMPLE_CODE_PAGE_SIZE) with --code-page-size perf record option:

   --code-page-size
      Record the sampled code address (ip) page size

which can then be extracted from each sample. Not the most efficient way, and requires an extra option, but it'll work.

I tried finding the page size used for mapping in PERF_RECORD_MMAP2 but there's none:

struct {
  struct perf_event_header header;
  u32    pid;
  u32    tid;
  u64    addr;
  u64    len;
  u64    pgoff;
  union {
      struct {
          u32    maj;
          u32    min;
          u64    ino;
          u64    ino_generation;
      };
      struct {   /* if PERF_RECORD_MISC_MMAP_BUILD_ID */
          u8     build_id_size;
          u8     __reserved_1;
          u16    __reserved_2;
          u8     build_id[20];
      };
  };
  u32    prot;
  u32    flags;
  char   filename[];
  struct sample_id sample_id;
};