Wrong callq address displayed

[jschwinger233]

bpftool prog dump seems to give wrong callq address.

---

First let's Inspect a bpf prog with bpf_redirect call

$ s bpftool p s tag 71cf8deecfc1bb77 
1125: sched_cls  name tail_ipv4_policy  tag 71cf8deecfc1bb77  gpl
    loaded_at 2023-07-11T11:08:05+0800  uid 0
    xlated 9696B  jited 5594B  memlock 12288B  map_ids 223,118,140,128,222,182,127,114,115,116,143,145
    btf_id 662

$ s bpftool p d j tag 71cf8deecfc1bb77 | grep 'redirect(' -A3
; return redirect(ifindex, flags);
15a1:   xorl    %esi, %esi
15a3:   movq    -192(%rbp), %rax
15aa:   callq   0xffffffffc377aedc

As we see the callq address is 0xffffffffc377aedc.

However that seems to be wrong, because ksym says bpf_redirect is at ffffffff85649ef0

$ s cat /proc/kallsyms | awk '$2 ~ /T/ && /bpf_redirect$/'
ffffffff85649ef0 T bpf_redirect

Let's check the opcode using bpftool p d j tag 71cf8deecfc1bb77 opcode:

$ s bpftool p d j tag 71cf8deecfc1bb77 opcode | grep 'redirect(' -A6
; return redirect(ifindex, flags);
15a1:   xorl    %esi, %esi
    31 f6 
15a3:   movq    -192(%rbp), %rax
    48 8b 85 40 ff ff ff 
15aa:   callq   0xffffffffc377aedc
    e8 2d 99 77 c3 

The opcode is e8 2d 99 77 c3, it can be explained as:

1. e8 is x64's offset-relative call.
2. offset is 2d 99 77 c3 in little endian, which is 0xc377992d in hex, or, 0xc377992d - (1<<32) = -0x3c8866d3 in 2's complement.
3. so the target address should be addressof(this_bpf_prog) + 0x15aa + 5 - 0x3c8866d3

As addressof(this_bpf_prog) can be found in ksym:

$ s cat /proc/kallsyms | grep 71cf8deecfc1bb77
ffffffffc1ecf014 t bpf_prog_71cf8deecfc1bb77_tail_ipv4_policy   [bpf]

We finally calculate the call address: 0xffffffffc1ecf014 + 0x15aa + 5 - 0x3c8866d3 = 0xffffffff85649ef0, which is exactly the bpf_redirect address showed in the ksym.

Therefore, I suspect bpftool calculates the wrong callq address.

[jschwinger233]

Sorry I missed the version info, just downloaded the latest version from release page, and issue persists.

bpftool v7.2.0
using libbpf v1.2
features: llvm, skeletons

[qmonnet]

Thanks! Could you please try and compare with the libbfd-based disassembler? I'd be curious to see whereas it does the same.

To do so you would need to compile with libbfd installed (usually shipped with binutils-dev), and to disable the llvm feature in the Makefile (Commenting FEATURE_TESTS += llvm should work).

[jschwinger233]

@qmonnet

It still showed call 0xffffffffc377aedc, which is the same wrong address.

My steps:

1. comment FEATURE_TESTS += llvm in src/Makefile
2. cd src; make

The stdout from make was:

$ make 
...                        libbfd: [ on  ]
...               clang-bpf-co-re: [ on  ]
...                          llvm: [ OFF ]
...                        libcap: [ OFF ]

3. run the same commands to check bpf_redirect call

$ s ./bpftool p d j tag 71cf8deecfc1bb77 opcode | grep 'redirect(' -A6
; return redirect(ifindex, flags);
15a1:   xor    %esi,%esi
    31 f6 
15a3:   mov    -0xc0(%rbp),%rax
    48 8b 85 40 ff ff ff 
15aa:   call   0xffffffffc377aedc
    e8 2d 99 77 c3 

[qmonnet]

OK thank you.

So this is probably not implemented in bpftool itself (bpftool doesn't directly implement disassembling), it likely comes from the disassemblers we use.

Given than both library behave the same, I suspect they have a reason to do so, but it would be nice to understand the reason behind it and to make sure this is the behaviour we want (or to adjust it otherwise). I don't know if there's something we can do about it - I need to look into this into more details (won't be before next week at least).

Thanks for the report and tests!

[qmonnet]

I think this is because we're missing the base address for the program when doing so.

Looking at your numbers:

$ python3 -c 'print(hex(0xffffffffc377992d + 5 + 0x15aa))'
0xffffffffc377aedc

We get the value printed by bpftool by taking the address in the opcodes, adding all offsets except for the base address for the program.

We should be able to retrieve this base address by parsing /proc/kallsyms in bpftool (we do it for other commands), although I'm not sure how we can plug that into the disassembler.

[qmonnet]

@jschwinger233 Hi, would you have a moment to test kernel-patches/bpf#7983 (patchwork, lore) by any chance, please?

[jschwinger233]

Working now, thank you @Asphaltt