eBPF verifier, eBPF maps, and struct keys.

[tjcw]

I have put aside my attempt to inject packets into the kernel from userspace for the time being, and I working on the next stage. I am trying to set up my test case so that the first frame of a TCP flow gets pushed to user space, the user space code updates an eBPF map and discards the frame, and the client will timeout and resend the first frame which will then be passed to the kernel with XDP_PASS.

However when I try loading the eBPF, I get a verifier failure which I do not understand. My test case is under https://github.com/tjcw/xdp-tutorial/tree/master/ebpf-filter-snapshot-20221018 ; when I run it (the eBPF code has to be copied to libxdp and built there, and then you run the test case with "sudo ./run.sh") I get

44: (6b) *(u16 *)(r10 -16) = r2
45: (69) r1 = *(u16 *)(r1 +2)
46: (05) goto pc+2
; 
49: (6b) *(u16 *)(r10 -14) = r1
50: (bf) r2 = r10
51: (07) r2 += -24
52: (18) r1 = 0xffff95ae4bb3bc00
54: (85) call bpf_map_lookup_elem#1
invalid indirect read from stack R2 off -24+13 size 16
processed 73 insns (limit 1000000) max_states_per_insn 0 total_states 7 peak_states 7 mark_read 4
-- END PROG LOAD LOG --
libbpf: prog 'xsk_def_prog': failed to load: -13
libbpf: failed to load object '/usr/local/lib/bpf/xsk_def_xdp_prog.o'
libxdp: Failed to load program xsk_def_prog: Permission denied
xsk_socket__create returns -13
ERROR: Cannot set up socket 0
ERROR: Can't setup AF_XDP sockets "Permission denied"
[tjcw@r28b29-n10](mailto:tjcw@r28b29-n10):~/workspace/xdp-tutorial/ebpf-filter$ 

Looking at the objdump of the eBPF shows

      55: 61 02 00 00 00 00 00 00 r2 = *(u32 *)(r0 + 0)
;  if ( action == XDP_REDIRECT) {
      56: 55 02 1e 00 04 00 00 00 if r2 != 4 goto +30 <LBB0_18>
      57: b7 01 00 00 04 00 00 00 r1 = 4
      58: 63 1a fc ff 00 00 00 00 *(u32 *)(r10 - 4) = r1
      59: bf a2 00 00 00 00 00 00 r2 = r10
      60: 07 02 00 00 fc ff ff ff r2 += -4
;  struct datarec *rec = bpf_map_lookup_elem(&xdp_stats_map, &action);
      61: 18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll
      63: 85 00 00 00 01 00 00 00 call 1
;  if (!rec)
      64: 15 00 0b 00 00 00 00 00 if r0 == 0 goto +11 <LBB0_17>

which I think is the region surrounding the verifier failure, and seems to be an indication that

void * v_permit=bpf_map_lookup_elem(&accept_map, &f) ;

is trying to access memory out of bounds. But I do not know if the problem is with the accept_map or with f, and I don't think either should give a problem. f is a 'five-tuple' struct of source IP, destination IP, source port, destination port, and protocol.

Can you throw any light on what is broken in my test case ? @tohojo @magnus-karlsson

[tjcw]

I think I see a couple of problems. The 'fivetuple' was a struct of 13 bytes; when I added padding to make this 16 bytes the verifier got futher. The next problem seems to be that I am unconditonally dereferencing a pointer which may be NULL. I will change this and report what happens then.

[tjcw]

Yes, with that fix I can load my eBPF. Closing.