How to transmit data over AF_XDP to another linux machine

[waqas-hafique]

I'm developing a user space application where my end goal is to send data from a Linux machine A (an embedded device) and receive on another Linux machine B (an embedded device) over AF_XDP. I want to use AF_XDP to achieve low latency.

I understand for my user space application to receive data over AF_XDP I should use XDP_REDIRECT (please correct me if I'm wrong), what I can't understand is which XDP option (XDP_REDIRECT, XDP_TX etc) should I use to transmit data from my user space application over AF_XDP?

Is there any working example in this library for my use case?

[tohojo]

There's a sample forwarding application in the bpf-examples repository: https://github.com/xdp-project/bpf-examples/tree/master/AF_XDP-forwarding

[waqas-hafique]

@tohojo Thanks,but it appears this example is for 2 sockets on same machine. what I'm looking for is send some data from Linux machine A over AF_XDP and receive on another Linux machine over AF_XDP. Both machines are say connected via eth1 interface.

[tohojo]

waqas-hafique @.***> writes:

@tohojo Thanks,but it appears this example is for 2 sockets on same machine. what I'm looking for is send some data from Linux machine A over AF_XDP and receive on another Linux machine over AF_XDP. Both machines are say connected via eth1 interface.

Ah, I see. Then you'd probably want to look at the simpler example program instead: https://github.com/xdp-project/bpf-examples/tree/master/AF_XDP-example

[waqas-hafique]

@tohojo thanks, I tried but this example does not build, there were dependencies error which I solved by installing required libs but after this I got too many syntax errors.

[waqas-hafique]

Below is error log

 make

lib

  util
make[2]: Nothing to be done for 'all'.

encap-forward
make[1]: Nothing to be done for 'all'.

ktrace-CO-RE
make[1]: Nothing to be done for 'all'.

lsm-nobpf
make[1]: Nothing to be done for 'all'.

nat64-bpf
    CLANG    nat64_kern.o
    LLC      nat64_kern.o
LLVM ERROR: Cannot select: 0x1abeb10: i64,ch = AtomicLoadOr<(load store seq_cst 8 on `i64* getelementptr inbounds (%struct.nat64_config, %struct.nat64_config* @config, i64 0, i32 2)`)> 0x1ac0a60, 0x1a7f818, Constant:i64<0>, nat64_kern.c:364:29 @[ nat64_kern.c:590:28 @[ nat64_kern.c:657:10 ] ]
  0x1a7f818: i64 = add nuw 0x1ab4870, Constant:i64<24>, nat64_kern.c:364:29 @[ nat64_kern.c:590:28 @[ nat64_kern.c:657:10 ] ]
    0x1ab4870: i64 = BPFISD::Wrapper TargetGlobalAddress:i64<%struct.nat64_config* @config> 0, nat64_kern.c:357:32 @[ nat64_kern.c:590:28 @[ nat64_kern.c:657:10 ] ]
      0x1ac0928: i64 = TargetGlobalAddress<%struct.nat64_config* @config> 0, nat64_kern.c:357:32 @[ nat64_kern.c:590:28 @[ nat64_kern.c:657:10 ] ]
    0x1a7e9e0: i64 = Constant<24>
  0x1ac0310: i64 = Constant<0>
In function: nat64_handler
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace.
Stack dump:
0.  Program arguments: llc -march=bpf -filetype=obj -o nat64_kern.o nat64_kern.ll 
1.  Running pass 'Function Pass Manager' on module 'nat64_kern.ll'.
2.  Running pass 'BPF DAG->DAG Pattern Instruction Selection' on function '@nat64_handler'
 #0 0x00007fb75e7d0f8f llvm::sys::PrintStackTrace(llvm::raw_ostream&) (/lib/x86_64-linux-gnu/libLLVM-11.so.1+0xbd0f8f)
 #1 0x00007fb75e7cf2f0 llvm::sys::RunSignalHandlers() (/lib/x86_64-linux-gnu/libLLVM-11.so.1+0xbcf2f0)
 #2 0x00007fb75e7d1465 (/lib/x86_64-linux-gnu/libLLVM-11.so.1+0xbd1465)
 #3 0x00007fb762dd0140 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x13140)
 #4 0x00007fb75d752ce1 raise (/lib/x86_64-linux-gnu/libc.so.6+0x38ce1)
 #5 0x00007fb75d73c537 abort (/lib/x86_64-linux-gnu/libc.so.6+0x22537)
 #6 0x00007fb75e720d68 (/lib/x86_64-linux-gnu/libLLVM-11.so.1+0xb20d68)
 #7 0x00007fb75e720d87 (/lib/x86_64-linux-gnu/libLLVM-11.so.1+0xb20d87)
 #8 0x00007fb75ee3c2f2 (/lib/x86_64-linux-gnu/libLLVM-11.so.1+0x123c2f2)
 #9 0x00007fb75ee3b9c5 llvm::SelectionDAGISel::SelectCodeCommon(llvm::SDNode*, unsigned char const*, unsigned int) (/lib/x86_64-linux-gnu/libLLVM-11.so.1+0x123b9c5)
#10 0x00007fb760265573 (/lib/x86_64-linux-gnu/libLLVM-11.so.1+0x2665573)
#11 0x00007fb75ee350bf llvm::SelectionDAGISel::DoInstructionSelection() (/lib/x86_64-linux-gnu/libLLVM-11.so.1+0x12350bf)
#12 0x00007fb75ee34730 llvm::SelectionDAGISel::CodeGenAndEmitDAG() (/lib/x86_64-linux-gnu/libLLVM-11.so.1+0x1234730)
#13 0x00007fb75ee33a52 llvm::SelectionDAGISel::SelectAllBasicBlocks(llvm::Function const&) (/lib/x86_64-linux-gnu/libLLVM-11.so.1+0x1233a52)
#14 0x00007fb75ee31751 llvm::SelectionDAGISel::runOnMachineFunction(llvm::MachineFunction&) (/lib/x86_64-linux-gnu/libLLVM-11.so.1+0x1231751)
#15 0x00007fb75eaa41ae llvm::MachineFunctionPass::runOnFunction(llvm::Function&) (/lib/x86_64-linux-gnu/libLLVM-11.so.1+0xea41ae)
#16 0x00007fb75e8e0059 llvm::FPPassManager::runOnFunction(llvm::Function&) (/lib/x86_64-linux-gnu/libLLVM-11.so.1+0xce0059)
#17 0x00007fb75e8e5603 llvm::FPPassManager::runOnModule(llvm::Module&) (/lib/x86_64-linux-gnu/libLLVM-11.so.1+0xce5603)
#18 0x00007fb75e8e0670 llvm::legacy::PassManagerImpl::run(llvm::Module&) (/lib/x86_64-linux-gnu/libLLVM-11.so.1+0xce0670)
#19 0x000000000040e0df main (/usr/lib/llvm-11/bin/llc+0x40e0df)
#20 0x00007fb75d73dd0a __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x23d0a)
#21 0x000000000040ba2a _start (/usr/lib/llvm-11/bin/llc+0x40ba2a)
Aborted
make[1]: *** [../lib/common.mk:88: nat64_kern.o] Error 134
make: *** [Makefile:37: nat64-bpf] Error 2

[waqas-hafique]

It seems I can't build the whole repo "bpf-examples" but I can build the specific project "AF_XDP-example" .

[tohojo]

waqas-hafique @.***> writes:

It seems I can't build the whole repo "bpf-examples" but I can build the specific project "AF_XDP-example" .

Right. I think you may just need a newer version of LLVM to build the other examples...

[waqas-hafique]

@tohojo thanks, will try a new LLVM to build other programs. However for "AF_XDP-example" which I was able to build , when I run this program I get "argument list too long" error.

My kernel is "5.4.143-rt63-intel-ese-standard-lts-rt".

taskset -c 0 ./xdpsock -i eth1 -q 1 -t -N 
libbpf: elf: skipping unrecogniz[   77.412232] 000: stmmaceth 0000:00:1d.2 eth1: Register XDP MEM_TYPE_ZERO_COPY RxQ-1
ed data section([   77.412238] 000: stmmaceth 0000:00:1d.2 eth1: Failed to alloc Rx UMEM at Q-1
7) .xdp_run_config
libbpf: elf: skipping unrecognized data section(8) xdp_metadata
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libbpf: elf: skipping unrecognized data section(7) xdp_metadata
libbpf: prog 'xdp_dispatcher': BPF program load failed: Argument list too long
libbpf: prog 'xdp_dispatcher': failed to load: -7
libbpf: failed to load object 'xdp-dispatcher.o'
libxdp: Compatibility check for dispatcher program failed: Argument list too long
libxdp: Falling back to loading single prog without dispatcher

 sock0@eth1:1 txonly xdp-drv 
                   pps            pkts           1.01          
rx                 0              0             
tx                 6011           6080          

 sock0@eth1:1 txonly xdp-drv 
                   pps            pkts           1.00          
rx                 0              0             
tx                 0              6080          
^C
 sock0@eth1:1 txonly xdp-drv 
                   pps            pkts           1.00          
rx                 0              0             
tx                 0              6080 

[tohojo]

waqas-hafique @.***> writes:

thanks, will try a new LLVM to build other programs. However for "AF_XDP-example" which I was able to build , when I run this program I get "argument list too long error.

That's because your kernel is too old to support that feature, but as you can see from the line below, libxdp will just fall back to a different operating mode and things still work...

[waqas-hafique]

@tohojo it seems it does not work because as you see pps is 0, also I can't see any data on the 2nd linux machine connected via eth1 where I'm using following command with -r or -l option

./xdpsock -i eth1 -q 1 -r -N
./xdpsock -i eth1 -q 1 -l -N

I was thinking to use the same example "xdpsock.c" provided with my kernel version 5.4.143.

[tohojo]

waqas-hafique @.***> writes:

@tohojo it seems it does not work because as you see pps is 0, also I can't see any data on the 2nd linux machine connected via eth1 where I'm using following command with -r or -l option

./xdpsock -i eth1 -q 1 -r -N
./xdpsock -i eth1 -q 1 -l -N

I was thinking to use the same example "xdpsock.c" provided with my kernel version 5.4.143.

Well, you also get this error from somewhere?

000: stmmaceth 0000:00:1d.2 eth1: Failed to alloc Rx UMEM at Q-17

The stmmac driver didn't gain XDP support until kernel 5.13, though, so I'm not sure how it's even trying. Is this a patched kernel?

[waqas-hafique]

@tohojo yes I do face the error you mentioned as I asked here

https://github.com/xdp-project/xdp-tutorial/issues/343

so you mean the stmmac driver didn't gain XDP support until kernel 5.13 but my kernel is newer than this I mean 5.4.143. Yes its a patched kernel from Intel, I used this to evaluate TSN technology provided by Intel from following repo

https://github.com/intel/iotg_tsn_ref_sw

In this repo , I can run AF_XDP but now I'm trying to use a generic example without TSN.

[tohojo]

waqas-hafique @.***> writes:

@tohojo yes I do face the error you mentioned as I asked here

https://github.com/xdp-project/xdp-tutorial/issues/343

so you mean the stmmac driver didn't gain XDP support until kernel 5.13 but my kernel is newer than this I mean 5.4.143.

5.4.143 is not newer than 5.13. The stable updates for 5.4 does not include new features (like XDP support in a driver).

Yes its a patched kernel from Intel, I used this to evaluate TSN technology provided by Intel from following repo

Well, if you're using a custom kernel, I'd suggest asking the vendor about it. Or maybe @magnus-karlsson has an idea, since it's Intel stuff?

[magnus-karlsson]

waqas-hafique @.***> writes: @tohojo yes I do face the error you mentioned as I asked here #343 so you mean the stmmac driver didn't gain XDP support until kernel 5.13 but my kernel is newer than this I mean 5.4.143. 5.4.143 is not newer than 5.13. The stable updates for 5.4 does not include new features (like XDP support in a driver). Yes its a patched kernel from Intel, I used this to evaluate TSN technology provided by Intel from following repo Well, if you're using a custom kernel, I'd suggest asking the vendor about it. Or maybe @magnus-karlsson has an idea, since it's Intel stuff?

I do not know what this custom kernel is. Intel is a huge company :-). The easiest way is likely for you to contact the maintainers of that repo. They should be able to help you.

Maybe just start with the latest bpf-next or stable Linux kernel instead and see if that works for you?