search

xdp-project / bpf-examples

Making eBPF programming easier via build env and examples
422 stars 84 forks source link

Sharing UMEM with two netdevs and creating multiple sockets for a single queue (for a netdev) #85

Open arukshani opened 1 year ago

arukshani commented 1 year ago

Hi,

I have a use case where packet forwarding is done between veth and NIC. I have two sockets. One for the veth queue and one for the NIC queue and the UMEM is shared between these two sockets so each socket has its own fill ring and completion ring. The throughput we got for this scenario is ~7Gbps.

To increase the throughput we thought to create multiple sockets for both veth queue and NIC queue. For 'xsk_socket__create_shared' function when I pass the same fill ring and completion ring for the NIC queue, it throws an error. I just want to know when the UMEM is shared between two netdevs, can we also create multiple sockets for each netdev?

Thanks!

magnus-karlsson commented 1 year ago

So you are sharing an umem between netdev1,qid1 and netdev2,qid2 and you would like to bind more sockets to for example netdev1,qid1? This in contrast to binding further sockets to netdev1,qid2, netdev1,qid3, and so on.

magnus-karlsson commented 1 year ago

When looking at the code, it is slightly weird. If you create socket1 reg a umem and bind it to netdev1,qid1 first, then it is possible to bind further sockets to netdev1/qid1 (all sharing the same fill/comp ring pair). You can also bind a new socket sharing the same umem to netdev2/qid2, but it is not possible binding further sockets to netdev2/qid2.

In principle, it should be possible to bind more sockets to netdev2/qid2 as long as they share the same fill and completion ring as the first socket that was bound to netdev2/qid2. But the code does not support this today.

Please try creating and binding more sockets on netdev1/qid1, or do you need to bind more sockets to both netdev1 and netdev2?

arukshani commented 1 year ago

I need to bind more sockets to both netdev1/q1 and netdev2/q2. As you mentioned it allowed me to create multiple sockets for netdev1/q1 but only one socket for netdev2/q2. Is there a particular reason for this restriction?

Thank you.

magnus-karlsson commented 1 year ago

No particular reason. Just that I never thought of the case and no one, up until now, has asked for it. After I have released the multi-buffer feature for AF_XDP, I will take a look at this and see if what it would take.