advanced03-AF_XDP -- Can't create umem "Cannot allocate memory"

[ferrieux]

Hello, Simply running af_xdp_user on an updated Stretch (4.19.0-5-amd64):

 # grep XDP /boot/config-4.19.0-5-amd64 
 CONFIG_XDP_SOCKETS=y
 # ./af_xdp_user -d eth20
 ERROR: Can't create umem "Cannot allocate memory"

A bit of gdb shows this happens in xsk_umem_create() :

   map = mmap(NULL, off.cr.desc + umem->config.comp_size * sizeof(__u64),
               PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE, umem->fd,
               XDP_UMEM_PGOFF_COMPLETION_RING);

The mmap fails because off.cr.desc is a ridiculous number:

(gdb) print off $2 = {rx = {producer = 0, consumer = 64, desc = 128, flags = 0}, tx = {producer = 64, consumer = 128, desc = 0, flags = 64}, fr = {producer = 128, consumer = 0, desc = 64, flags = 128}, cr = {producer = 112, consumer = 134112, desc = 140737353639664, flags = 93824992375840}}

[tohojo]

You'll need to update your kernel to use AF_XDP. At least kernel 5.2, but there's a bug in libbpf for older kernels so I'll recommend you go for 5.3...

On Tuesday, September 10, 2019, Alexandre Ferrieux notifications@github.com wrote:

Hello, Simply running af_xdp_user on an updated Stretch (4.19.0-5-amd64):

grep XDP /boot/config-4.19.0-5-amd64

CONFIG_XDP_SOCKETS=y

./af_xdp_user -d eth20

ERROR: Can't create umem "Cannot allocate memory"

A bit of gdb shows this happens in xsk_umem_create() :

map = mmap(NULL, off.cr.desc + umem->config.comp_size * sizeof(__u64), PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE, umem->fd, XDP_UMEM_PGOFF_COMPLETION_RING);

The mmap fails because off.cr.desc is a ridiculous number:

(gdb) print off $2 = {rx = {producer = 0, consumer = 64, desc = 128, flags = 0}, tx = {producer = 64, consumer = 128, desc = 0, flags = 64}, fr = {producer = 128, consumer = 0, desc = 64, flags = 128}, cr = {producer = 112, consumer = 134112, desc = 140737353639664, flags = 93824992375840}}

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/xdp-project/xdp-tutorial/issues/76?email_source=notifications&email_token=AADA24I7EFYZUJ5LAVTC2KDQI62TBA5CNFSM4IVJBXUKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HKPKEOQ, or mute the thread https://github.com/notifications/unsubscribe-auth/AADA24MACBHB2XQ2STJLQL3QI62TBANCNFSM4IVJBXUA .

[ferrieux]

Ah OK thanks. Did I miss that restriction in a README ? Better yet, what about a version check at runtime ?

[tohojo]

Alexandre Ferrieux notifications@github.com writes:

Ah OK thanks. Did I miss that restriction in a README ? Better yet, what about a version check at runtime ?

No, we don't mention that; the AF_XDP lesson is fairly new...

@chaudron care to add something to the README about this?

[chaudron]

Yes will update the readme, and also add some run-time checks... Will try to do it early next week.

[ferrieux]

Thanks guys ! Just to report, being constrained by the distro (Debian Buster), despite your advice to go for 5.3 I made it work on 5.2; for this I had to tweak the makefiles to use the libbpf from the distro (in buster-backports), because the git submodule libbpf assumed later versions of if_xdp.h (for example, a fourth "flags" field in xdp_ring_offset, not yet present in 5.2, causing the error I reported in this issue). Naturally this leads to ask: is there a strong reason to depend on bleeding-edge libbpf (instead of the one shipped with the running kernel), given that this implies that your minimum kernel version will keep increasing, following this moving target ?

[vcunat]

Kernel versions: things seem OK-ish for me even on 5.0, but it appears a bit "fragile" in terms of libbpf version and there are missing features. I'm a bit wary of restricting deployments to bleeding-edge kernels.

[PhilippHomann]

I'm experiencing the same problem. I'm on Fedora 30 but with kernel 5.3.0-0.rc7.git0.1.fc32.x86_64.

$ grep XDP /boot/config-5.3.0-0.rc7.git0.1.fc32.x86_64 
CONFIG_XDP_SOCKETS=y
CONFIG_XDP_SOCKETS_DIAG=m

(gdb) print off
$2 = {rx = {producer = 0, consumer = 64, desc = 128, flags = 0}, tx = {producer = 64, consumer = 128, desc = 0, flags = 64}, fr = {producer = 128, consumer = 0, desc = 64, flags = 128}, cr = {producer = 140737353518176, consumer = 112, 
    desc = 140737353537040, flags = 4334624}}

[vcunat]

@PhilippHomann: as described above, that should be due to using a wrong (buggy) libbpf version. EDIT: I suppose it's generally the safest bet to use the one extracted from the kernel source of the version you want to use.

[PhilippHomann]

@vcunat: Indeed you are right.. My problem was that I've replaced LIBBPF_DIR in common.mk instead of in the Makefile of advanced03.. my bad..

[tohojo]

Alexandre Ferrieux notifications@github.com writes:

Just to report, being constrained by the distro (Debian Buster), despite your advice to go for 5.3 I made it work on 5.2; for this I had to tweak the makefiles to use the libbpf from the distro (in buster-backports), because the git submodule libbpf assumed later versions of if_xdp.h (for example, a fourth "flags" field in xdp_ring_offset, not yet present in 5.2, causing the error I reported in this issue).

Naturally this leads to ask: is there a strong reason to depend on bleeding-edge libbpf (instead of the one shipped with the running kernel), given that this implies that your minimum kernel version will keep increasing, following this moving target ?

Well, libbpf is supposed to be backwards compatible with older kernels; but in this case there's a bug in that compatibility. See https://patchwork.ozlabs.org/patch/1159891/ and #75.

No matter what we do, this is probably going to be a bit brittle until upstream stabilises: this is quite new technology. However, we'll obviously trying to do our best to limit the incompatibility. For instance, we should probably be supporting using the distro version of libbpf in the Makefile. Care to send a pull request for that?

[ferrieux]

Yes, will do :) But is there a reason to use the upstream libbpf by default ? Do you depend on some new feature or bugfix ? If you're to add a version check, and include the proper threshold on libbpf too, you can just list it in the prereqs like libelf, say "-lbpf" in the Makefile, and a whole family of beginner issues will vanish ;)

[tohojo]

Alexandre Ferrieux notifications@github.com writes:

Yes, will do :)

But is there a reason to use the upstream libbpf by default ? Do you depend on some new feature or bugfix ? If you're to add a version check, and include the proper threshold on libbpf too, you can just list it in the prereqs like libelf, say "-lbpf" in the Makefile, and a whole family of beginner issues will vanish ;)

libbpf is quite new, so it's not packaged by all distributions yet. We're working on this upstream (making libbpf easier to package, etc); hopefully we'll get to the point where we no longer need to use the git submodules.

And yeah, we do also occasionally use quite new features (AF_XDP is quite new, for example).