search

Igalia / snabb

Snabb Switch: Fast open source packet processing
Apache License 2.0
47 stars 5 forks source link

snabb config load doesn't release hugepages #1192

Open mwiget opened 5 years ago

mwiget commented 5 years ago

Running snabb config load consumes 16 x 1GB hugepages during the process but doesn't free them when finished. Running snabb lwaftr compile-configuration also consumes hugepages but they are properly released:

mwiget@xeon:~/vmx-docker-lwaftr$ cat /proc/meminfo |grep Huge
AnonHugePages:   2097152 kB
ShmemHugePages:        0 kB
HugePages_Total:      32
HugePages_Free:       27
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB
mwiget@xeon:~/vmx-docker-lwaftr$ sudo snabb/src/snabb config load -s snabb-softwire-v2 lwaftr snabb.conf.large
snabb.conf.large: loading source configuration
snabb.conf.large: wrote compiled configuration snabb.conf.large.o
mwiget@xeon:~/vmx-docker-lwaftr$ cat /proc/meminfo |grep Huge
AnonHugePages:   2097152 kB
ShmemHugePages:        0 kB
HugePages_Total:      32
HugePages_Free:       11
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB

Now running compile-configuration:

mwiget@xeon:~/vmx-docker-lwaftr$ !cat
cat /proc/meminfo |grep Huge
AnonHugePages:   2097152 kB
ShmemHugePages:        0 kB
HugePages_Total:      32
HugePages_Free:       11
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB
mwiget@xeon:~/vmx-docker-lwaftr$ rm snabb.conf.large.o
rm: remove write-protected regular file 'snabb.conf.large.o'? y
mwiget@xeon:~/vmx-docker-lwaftr$ sudo snabb/src/snabb lwaftr compile-configuration snabb.conf.large
snabb.conf.large: loading source configuration
snabb.conf.large: wrote compiled configuration snabb.conf.large.o
mwiget@xeon:~/vmx-docker-lwaftr$
mwiget@xeon:~/vmx-docker-lwaftr$ cat /proc/meminfo |grep Huge
AnonHugePages:   2097152 kB
ShmemHugePages:        0 kB
HugePages_Total:      32
HugePages_Free:       11
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB

Running snabb version with this latest commit:

commit 9bbbb09870ed6fd162739305ef149bb5a89460ee (tag: v2018.09.01, igalia/lwaftr)

I do see Mem:grow_buffer() being called many times (lib/stream/mem.lua) asking for 1024 bytes, which is handled by the func new_buffer(), which is in fact calling ffi.new('uint8_t[?]', len).

mwiget commented 5 years ago

Hmm, not clear the issue is within config load. After trying to load a new config a few times, I ran out of hugepages and get the following errors from the running snabb instances being reconfigured:

hugetlb mmap failed (Cannot allocate memory), falling back.
hugetlb mmap failed (Cannot allocate memory), falling back.
hugetlb mmap failed (Cannot allocate memory), falling back.
hugetlb mmap failed (Cannot allocate memory), falling back.
mwiget commented 5 years ago

Yeah, the hugepages are released once the target snabb process and workers have terminated. This makes be believe, the issue is within the target snabb process or my false understanding on how to use snabb config ;-).