search

michalpolkowski / alt-f

Automatically exported from code.google.com/p/alt-f
0 stars 0 forks source link

Jumbo frames make linux start swaping #18

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Jumbo frames greater that 3700 bytes can make linux start swaping, effectively 
reducing network performance.

Low level memory allocation in linux happens in blocks of 4096 contiguos bytes, 
and in powers of two.

A frame size greater than 3700 bytes, plus some memory management overheads, 
exceeds the 4KB page, thus asking linux to allocate a new pool of 130 blocks of 
8196 continuous bytes (that is what the network driver asks for, after a 
"ifconfig eth0 mtu xxx" , xxx >3700 happens, cat /proc/slabinfo)

If this happens when memory is fragmented, some pages have to be swaped to disk.

Filesystems also use block of 4096 bytes, and contention between pages 
requested by the filesystem and the network subsystems arise.

This is more likely to happen on Gbit networks, where frames show-up at a fast 
pace, and after the system has been running for a while.

This needs more research.

Original issue reported on code.google.com by whoami.j...@gmail.com on 29 Nov 2010 at 10:40

GoogleCodeExporter commented 9 years ago 
This is an issue only if the MTU is changed after the unity has been working 
after a while, with several services started.

The recommended procedure is to change the MTU right after boot; if changing it 
much latter, with values higher than 3700 bytes, the box must be rebooted.

Why is this? for values higher than 3700 bytes, two *contiguous* 4096 bytes 
buffers must be obtained for each packet by the slab allocator from the buddy 
allocator, and the network driver tries to allocate 130 of them. If memory is 
fragmented, swapping occurs continuously.

For values lower than 3700 bytes there is no problem, as the slab allocator can 
use already allocated 4096 pages and ask the buddy allocator for more 4096 
pages. As they don't need to be contiguous, there is no problem.

Anyway, a MTU greater that +/-7500 bytes asks for *3* contiguous 4096 pages, 
and most of these memory is wasted anyway: max MTU=9000 bytes, allocated 
3*4096=12288 bytes, gives 3288 unused bytes, times 130 reserved slots by the 
driver totals 427440 of unused and precious memory!

Original comment by whoami.j...@gmail.com on 7 Jan 2011 at 3:53