chengguozhen / netmap

Automatically exported from code.google.com/p/netmap
0 stars 0 forks source link
    Netmap - a framework for fast packet I/O
VALE -  a Virtual Local Ethernet using the netmap API

========================================================================

NETMAP is a framework for very fast packet I/O from userspace. VALE is an equally fast in-kernel software switch using the netmap API. Both are implemented as a single kernel module for FreeBSD and Linux, and can deal with line rate on real or emulated 10 Gbit ports. See details at

http://info.iet.unipi.it/~luigi/netmap/

In this directory you can find source code (BSD-Copyright) for FreeBSD and Linux. Note that recent FreeBSD distributions already include both NETMAP and VALE.

For more details please look at the manpage (netmap.4) and netmap home page above.

What is this good for

Netmap is mostly useful for userspace applications that must deal with raw packets: traffic generators, sinks, monitors, loggers, software switches and routers, generic middleboxes, interconnection of virtual machines.

In this distribution you will find some example userspace code to build a generator, a sink, and a simple bridge. The kernel module implements a learning ethernet bridge. We also include patches for some applications (noticeably libpcap) so you can run any libpcap client on top of netmap hopefully at a higher speed.

Netmap alone DOES NOT accelerate your TCP. For that you need to implement your own tcp/ip stack probably using some of the techniques indicated below to reduce the processing costs.

Architecture

netmap uses a number of techniques to establish a fast and efficient path between applications and the network. In order of importance:

1. I/O batching
2. efficient device drivers
3. pre-allocated tx/rx buffers
4. memory mapped buffers

Despite the name, memory mapping is NOT the key feature for netmap's speed; systems that do not apply all these techniques do not achieve the same speed and efficiency.

Netmap clients use a select()-able file descriptor to synchronize with the network card/software switch, and exchange multiple packets per system call through device-independent memory mapped buffers and descriptors. Device drivers are completely in the kernel, and the system does not rely on IOMMU or other special mechanisms.

Installation instructions

A kernel module (netmap.ko or netmap_lin.ko) implements the core NETMAP routines and the VALE switch. Netmap-aware device drivers are needed to use netmap on ethernet ports. To date, we have support for Intel ixgbe (10G), e1000/e1000e/igb (1G), Realtek 8169 (1G) and Nvidia (1G).

If you do not have a supported device, you can still try out netmap (with reduced performance) because the main kernel module emulates the netmap API on top of standard device drivers.

FreeBSD instructions:

Since recent FreeBSD distributions already include netmap, you only need build the new kernel or modules as below:

Applications

The directory examples/ contains some programs that use the netmap API

pkt-gen.c   a packet generator/receiver working at line rate at 10Gbit/s
vale-cfg.c  utility to configure ports of a VALE switch
bridge.c    a utility that bridges two interfaces or one interface
    with the host stack

For libpcap and other applications look at the extra/ directory.

Testing

pkt-gen is a generic test program which can act as a sender or receiver. It has a large number of options, but the simplest form is:

pkt-gen -i ix0 -f rx    # receive and print stats
pkt-gen -i ix0 -f tx -l 60  # send a stream of 60-byte packets

(replace ix0 with the name of the interface or VALE port). This should be able to work at line rate (up to 14.88 Mpps on 10 Gbit/interfaces, even higher on VALE) but note the following

OPERATING SPEED

Netmap is able to send packets at very high rates, and for simple packet transmission and reception, speed generally not limited by the CPU but by other factors (link speed, bus or NIC hw limitations).

For a physical link, the maximum numer of packets per second can be computed with the formula:

pps = line_rate / (672 + 8 * pkt_size)

where "line_rate" is the nominal link rate (e.g 10 Gbit/s) and pkt_size is the actual packet size including MAC headers and CRC. The following table summarizes some results

        LINE RATE
pkt_size \  100M    1G  10G 40G

  64    .1488   1.488   14.88   59.52
 128    .0589   0.589    5.89   23.58
 256    .0367   0.367    3.67   14.70
 512    .0209   0.209    2.09    8.38
1024    .0113   0.113    1.13    4.51
1518    .0078   0.078    0.78    3.12

On VALE ports, there is no physical link and the throughput is limited by CPU or memory depending on the packet size.

COMMON PROBLEMS

Before reporting slow send or receive speed on a physical interface, check ALL of the following:

CANNOT SET THE DEVICE IN NETMAP MODE:

SENDER DOES NOT TRANSMIT

RECEIVER DOES NOT RECEIVE

LOWER SPEED THAN LINE RATE

Credits

NETMAP and VALE are projects of the Universita` di Pisa, partially supported by various entities including: Intel Research Berkeley, EU FP7 projects CHANGE and OPENLAB, Netapp/Silicon Valley Community Foundation, ICSI

Author: Luigi Rizzo Contributors: Giuseppe Lettieri Michio Honda Marta Carbone Gaetano Catalli Matteo Landi Vincenzo Maffione

References

There are a few academic papers describing netmap, VALE and applications. You can find the papers at http://info.iet.unipi.it/~luigi/research.html