search

cscrj / netmap

Automatically exported from code.google.com/p/netmap
0 stars 0 forks source link

Two instances of netmap not working on tcpreplay #16

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
1. compile tcpreplay with netmap capability
2. compile tcpdump with netmap libpcap support
3. try to run both simultaneously

What is the expected output? What do you see instead?
both should work simultaneously

What version of the product are you using? On what operating system?
tcpreplay goes to 100% CPU

Please provide any additional information below.
I am maintainer of Tcpreplay, but have not been able to resolve this. Issue was 
reported here .. https://github.com/appneta/tcpreplay/issues/120. It seems 
similar to https://github.com/appneta/tcpreplay/issues/79

Original issue reported on code.google.com by fklas...@appneta.com on 15 Aug 2014 at 5:28

GoogleCodeExporter commented 9 years ago 
Hi, 

can you provide some more information?
- Are there any netmap-related error messages in the kernel log?
- where is tcpreplay spinning? e.g., is it continually returning from the 
poll() with an error?

Original comment by giuseppe.lettieri73 on 16 Aug 2014 at 4:06

GoogleCodeExporter commented 9 years ago 
Hi, I am the original reporter of this issue.

I first noticed that tcpreplay 4.0.4 would hang in netmap mode while tcpdump is 
also running in netmap mode. 
I have since then updated to tcpreplay version 4.0.5 beta 3. The symptoms have 
changed a bit, but the use
case is still not working

Running tcpreplay 4.0.5 beta 3

Setup: 
Eth16 and Eth17 are connected back to back, so traffic leaving eth16 will 
arrive eth17. Goal is to play Pcap from
eth16 using netmap and have tcpdump running with netmap on eth17 (saving to 
RAMdisk). Both eth16 and eth17 are ixgbe
interfaces running netmap patched drivers. 

Pre-req:
I can run both tcpreplay and tcpdump with netmap seperately (i can get good 
performance on both). But I can't seem to
run them at the same time. 

Step 1:
Launches tcpdump with netmap first 

devuser@UTraffic22:~$ sudo taskset -c 3 sudo tcpdump -s 0 -nei netmap:eth17 -w 
/mnt/tmpfs/pk1.pcap -c 100000
[sudo] password for devuser:
pcap_netmap_activate device netmap:eth17 priv 0x18b6c50 fd 3 ports 0..11
pcap_netmap_ioctl eth17 ioctl 0x8913 returns 0
pcap_netmap_ioctl eth17 ioctl 0x8914 returns 0
tcpdump: listening on netmap:eth17, link-type EN10MB (Ethernet), capture size 
262144 bytes

Step 2:
Launch tcpreplay 4.0.5 beta 3 playing pcap, it runs and exits 

devuser@UTraffic22:~$ sudo taskset -c 11 sudo tcpreplay --netmap -i eth16 -M 
8000 -K /tmp/gre_ipv4.pcap
[sudo] password for devuser:
Switching network driver for eth16 to netmap bypass mode... done!
File Cache is enabled
Actual: 1000000 packets (1020000000 bytes) sent in 1.00 seconds.
Rated: 999975490.7 Bps, 7999.80 Mbps, 980368.12 pps
Flows: 1000000 flows, 980368.12 fps, 1000000 flow packets, 0 non-flow
Statistics for network device: eth16
        Attempted packets:         1000000
        Successful packets:        1000000
        Failed packets:            0
        Truncated packets:         0
        Retried packets (ENOBUFS): 0
        Retried packets (EAGAIN):  0
Switching network driver for eth16 to normal mode... done!

Step 3:
However, the first tcpdump instance (which would exit upcon receiving 1000000 
packets with the "-c" option) 
did not exit

devuser@UTraffic22:~$ sudo taskset -c 3 sudo tcpdump -s 0 -nei netmap:eth17 -w 
/mnt/tmpfs/pk1.pcap -c 100000
[sudo] password for devuser:
pcap_netmap_activate device netmap:eth17 priv 0x18b6c50 fd 3 ports 0..11
pcap_netmap_ioctl eth17 ioctl 0x8913 returns 0
pcap_netmap_ioctl eth17 ioctl 0x8914 returns 0
tcpdump: listening on netmap:eth17, link-type EN10MB (Ethernet), capture size 
262144 bytes
^C0 packets captured
0 packets received by filter
0 packets dropped by kernel
pcap_netmap_ioctl eth17 ioctl 0x8913 returns 0

Step 4:
Re-run step 1 and 2 but with tcpdump NOT in netmap mode (tcpreplay in netmap 
mode), tcpdump logs the packets just fine

devuser@UTraffic22:~$ sudo taskset -c 3 sudo tcpdump -s 0 -nei eth17 -w 
/mnt/tmpfs/pk1.pcap -c 100000

tcpdump: listening on eth17, link-type EN10MB (Ethernet), capture size 262144 
bytes
100000 packets captured
100257 packets received by filter
0 packets dropped by kernel

I tried 4 various combinations between tcpreplay and tcpdump (with and without 
netmap mode), below are the results

Here is another interesting observation, below is a summary of the three 
combinations:

tcpreplay (netmap enabled) -> tcpdump (netmap enabled) = Does not work
tcpreplay (netmap enabled) -> tcpdump (no netmap) = Works fine
tcpreplay (no netmap) -> tcpdump (netmap enabled) = Does not work
tcpreplay (no netmap) -> tcpdump (no netmap) = Works fine

I also noticed that the TX counter for my network interface never seems to 
increment when running in netmap mode. 

Examming dmeg and kerner.log, I don't see anything strange, with no errors in 
particular

[894792.609623] ixgbe 0000:44:00.1 eth17: detected SFP+: 6
[894792.933803] ixgbe 0000:44:00.1 eth17: NIC Link is Up 10 Gbps, Flow Control: 
RX/TX
[894819.342554] 026.358522 [ 753] generic_netmap_dtor       Restored native NA  
         (null)
[894819.377905] 026.393847 [ 753] generic_netmap_dtor       Restored native NA  
         (null)
[894820.711718] ixgbe 0000:44:00.0 eth16: detected SFP+: 5
[894820.851576] ixgbe 0000:44:00.0 eth16: NIC Link is Up 10 Gbps, Flow Control: 
RX/TX
[894828.505500] ixgbe 0000:44:00.0 eth16: detected SFP+: 5
[894828.637627] ixgbe 0000:44:00.0 eth16: NIC Link is Up 10 Gbps, Flow Control: 
RX/TX
[894836.087561] ixgbe 0000:44:00.1 eth17: detected SFP+: 6
[894836.219567] ixgbe 0000:44:00.1 eth17: NIC Link is Up 10 Gbps, Flow Control: 
RX/TX

Original comment by morph...@gmail.com on 18 Aug 2014 at 6:29

GoogleCodeExporter commented 9 years ago 
I'm running tcpdump with netmap-libpcap

apcon@UTraffic22:~/working/tcpreplay_interactions$ sudo tcpdump --version
[sudo] password for apcon:
tcpdump version 4.6.1
libpcap version 1.6.0-PRE-GIT_2014_08_04

Netmap and netmap-libpcap are both recent clones (August 4th). 
I'm running Ubuntu 14.04, running ixgbe driver patched with netmap's ixgbe.patch

apcon@UTraffic22:~/netmap$ sudo ethtool -i eth16
driver: ixgbe
version: 3.15.1-k
firmware-version: 0x80000208
bus-info: 0000:44:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

Below is dmesg in regard to the ixgbe netmap interface

[  250.241049] 155.199290 [2259] netmap_attach             success for eth16
[  250.284602] systemd-udevd[2487]: renamed network interface eth0 to eth16
[  250.437586] ixgbe 0000:44:00.1: irq 306 for MSI/MSI-X
[  250.437605] ixgbe 0000:44:00.1: irq 307 for MSI/MSI-X
[  250.437622] ixgbe 0000:44:00.1: irq 308 for MSI/MSI-X
[  250.437638] ixgbe 0000:44:00.1: irq 309 for MSI/MSI-X
[  250.437660] ixgbe 0000:44:00.1: irq 310 for MSI/MSI-X
[  250.437677] ixgbe 0000:44:00.1: irq 311 for MSI/MSI-X
[  250.437693] ixgbe 0000:44:00.1: irq 312 for MSI/MSI-X
[  250.437709] ixgbe 0000:44:00.1: irq 313 for MSI/MSI-X
[  250.437725] ixgbe 0000:44:00.1: irq 314 for MSI/MSI-X
[  250.437742] ixgbe 0000:44:00.1: irq 315 for MSI/MSI-X
[  250.437758] ixgbe 0000:44:00.1: irq 316 for MSI/MSI-X
[  250.437775] ixgbe 0000:44:00.1: irq 317 for MSI/MSI-X
[  250.437791] ixgbe 0000:44:00.1: irq 318 for MSI/MSI-X
[  250.437839] ixgbe 0000:44:00.1: Multiqueue Enabled: Rx Queue count = 12, Tx 
Queue count = 12
[  250.437974] ixgbe 0000:44:00.1: PCI Express bandwidth of 32GT/s available
[  250.437976] ixgbe 0000:44:00.1: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[  250.438057] ixgbe 0000:44:00.1: MAC: 2, PHY: 12, SFP+: 6, PBA No: 400900-000
[  250.438058] ixgbe 0000:44:00.1: 00:e0:ed:49:90:99
[  250.497171] ixgbe 0000:44:00.1: Intel(R) 10 Gigabit Network Connection
[  250.497188] 155.455229 [2259] netmap_attach             success for eth17
[  250.539913] systemd-udevd[2487]: renamed network interface eth0 to eth17
[  250.693937] ixgbe 0000:46:00.0: irq 319 for MSI/MSI-X
[  250.693956] ixgbe 0000:46:00.0: irq 320 for MSI/MSI-X
[  250.693973] ixgbe 0000:46:00.0: irq 321 for MSI/MSI-X
[  250.693989] ixgbe 0000:46:00.0: irq 322 for MSI/MSI-X
[  250.694005] ixgbe 0000:46:00.0: irq 323 for MSI/MSI-X
[  250.694021] ixgbe 0000:46:00.0: irq 324 for MSI/MSI-X
[  250.694037] ixgbe 0000:46:00.0: irq 325 for MSI/MSI-X
[  250.694053] ixgbe 0000:46:00.0: irq 326 for MSI/MSI-X
[  250.694072] ixgbe 0000:46:00.0: irq 327 for MSI/MSI-X
[  250.694088] ixgbe 0000:46:00.0: irq 328 for MSI/MSI-X
[  250.694104] ixgbe 0000:46:00.0: irq 329 for MSI/MSI-X
[  250.694120] ixgbe 0000:46:00.0: irq 330 for MSI/MSI-X
[  250.694136] ixgbe 0000:46:00.0: irq 331 for MSI/MSI-X
[  250.694180] ixgbe 0000:46:00.0: Multiqueue Enabled: Rx Queue count = 12, Tx 
Queue count = 12
[  250.694315] ixgbe 0000:46:00.0: PCI Express bandwidth of 32GT/s available
[  250.694317] ixgbe 0000:46:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[  250.694403] ixgbe 0000:46:00.0: MAC: 2, PHY: 12, SFP+: 5, PBA No: 400900-000
[  250.694405] ixgbe 0000:46:00.0: 00:e0:ed:49:90:9a
[  250.753364] ixgbe 0000:46:00.0: Intel(R) 10 Gigabit Network Connection

Original comment by morph...@gmail.com on 18 Aug 2014 at 6:37

GoogleCodeExporter commented 9 years ago 
Hi morphyno, thank you for the report. I need to reproduce the failure locally, 
since all the obvious culprits have been ruled out by either some of your 
tests, the kernel logs, or my looking at the relevant code in tcpreplay.

Unfortunately I am out of office this week, but there is a chance I can get 
remote access to a Linux machine with two ixgbe's connected back-to-back. 

Giuseppe

Original comment by giuseppe.lettieri73 on 18 Aug 2014 at 10:35

GoogleCodeExporter commented 9 years ago 
Hi Giuseppe, thanks for looking into this, let me know if there's anything I 
can do to assist

Original comment by morph...@gmail.com on 19 Aug 2014 at 5:23

GoogleCodeExporter commented 9 years ago 
Hi morphyno,

I finally had access to a suitable machine. Unfortunately, I am not able to 
reproduce the problem: I can send with tcpreplay --netmap on one ixgbe and 
receive with tcpdump -nei netmap:... on the other.

I have compiled everything from sources, here are the relevant commit hashes:

tcpreplay:      da25e37b78f39e3292ddb607a53b1703ba21949f
tcpdump:        29d83dbb61be640899d969357f2a8e1c3e02c7ee
netmap-libpcap: 0890b993dae48767af172b693f2d664181dbe943

For netmap proper, I have tried bot the current 'master' and 'next' branches 
available here. Everything works fine.

We need to do some more investigation on your own system, I am afraid. Does 
tcpdump with netmap loses all the packets, or just a fraction of them? (i.e., 
does tcpdump exits for some lower value of -c?)

Original comment by giuseppe.lettieri73 on 26 Aug 2014 at 12:30

GoogleCodeExporter commented 9 years ago 
Hi Giuseppe:

Thanks for looking into this. From my observations, when tcpreplay is using a 
sending interface in netmap mode, the receiving tcpdump (in netmap mode) does 
not receive any packets. 

my netmap-libpcap matches yours, but I'm using official tcpdump release (4.6.1) 
and tcpreplay release (4.0.5beta3). I will try again using the same versions 
you did (i'm assuming you just downloaded the latest from respective GIT repo's)

Out of curiosity, was your testing conducted on Ubuntu 14.04 as well?

I also have another question, does netmap put he IXGBE in "poll-mode"? I know 
for PF_RING, their IXGBE modifications were with NAPI (rx polling) enabled, and 
it doesn't look netmap is doing that. Given netmap's great improvement gains, 
will putting the driver in poll mode greatly increase overall performance?

Original comment by morph...@gmail.com on 26 Aug 2014 at 6:46

GoogleCodeExporter commented 9 years ago 
> Out of curiosity, was your testing conducted on Ubuntu 14.04 as well?

yes, I was using Ubuntu 14.04 with kernel 3.13.0-34.

Re polling and NAPI, we are also using NAPI to wake up the receiving process 
(which then does all the work in the poll() system call). NAPI does the 
switching between polling and interrupts on its own, we have not changed that. 
We have also not tried to tune that either, since performance was already good, 
but that may change in the future.

Going back to the issue, my failure to reproduce it locally is very bad news, 
but we can still try something:
- can you please attach the pcap file you are using? The one I was using was 
very simple (a capture of pkt-gen) and that may have not triggered some bug, 
e.g., in the handling of packet lengths. 
- can you put netmap in verbose mode and see if something interesting shows up 
in the kernel logs?
You can enable verbose mode with:
# echo 1 > /sys/module/netmap_lin/parameters/verbose
- is there any way to slow down the rate of the packets sent by tcpreplay, to 
see if the problem is only triggered by high packet rates?

Original comment by giuseppe.lettieri73 on 29 Aug 2014 at 7:56

GoogleCodeExporter commented 9 years ago 
Hi Giuseppe:

For some reason, when I installed a newer version of tcpdump, and built it 
against netmap-libpcap, the problem is no longer occurring.

Original comment by morph...@gmail.com on 5 Sep 2014 at 4:38

GoogleCodeExporter commented 9 years ago 
Hi Giuseppe:

I have somehow recreated this failure scenario while putting together a second 
system. I suspect there is an issue with netmap-libpcap and tcpdump linking as 
in this case netmap mode for tcpdump (official version 4.6.1) is not working at 
all. 

I have enabled verbose output for netmap_lin module and I notice while the 
tcpdump is in netmap mode, it continuously prints the RX and TX queues.

This is the command I'm using

apcon@UTraffic22:~$ sudo taskset -c 2 sudo tcpdump -s 0 -nei netmap:eth3 -w 
/mnt/tmpfs/eth3.pcap -c 1
pcap_netmap_activate device netmap:eth3 priv 0x24dbc50 fd 3 ports 0..11
pcap_netmap_ioctl eth3 ioctl 0x8913 returns 0
pcap_netmap_ioctl eth3 ioctl 0x8914 returns 0
tcpdump: listening on netmap:eth3, link-type EN10MB (Ethernet), capture size 
262144 bytes
^C0 packets captured
0 packets received by filter
0 packets dropped by kernel
pcap_netmap_ioctl eth3 ioctl 0x8913 returns 0

Attached is the dmesg collected during the tcpdump session (it prints to screen 
about every 2 seconds during the process's life)

Original comment by morph...@gmail.com on 8 Sep 2014 at 9:17

Attachments:

GoogleCodeExporter commented 9 years ago 
Hi Giuseppe:

I know it sounds silly, but what is the correct method of install for 
netmap-libpcap? I have done three system installs, same hardware and software, 
but with different results: 

case 1: tcpdump and tcpreplay both works with netmap
case 2: tcprelay works with netmap but tcpdump does not (recognizes 
"netmap:ethX", but does not capture packes)
case 3: tcpdump does not recognize the "netmap:ethX" interface, error message is
tcpdump: netmap:eth0: No such device exists
(SIOCGIFHWADDR: No such device)

I'm suspecting this is more of an install issue and I'm not doing something 
correctly.

This is how I installed netmap-libpcap
1) make sure netmap_lin is loaded
2) glone netmap-libpcap 
3) ./configure && make && make install
4) Install tcpdump with standard (configure && make && make install)

I'm also curious about how netmap maps the interfaces, because I have a large 
number of interfaces (16 NIC's), I have a modified 70-net-persistent file so 
the interfaces gets renamed (see dmesg example below), could this cause an 
issue with netmap?

[11919.671298] ixgbe 0000:06:00.0: Multiqueue Enabled: Rx Queue count = 12, Tx 
Queue count = 12
[11919.671433] ixgbe 0000:06:00.0: PCI Express bandwidth of 32GT/s available
[11919.671435] ixgbe 0000:06:00.0: (Speed:5.0GT/s, Width: x8, Encoding Loss:20%)
[11919.671522] ixgbe 0000:06:00.0: MAC: 2, PHY: 12, SFP+: 5, PBA No: 400900-000
[11919.671523] ixgbe 0000:06:00.0: 00:e0:ed:49:90:94
[11919.672877] ixgbe 0000:06:00.0: Intel(R) 10 Gigabit Network Connection
[11919.672932] 147.576627 [2259] netmap_attach             success for eth0
[11919.725880] systemd-udevd[21421]: renamed network interface eth0 to eth4

Original comment by morph...@gmail.com on 19 Sep 2014 at 12:45

GoogleCodeExporter commented 9 years ago 
Ok, I figured out the discrepancies. For the last case, I didn't copy the 
netmap/sys/net headers to "/usr/include". As of the second case, it is directly 
related to my set cpu affinity script. This issue can be closed

Original comment by morph...@gmail.com on 23 Sep 2014 at 6:05

GoogleCodeExporter commented 9 years ago 
Thanks for the update, I am closing the issue now.

Original comment by giuseppe.lettieri73 on 25 Sep 2014 at 7:18

GoogleCodeExporter commented 9 years ago 
Hi all,
my problem while installing netmap,
tcpdump does not recognize the "netmap:ethX" interface, error message is
tcpdump: netmap:eth0: No such device exists
(SIOCGIFHWADDR: No such device)

Without copying "netmap/sys/net" headers to "/usr/include",netmap support is 
not there.when I try to configure in libpcap,

./configure | grep netmap

I am getting as following,
checking for net/netmap_user.h... no
configure: netmap is not supported

can anyone help me to install it??
Thanks in advance...

Original comment by kamalasu...@multicorewareinc.com on 9 Mar 2015 at 6:50