Receiving more packets then sent + plus kring error, when pkt-gen use multiple threads

[GoogleCodeExporter]

What steps will reproduce the problem?
1. ixgbe-3.21.2 source is downloaded, patched with "diff--ixgbe--30500--30900" 
in the netmap source, and new driver ixgbe.ko is compiled without error, loaded.
2. insmod netmap/LINUX/netmap_lin.ko
3. insmod ixgbe-3.21.2/src/ixgbe.ko allow_unsupported_sfp=1,1 LRO=0,0
4. ./pkt-gen -i eth3 -f rx -c 4 -p 4 
   ...
   Receiving from netmap:eth3:24 queues 4 threads and 4 CPUs.
   ... 

What is the expected output? What do you see instead?

I sent 20 M packets exactly ( from another server connected to eth3 via cross 
cable )
When sending operation is finished and pkt-gen main_thread continously reports 
"0 pps" I press "Ctrl+C" and netmap reported 20,4 M packets received. There are 
400K packets received by multiple threads.

In addition, /var/log/messages continously reported "nm_rxsync_prologue kring 
error: hwcur ..." 

What version of the product are you using? On what operating system?

I'm using Oracle Linux. uname -a:
"3.8.13-16.2.1.el6uek.x86_64"

lspci:

24:00.0 Intel Corporation 82599ES 10 Gigabit SFI/SFP Network connection ( rev 
01 )
24:00.1 Intel Corporation 82599ES 10 Gigabit SFI/SFP Network connection ( rev 
01 )

Please provide any additional information below.

There is no problem if I run pkt-gen with only single thread.
A simple question: How can I guarantee such that netmap uses modified driver ? 
If I load ixgbe driver before loading netmap_lin.ko, does netmap run in 
modified driver mode ?  

Original issue reported on code.google.com by berraa...@gmail.com on 15 Oct 2014 at 2:49

[GoogleCodeExporter]

Sorry, I forgat to say,
I apply the following ethtool commands also to the NIC:

ethtool -A eth3 autoneg off rx off tx off
ethtool -K eth3 rx off
ethtool -K eth3 gro off

thanks.

Original comment by berraa...@gmail.com on 15 Oct 2014 at 3:02

[GoogleCodeExporter]

Hi berraazra,

I need some additional details on how you compile your ixgbe driver. Netmap 
support is only included if you pass CONFIG_NETMAP=m to make. Moreover, since 
you are building outside the netmap directory, you should also pass some -I... 
options via EXTRA_CFLAGS, otherwise you should get errors for missing files. If 
you have done nothing of the sort, you have likely compiled the ixgbe driver 
without native netmap support. You can check this easily: if you are able to 
load the ixgbe driver before netmap_lin.ko, then netmap support is not there, 
since otherwise you would get errors for missing symbols.

If netmap support is not compiled in your ixgbe driver, you are using the 
generic netmap adapter. The errors you get need to be addressed either way, but 
first we need to know the exact setup.

Original comment by giuseppe.lettieri73 on 16 Oct 2014 at 10:36

[GoogleCodeExporter]

Hi,

I wish to use newest ixgbe driver. All steps of compiling the driver and netmap 
is avail in the attachment. So, first I apply the matching patch to the driver 
source code, make it. On the other hand, I make netmap with "NODRIVER=1" 
option. So I think, "#ifdef DEV_NETMAP" parts are not included in ixgbe.ko, 
isn't it ? How should I compile the newest driver ? Copying the source codes to 
the "../drivers/net/ethernet/intel/ixgbe", and compiling netmap with drivers 
didn't work.

Thanks.

Original comment by berraa...@gmail.com on 16 Oct 2014 at 12:17

Attachments:

• CompilingNetmapAndIxgbe.txt

[GoogleCodeExporter]

Hi, 
When I try to compile netmap module with original driver sources in the kernel 
source, I couldn't achieve. 
Path for kernel configuration-headers is 
"/usr/src/kernels/3.8.13-16-2-1.el6uk.x86_64"
Path for kernel source is 
"/usr/src/debug/kernel-3.8.13/linux-3.8.13-16-2-1.el6uk"

Results are in the attachment.

Thanks.

Original comment by berraa...@gmail.com on 16 Oct 2014 at 12:38

[GoogleCodeExporter]

Original comment by berraa...@gmail.com on 16 Oct 2014 at 12:39

Attachments:

• CompilingNetmap.txt

[GoogleCodeExporter]

Hi,
Can you please use the 'next' branch of netmap? It contains a configure script 
that should make things a little easier. 

In the LINUX subdirectory, run:

./configure --kernel-sources=/path/to/full/kernel/sources

The problem you mention in #4 is caused by the patches of igb and e1000e. If 
you are not interested in these drivers, you can add the following options to 
configure:

--no-driver=igb --no-driver=e1000e

The compilation of the out-of-tree version of ixgbe is not directly supported, 
but you can still do it, if you are sufficiently determined :)
Once ./configure has finished, you should obtain a drivers.mak file in the 
LINUX directory. Open it and look for the get-ixgbe target, then change the cp 
operation to something like

cp -Rp /path/to/your/preferred/sources ixgbe

(be sure to also replace the '.' with 'ixgbe', otherwise you might get the 
wrong directory name from the copy)

Now you can try to run make.

Original comment by giuseppe.lettieri73 on 16 Oct 2014 at 1:06

[GoogleCodeExporter]

Very sorry, may be so dummy but

What does "Can you please use the 'next' branch of netmap?" mean ? 
I performed "git clone https://code.google.com/p/netmap/" but there is no new 
configure script. Where is the 'next' branch of netmap.

Thanks a lot.

Original comment by berraa...@gmail.com on 16 Oct 2014 at 1:56

[GoogleCodeExporter]

That is a git branch, you already have it if you have cloned the repository, 
you just have to check it out by issuing

git checkout -b next origin/next

in your netmap directory.

Original comment by giuseppe.lettieri73 on 16 Oct 2014 at 2:14

[GoogleCodeExporter]

Ok, I got it, I will try your solution.

To inform;
In #2, when you say "The errors you get need to be addressed either way...", I 
understand that duplicate packets are still problem for generic netmap adapter 
without driver changes, isn't it ? 
Because, I modify my packet sender program, in order to hold a packet order 
from 1 to 20M in the IP checksum and frag id fields of each packet; also 
changed the pkt-gen to count total number of capture for each unique packet id. 
Thus, I proved the multiple capture of a unique packet by multiple threads up 
to the number specified by "-p " option. 

Very thanks.

Original comment by berraa...@gmail.com on 16 Oct 2014 at 2:31

[GoogleCodeExporter]

Yes, we need to investigate the problem you are seeing. It may be a bug in the 
generic adapter, or somewhere else. Therefore, it is interesting to know if you 
still can see the problem when you use the modified driver.

Original comment by giuseppe.lettieri73 on 16 Oct 2014 at 2:39

[GoogleCodeExporter]

Hi,

I made a fresh start by downloading linux-3.8.13 from kernel.org and recompile.
Now, my server is booted with 3.8.13 kernel, and sources,headers etc. are in 
the same place.
I compiled netmap and ixgbe ( original one in the kernel source ) but can't 
load netmap module. "insmod netmap.ko" results in "-1 unknown symbol in 
module". There is a problem with 'split_page' whatever.
Details are in the attachment.

Thanks.

Original comment by berraa...@gmail.com on 17 Oct 2014 at 8:11

Attachments:

• CompileNetmapLast.txt

[GoogleCodeExporter]

In addition, after compilation, I expect "netmap_lin.ko" but there is no 
"netmap_lin.ko" module, only "netmap.ko" was produced.

Original comment by berraa...@gmail.com on 17 Oct 2014 at 8:21

[GoogleCodeExporter]

I have pushed some updates that should, hopefully, solve the split_page() 
issue. 

In the latest versions of netmap the module is called 'netmap.ko', as you have 
noticed already.

Original comment by giuseppe.lettieri73 on 17 Oct 2014 at 3:20

[GoogleCodeExporter]

Hi Giuseppe,

Last update solved the compilation problem.

Without any error, I compiled ixgbe and also netmap.ko.

As you expressed before, ixgbe can't be loaded before netmap had been loaded.

There is no kring errors in messages when pkt-gen is run.

But new problem is DROPS ! :)

I'm sending 20M packets in 49 seconds, average packet length is ~780 byte; 
roughly at 400K packets/sec speed. Then ~%75 packets are dropped.

IP cheksum of the packets are valid.
I tried "-b 65536" for the pkt-gen, but there is no change.
I changed ring size to 2**17, but it doesn't have any affect.
I try single thread also.
There are two attachements, one for system log when I loaded netmap,ixgbe and 
run pkt-gen. The other one is the pkt-gen output. ifconfing reports severe 
drops.

I didn't care SFP warning, because without netmap, my simple packet handler 
kernel module can capture without drops.

Thanks for all.

Original comment by berraa...@gmail.com on 20 Oct 2014 at 8:23

Attachments:

• VarLogMessages.txt
• RunningPktGen.txt

[GoogleCodeExporter]

Hi,

I repeated the test after setting 
"/sys/devices/system/cpu/*/cpufreq/scaling_governor" to the "performance".

I realized that my new ixgbe driver with netmap support drops packets directly 
qhile no pkt-gen capture is running. 
When "ethtool -S" reported "rx_missed_errors" for the drops.

SystemStatus file includes /var/log/messages, lsmod, top, and severel ethtool 
results before I begin the test.
Results include ifconfig and ethtool -S results.

~7M packets over 20M packets are dropped.

20M packets are sent in 145 seconds,resulting  ~138K packets/sec speed and ~825 
Mbit/sec bandwidth. Average packet size is ~780 byte.

Where is the problem ?

Original comment by berraa...@gmail.com on 23 Oct 2014 at 12:20

Attachments:

• Results_20141023.txt
• SystemStatus_20141023.txt

[GoogleCodeExporter]

I couldn't hear any voice since October 20, I wonder if I closed my issue 
mistakenly ? If so, #14 and #15 are still waiting to be replied. Can you inform 
at least "we are looking for the problem..." or "we closed the issue due to..."

thanks.

Original comment by berraa...@gmail.com on 6 Nov 2014 at 3:03

[GoogleCodeExporter]

My problem is solved with setting the coalesce variable rx-usesc to 1000 when 
multiple RSS queues are used. 

In addition, I have changed ixgbe driver code a little to disable multiple RSS 
queues to use single queue. I repeated the tests for multiple queue disabled. 
In this case, I realized that there is no need to set rx-usecs, netmap runs 
correctly without drop.

So, the problem ( multiple captured packets + kring_error) is valid for generic 
netmap adapter.

Original comment by berraa...@gmail.com on 7 Nov 2014 at 3:40

[GoogleCodeExporter]

Hi berraazra,

can you please try the attached patch to see if it has the same effect as 
'rx-usecs 1000'?

Cheers,
Giuseppe

Original comment by giuseppe.lettieri73 on 10 Nov 2014 at 11:57

Attachments:

• ixgbe-itr.patch

[GoogleCodeExporter]

I tried the patch but it did no affect.

For single-queue, there is no packet loss when rx-usecs = 1

For multi-queue, there is no packet loss when rx-usecs ~ > 600.

I attached patch results and multi-queue output when 40M packets are sent and 
rx-usecs=300.

I have another question:

When I try to run two independant pkt-gen application to receive packets, 
neither one can capture the packets; both capture a few hundred packets. This 
is a expected result ?

Original comment by berraa...@gmail.com on 10 Nov 2014 at 4:21

[GoogleCodeExporter]

Original comment by berraa...@gmail.com on 11 Nov 2014 at 12:56

Attachments:

• Results_20141111.txt