search

ntop / PF_RING

High-speed packet processing framework
http://www.ntop.org
GNU Lesser General Public License v2.1
2.71k stars 349 forks source link

Complete traffic drop in ZC mode with ixgbe driver in multi queue mode #7

Closed pavel-odintsov closed 9 years ago

pavel-odintsov commented 9 years ago

Hello, folks!

I have problems with GIT version of PF_RING. I have ixgbe and Debian Jessie with 3.16 kernel.

I installed PF_RING, build libs and modules and run driver with ./load_module and everything worker perfectly.

But my tool bound to single CPU core:

1  [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||100.0%]     Tasks: 37, 98 thr, 71 kthr; 2 running
  2  [||                                                           1.9%]     Load average: 2.36 2.07 1.72 
  3  [|||                                                          3.4%]     Uptime: 00:35:50
  4  [|                                                            0.9%]
  Mem[||||                                                  688/32205MB]
  Swp[                                                         0/8190MB]

So, I switched:insmod ./ixgbe.ko RSS=1,1,1, To:`insmod ./ixgbe.ko RSS=4,4,4,4

And call ./load_module again. After this changes I can't saw any traffic on this interface.

And zcount show following:

./zcount -i zc:eth4 -c 0
#########################################################################
# ERROR: You do not seem to have a valid PF_RING ZC license 6.0.3.150325 for eth4 [Intel 10 Gbit ixgbe 82599-based]
# ERROR: Please get one at http://shop.ntop.org/.
#########################################################################
# We're now working in demo mode with packet capture and          
# transmission limited to 5 minutes
#########################################################################
#########################################################################
# ERROR: You do not seem to have a valid PF_RING ZC license 6.0.3.150325 for eth4 [Intel 10 Gbit ixgbe 82599-based]
# ERROR: Please get one at http://shop.ntop.org/.
#########################################################################
=========================
Absolute Stats: 0 pkts (14'832'738 drops) - 0 bytes
=========================

=========================
Absolute Stats: 0 pkts (29'662'683 drops) - 0 bytes
Actual Stats: 0.00 pps (14'828'803.18 drops) - 0.00 Gbps
=========================

=========================
Absolute Stats: 0 pkts (44'493'780 drops) - 0 bytes
Actual Stats: 0.00 pps (14'828'931.97 drops) - 0.00 Gbps
=========================

=========================
Absolute Stats: 0 pkts (59'324'327 drops) - 0 bytes
Actual Stats: 0.00 pps (14'829'612.73 drops) - 0.00 Gbps
=========================

=========================
Absolute Stats: 0 pkts (74'157'039 drops) - 0 bytes
Actual Stats: 0.00 pps (14'830'754.34 drops) - 0.00 Gbps

Btw, my toolkit internal stats show something similar:

PF_RING ZC in queue statistics
Received:       0
Sent:           0
Dropped:        1209057743
Dropped:        0.00 %

Code:

       int stats_res = pfring_zc_stats(inzq[0], &stats);

        if (stats_res) {
            logger<<log4cpp::Priority::ERROR<<"Can't get PF_RING ZC stats for in queue";
        } else {
            double dropped_percent = 0;

            if (stats.recv + stats.sent > 0) {    
                dropped_percent = (double)stats.drop / ((double)stats.recv + (double)stats.sent) * 100;
            }

            output_buffer<<"\n";
            output_buffer<<"PF_RING ZC in queue statistics\n";
            output_buffer<<"Received:\t"<<stats.recv<<"\n";
            output_buffer<<"Sent:\t\t"<<stats.sent<<"\n";
            output_buffer<<"Dropped:\t"<<stats.drop<<"\n";
            output_buffer<<"Dropped:\t"<<std::fixed << std::setprecision(2)<<dropped_percent<<" %\n";
        }
cardigliano commented 9 years ago

Hi Pavel zc:eth4 means zc:eth4@0, in essence you are capturing from queue 0 only.

Alfredo

On 13 May 2015, at 11:00, Pavel Odintsov notifications@github.com wrote:

Hello, folks!

I have problems with GIT version of PF_RING. I have ixgbe and Debian Jessie with 3.16 kernel.

I installed PF_RING, build libs and modules and run driver with ./load_module and everything worker perfectly.

But my tool bound to single CPU core:

1 [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||100.0%] Tasks: 37, 98 thr, 71 kthr; 2 running 2 [|| 1.9%] Load average: 2.36 2.07 1.72 3 [||| 3.4%] Uptime: 00:35:50 4 [| 0.9%] Mem[|||| 688/32205MB] Swp[ 0/8190MB] So, I switched:insmod ./ixgbe.ko RSS=1,1,1, To:`insmod ./ixgbe.ko RSS=4,4,4,4

And call ./load_module again. After this changes I can't saw any traffic on this interface.

And zcount show following:

./zcount -i zc:eth4 -c 0 #########################################################################

ERROR: You do not seem to have a valid PF_RING ZC license 6.0.3.150325 for eth4 [Intel 10 Gbit ixgbe 82599-based]

ERROR: Please get one at http://shop.ntop.org/.

#########################################################################

We're now working in demo mode with packet capture and

transmission limited to 5 minutes

######################################################################### #########################################################################

ERROR: You do not seem to have a valid PF_RING ZC license 6.0.3.150325 for eth4 [Intel 10 Gbit ixgbe 82599-based]

ERROR: Please get one at http://shop.ntop.org/.

Absolute Stats: 0 pkts (14'832'738 drops) - 0 bytes

Absolute Stats: 0 pkts (29'662'683 drops) - 0 bytes

Actual Stats: 0.00 pps (14'828'803.18 drops) - 0.00 Gbps

Absolute Stats: 0 pkts (44'493'780 drops) - 0 bytes

Actual Stats: 0.00 pps (14'828'931.97 drops) - 0.00 Gbps

Absolute Stats: 0 pkts (59'324'327 drops) - 0 bytes

Actual Stats: 0.00 pps (14'829'612.73 drops) - 0.00 Gbps

Absolute Stats: 0 pkts (74'157'039 drops) - 0 bytes Actual Stats: 0.00 pps (14'830'754.34 drops) - 0.00 Gbps Btw, my toolkit internal stats show something similar:

PF_RING ZC in queue statistics Received: 0 Sent: 0 Dropped: 1209057743 Dropped: 0.00 % Code:

   int stats_res = pfring_zc_stats(inzq[0], &stats);

    if (stats_res) {
        logger<<log4cpp::Priority::ERROR<<"Can't get PF_RING ZC stats for in queue";
    } else {
        double dropped_percent = 0;

        if (stats.recv + stats.sent > 0) {    
            dropped_percent = (double)stats.drop / ((double)stats.recv + (double)stats.sent) * 100;
        }

        output_buffer<<"\n";
        output_buffer<<"PF_RING ZC in queue statistics\n";
        output_buffer<<"Received:\t"<<stats.recv<<"\n";
        output_buffer<<"Sent:\t\t"<<stats.sent<<"\n";
        output_buffer<<"Dropped:\t"<<stats.drop<<"\n";
        output_buffer<<"Dropped:\t"<<std::fixed << std::setprecision(2)<<dropped_percent<<" %\n";
    }

— Reply to this email directly or view it on GitHub https://github.com/ntop/PF_RING/issues/7.

pavel-odintsov commented 9 years ago

Well, well. But when this behavior was changed? What name if interface should I use for listening on all interfaces as was before?

cardigliano commented 9 years ago

This has never been changed, this is the behaviour for kernel-bypass technologies (DNA and ZC), in fact in this case you have direct access to the NIC queues, while in kernel-mode the kernel is polling form all the queues.

Alfredo

On 13 May 2015, at 11:09, Pavel Odintsov notifications@github.com wrote:

Well, well. But when this behavior was changed? What name if interface should I use for listening on all interfaces as was before?

— Reply to this email directly or view it on GitHub https://github.com/ntop/PF_RING/issues/7#issuecomment-101582723.

pavel-odintsov commented 9 years ago

cardigliano, please sorry me. I work with single queue mode all time! :) Fixed now and works with charm!

pavel-odintsov commented 9 years ago

Well but I can't achieve wire speed with ZC.

I switched to single queue mode and run zcount -c 0 -i zc:eth4:

=========================
Absolute Stats: 734'746'930 pkts (24'499'577 drops) - 61'718'742'120 bytes
Actual Stats: 14'128'520.83 pps (470'158.52 drops) - 9.49 Gbps
=========================

Cpu load is not 100%:

 1  [|                                                              0.5%]     Tasks: 38, 94 thr, 68 kthr; 2 running
  2  [||||||||||||||||                                              22.4%]     Load average: 2.07 4.13 4.75 
  3  [|||||||||||||                                                 17.8%]     Uptime: 02:28:08
  4  [|||||                                                          5.8%]
  Mem[|||||                                                   706/32205MB]
  Swp[                                                           0/8190MB]

CPU: Intel(R) Xeon(R) CPU E5-2407 0 @ 2.20GHz Debian 7 Jessie 3.16

cardigliano commented 9 years ago

Please try adding -a, probably passive wait is not behaving well on this machine.

Alfredo

On 13 May 2015, at 12:40, Pavel Odintsov notifications@github.com wrote:

Well but I can't achieve wire speed with ZC.

I switched to single queue mode and run zcount -c 0 -i zc:eth4:

Absolute Stats: 734'746'930 pkts (24'499'577 drops) - 61'718'742'120 bytes

Actual Stats: 14'128'520.83 pps (470'158.52 drops) - 9.49 Gbps

Cpu load is not 100%:

1 [| 0.5%] Tasks: 38, 94 thr, 68 kthr; 2 running 2 [|||||||||||||||| 22.4%] Load average: 2.07 4.13 4.75 3 [||||||||||||| 17.8%] Uptime: 02:28:08 4 [||||| 5.8%] Mem[||||| 706/32205MB] Swp[ 0/8190MB] CPU: Intel(R) Xeon(R) CPU E5-2407 0 @ 2.20GHz Debian 7 Jessie 3.16

— Reply to this email directly or view it on GitHub https://github.com/ntop/PF_RING/issues/7#issuecomment-101619671.

pavel-odintsov commented 9 years ago

Work really well! Wire speed achieved! Thanks!

./zcount -c 0 -i zc:eth4 -a
Absolute Stats: 303'618'151 pkts (0 drops) - 25'503'924'684 bytes
Actual Stats: 0.00 pps (0.00 drops) - 0.00 Gbps

Could you explain when I should use wait_for_packet = 1 instead of default zero value?

cardigliano commented 9 years ago

This depends on your applicaiton, wait_for_packet is blocking and uses passive wait.

Alfredo

On 13 May 2015, at 12:51, Pavel Odintsov notifications@github.com wrote:

Work really well! Wire speed achieved! Thanks!

./zcount -c 0 -i zc:eth4 -a Absolute Stats: 303'618'151 pkts (0 drops) - 25'503'924'684 bytes Actual Stats: 0.00 pps (0.00 drops) - 0.00 Gbps Could you explain when I should use wait_for_packet = 1 instead of default zero value?

— Reply to this email directly or view it on GitHub https://github.com/ntop/PF_RING/issues/7#issuecomment-101624514.

pavel-odintsov commented 9 years ago

Roger that! Thanks!