search

ntop / PF_RING

High-speed packet processing framework
http://www.ntop.org
GNU Lesser General Public License v2.1
2.67k stars 353 forks source link

Not able to go beyond 20-22Gbps using disk2n and Napatech NIC #555

Closed mdiqbalahmad closed 4 years ago

mdiqbalahmad commented 4 years ago

Hi

I have installed disk2n(v.3.4.200220) on RHEL 7.5 server. This server has Napatech 40Gbps card on it. We want to repaly pcaps at 40Gbps on this card using disk2n. The other end is also connected to Napatech NIC of 40Gbpos. Using PF_RING v.7.6.0.

The sever has 8 SSD on Raid 0 configuration.

We are expecting to replay pcap traffic of 40Gbps but not able to reach beyond 22Gbps. Could you help in reaching the traffic up to 40Gbps?

Command used : /usr/local/bin/disk2n -m pcap-list-singtel.txt -i nt:0 -I 100 -g -C 65535 -b 40960 -U -c 1 -w 2 -S 3 -v

disk2nlog.txt

cardigliano commented 4 years ago

@mdiqbalahmad it seems the disk is fast enough reading traffic (at least it is faster then the transmission), please check the max transmission speed using pfsend -i nt:0 (you can use -l to change the avg packet size, I suggest to test it with -l 64 and -l 1500)

mdiqbalahmad commented 4 years ago

Hi Alfredo

I can see max TX of 40Gbps at -l 570. How do I achieve this using disk2n?

pfsend -i nt:0 -l 64

Sending packets on nt:0 #########################################################################

ERROR: You do not seem to have a valid PF_RING-Napatech 7.6.0.200220

license for port 0 (S/N 840-0044-01-10-0000-243228 / MAC 00.0D.E9.06.6C.D6)

ERROR: Please get one at http://shop.ntop.org/.

#########################################################################

We're now working in demo mode with packet capture and

transmission limited to 5 minutes

######################################################################### Using PF_RING v.7.6.0 Dumping statistics on /proc/net/pf_ring/stats/18765-none.35 TX rate: [current 26'607'666.37 pps/18.73 Gbps][average 26'607'666.37 pps/18.73 Gbps][total 26'608'438.00 pkts] TX rate: [current 29'271'415.07 pps/20.61 Gbps][average 27'939'558.03 pps/19.67 Gbps][total 55'881'463.00 pkts] TX rate: [current 29'286'181.83 pps/20.62 Gbps][average 28'388'431.74 pps/19.99 Gbps][total 85'168'787.00 pkts] TX rate: [current 29'290'749.07 pps/20.62 Gbps][average 28'614'011.07 pps/20.14 Gbps][total 114'460'737.00 pkts] TX rate: [current 29'285'557.14 pps/20.62 Gbps][average 28'748'319.96 pps/20.24 Gbps][total 143'747'407.00 pkts] ^CLeaving... TX rate: [current 29'293'308.96 pps/20.62 Gbps][average 28'826'485.47 pps/20.29 Gbps][total 168'272'937.00 pkts] Sent 168272937 packets [root@loadtestsrv2 ~]# pfsend -i nt:0 -l 1500 Sending packets on nt:0 #########################################################################

ERROR: You do not seem to have a valid PF_RING-Napatech 7.6.0.200220

license for port 0 (S/N 840-0044-01-10-0000-243228 / MAC 00.0D.E9.06.6C.D6)

ERROR: Please get one at http://shop.ntop.org/.

#########################################################################

We're now working in demo mode with packet capture and

transmission limited to 5 minutes

######################################################################### Using PF_RING v.7.6.0 Dumping statistics on /proc/net/pf_ring/stats/18799-none.36 TX rate: [current 2'566'526.73 pps/31.29 Gbps][average 2'566'526.73 pps/31.29 Gbps][total 2'566'614.00 pkts] TX rate: [current 2'835'262.06 pps/34.57 Gbps][average 2'700'895.81 pps/32.93 Gbps][total 5'402'032.00 pkts] TX rate: [current 2'833'269.00 pps/34.54 Gbps][average 2'745'020.48 pps/33.47 Gbps][total 8'235'454.00 pkts] TX rate: [current 2'838'014.42 pps/34.60 Gbps][average 2'768'269.04 pps/33.75 Gbps][total 11'073'616.00 pkts] ^CLeaving... TX rate: [current 2'834'500.14 pps/34.56 Gbps][average 2'779'366.66 pps/33.89 Gbps][total 13'355'913.00 pkts] Sent 13355913 packets

our average packet size is around 570 Bytes:

pfsend -i nt:0 -l 570

Sending packets on nt:0 #########################################################################

ERROR: You do not seem to have a valid PF_RING-Napatech 7.6.0.200220

license for port 0 (S/N 840-0044-01-10-0000-243228 / MAC 00.0D.E9.06.6C.D6)

ERROR: Please get one at http://shop.ntop.org/.

#########################################################################

We're now working in demo mode with packet capture and

transmission limited to 5 minutes

######################################################################### Using PF_RING v.7.6.0 Dumping statistics on /proc/net/pf_ring/stats/19128-none.40 TX rate: [current 7'781'103.47 pps/36.98 Gbps][average 7'781'103.47 pps/36.98 Gbps][total 7'781'298.00 pkts] TX rate: [current 8'417'613.36 pps/40.00 Gbps][average 8'099'363.82 pps/38.49 Gbps][total 16'199'408.00 pkts] TX rate: [current 8'417'544.79 pps/40.00 Gbps][average 8'205'424.43 pps/38.99 Gbps][total 24'617'340.00 pkts] TX rate: [current 8'416'732.24 pps/40.00 Gbps][average 8'258'251.45 pps/39.24 Gbps][total 33'034'451.00 pkts] TX rate: [current 8'417'982.77 pps/40.00 Gbps][average 8'290'197.77 pps/39.40 Gbps][total 41'452'821.00 pkts] TX rate: [current 8'417'318.63 pps/40.00 Gbps][average 8'311'384.58 pps/39.50 Gbps][total 49'870'510.00 pkts] TX rate: [current 8'416'414.09 pps/39.99 Gbps][average 8'326'388.78 pps/39.57 Gbps][total 58'287'286.00 pkts]

cardigliano commented 4 years ago

Could you try setting -I 1 in disk2n?

mdiqbalahmad commented 4 years ago

Still the throughput is around 23Gbps -# /usr/local/bin/disk2n -m pcap-list-singtel.txt -i nt:0 -I 1 -g -C 65535 -b 40960 -U -c 1 -w 2 -S 3 -v

disk2n-1.txt

mdiqbalahmad commented 4 years ago

When using pfsend -i nt:0 -g 3 -l 570 traffic is 40Gbps(this is the maximum line rate of Napatech card)

mdiqbalahmad commented 4 years ago

Hi Alfredo,

Do you have any suggestion for this.

-Iqbal

cardigliano commented 4 years ago

@mdiqbalahmad in theory disk2n should be able to achieve similar performance of pfsend, however there is some factor that could lead to lower performance, for instance the fact that it is using more memory bandwidth to move data from disk. Please 1. try replaying a small pcap and 2. provide some hw specs (output from "cat /proc/cpuinfo" and "dmidecode" should be enough)

cardigliano commented 4 years ago

Any update on this?

mdiqbalahmad commented 4 years ago

Hi Alfredo,

Playing small pcaps also did not help. Below is the cpuinfo and dmidecode output.

dmidecode.txt

cpuinfo.txt

cardigliano commented 4 years ago

Please note that the RAM configuration is not optimal, your CPU has 4 memory channels, but only 2 memory sticks installed on each CPU.

Please also provide 5-6 lines from the output of the below commands: n2membenchmark -c 0 n2membenchmark -c 6

mdiqbalahmad commented 4 years ago

Do I need to fill all the 4 memory channels with memory sticks? for optimal throughput.

Please find the required output

= # n2membenchmark -c 0 28111052.556317 pps/14.392859 Gbps 28151615.506220 pps/14.413627 Gbps 28314233.545257 pps/14.496888 Gbps 28105231.475047 pps/14.389879 Gbps 28066210.351586 pps/14.369900 Gbps 28116749.746261 pps/14.395775 Gbps 28303909.931873 pps/14.491602 Gbps 28083216.421694 pps/14.378607 Gbps 28138064.063245 pps/14.406688 Gbps 28322886.438770 pps/14.501318 Gbps 28134127.247039 pps/14.404673 Gbps

cardigliano commented 4 years ago

Do I need to fill all the 4 memory channels with memory sticks? for optimal throughput.

Yes

n2membenchmark -c 0

28134127.247039 pps/14.404673 Gbps

n2membenchmark -c 6

28471389.343328 pps/14.577351 Gbps

You should see those values increasing after adding 2 more memory sticks per CPU