search

ntop / PF_RING

High-speed packet processing framework
http://www.ntop.org
GNU Lesser General Public License v2.1
2.7k stars 349 forks source link

[Ph7-17] Packet loss seen at /proc/net/pf_ring/*ens2f0* on RHEL 7.9 #825

Closed dharmen8 closed 2 years ago

dharmen8 commented 2 years ago

Hello Team,

Hope you are doing good.

Background

We have all our production servers running in below software combination and we haven't seen frequent Tot Pkt Lost counter incremented in /proc/net/pf_ring/*eth0* stats file.

1). Server     :- HP DL 380 Gen9, 12 core @ 2.4 GHz
2). OS         :- RHEL 6.7
3). PF_RING    :- 6.6.0
4). PF_RING ZC :- 6.6.0
5). ixgbe      :- 4.1.5
6). 10G NIC    :- HP Ethernet 10Gb 2-port 560SPF+ Adapter, 82599ES
7). Mobileum Proxy Application built on the above dependent software version (proxy application that receives the GTP packets from the network).

Due to the RHEL 6.7 is EOM, end customer has requested us to move to "RHEL 7.9". Thus to fulfill their requirement, on our pilot/staging server, we have installed the following HW & SW combination;

1). Server     :- HP DL 380 Gen8, 16 core, 32 Threads @ 2.00 GHz
2). OS         :- RHEL 7.9
3). PF_RING    :- 7.5.0
4). PF_RING ZC :- 7.5.0
5). ixgbe      :- 5.3.7
6). 10G NIC    :- HP Ethernet 10Gb 2-port 560SPF+ Adapter, 82599ES

Both Production & Staging setup is currently getting the same GTP Traffic with throughput on 10g NIC port (ens2f0) as below:

1). During lean hours :- 300 ~ 500 Mbps
2). During avg. hours :- 500 ~ 900 Mbps
3). During peak hours :- 950 ~ 1200 Mbps
4). Peak Traffic observed (daily) :- 2100 ~ 2200 Mbps 

NOTE: 
1). Above traffic is nothing, System is designed and dimensioned to handle 2 times the current traffic.
2). Prior to COVID 19, Traffic used to be avg. 2.5 Gbps and Peaks at 3.6 Gbps on RHEL 6.7 setup.
3). And we did not observe the "Tot Pkt Lost" counter incremented in "/proc/net/pf_ring/*eth0*".

Issue & support required.

On our pilot system with new software shared by our Partners, i am experiencing frequent Tot Pkt Lost counter incremented in /proc/net/pf_ring/*ens2f0* stats file on an average, 1 packet per day. Issue is currently under investigation by our Partner for past two weeks and root cause is unknown.

One Packet Lost a day is not huge drop, but I wonder why production does not have this issue, when it is processing the same traffic as staging setup. To me issue is surely not caused due to staging hardware.

Now to investigate the issue further and identify the root cause, I am seeking out to you'll for support; 1) Under what known conditions would Pkt Lost counter be incremented? 2) Is the observed issue due to known/unknown limitation in pf_ring/zc in its current configuration and setup. 3) How to identify root cause of Pkt Lost. 4) What can we do to avoid Pkt Lost.

Quick Information on Staging Setup

1). Server BIOS is configured in performance mode.

System Option > Processor Options | Hyper Threading | Enabled
System Option > Processor Options | Intel Virtualization (VT-d and VT-x) | Enabled
System Option > Processor Options | Turbo boost | Optimized for performance
Basic Power Mngt Option | HPE Power Profile | Maximum Performance
Basic Power Mngt Option | HPE Power Regulator | HPE Static High Performance Mode
Basic Power Mngt Option | Redundant Power Supply | High Efficient Mode - Auto
Advance Power Mngt. Option | Minimum Processor Idle Power Core C-State | No C-Stat
Advance Power Mngt. Option | Minimum Processor Idle Power Package C-State | No Package State
Advance Power Mngt. Option | Energy/Performance Bias | Maximum Performance
Advance Power Mngt. Option | Collaborative Power Control | Disabled
Advance Power Mngt. Option | DIMM Voltage Preference | Optimized for performance
Advance Power Mngt. Option | Dynamic Power Capping Functionality | Disabled
Advance Power Mngt. Option | Dynamic Power Savings Mode Response | Fast
Service Options | Processor Power and Utilization Monitoring | Disabled
Service Options | Memory Patrol Scrubbing | Disabled
Service Options | Memory Refresh Rate | 1 x Refresh
Misc. Options | Memory Power Savings Mode | Maximum Performance
Misc. Options | Memory Pre-Failure Notification | Disabled
Misc. Options | Intel QPI Link Power Management | Disabled
Misc. Options | ACPI SLIT Preferences | Enabled

2). OS GRUB is configured for performance mode.

BOOT_IMAGE=/vmlinuz-3.10.0-1160.el7.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet processor.max_cstate=1 intel_idle.max_cstate=0 nosoftlockup mce=ignore_ce isolcpus=1-4 audit=0 idle=poll transparent_hugepage=never

3). Defragmentation disabled in pf_ring and enabled in proxy application.

pf_ring.ko min_num_slots=409600 enable_tx_capture=0 enable_frag_coherence=0 enable_ip_defrag=0

4). Following tuning is done when loading the module

    ethtool -G $IF rx 32768 tx 32768
    ethtool -K $IF rxvlan off txvlan off rx-vlan-filter off
    ethtool -K $IF gro on lro on
    ethtool -L $IF combined 4
    ethtool -C $IF rx-usecs 50
    ethtool -A $IF rx off tx off autoneg off
    ethtool -K $IF ntuple off

5). RSS set to 4

    ixgbe.ko RSS=4,4,4,4

6). Huge pages are created with below options

    HUGEPAGES=2560
    HUGEPAGES_SIZE=2048
    echo $HUGEPAGES > /sys/devices/system/node/node0/hugepages/hugepages-$(HUGEPAGES_SIZE)kB/nr_hugepages
    mount -t hugetlbfs -o gid=2000,pagesize=2M,size=10G nodev /mnt/huge

7). Proxy application is not opening the socket in ZC mode. 8). Proxy application receives the GTP packets from network. And below call back is used by the application to receive the packet. rcvLen = pfring_loop(tData->m_pcapHandle, CGTPPFRingHandler::ReceivePFRingPacket, (u_char*)tData, 1);

Detailed information

Furthermore, please find the detailed system level output and stats from the staging server where we are observing the problem. Kindly let me know, if any further details are required.

  1. numactl --hardware 
    available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23
node 0 size: 31972 MB
node 0 free: 16514 MB
node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31
node 1 size: 32250 MB
node 1 free: 29910 MB
node distances:
node   0   1
0:  10  21
1:  21  10
  2. sysctl -a | grep huge 
    vm.hugepages_treat_as_movable = 0
vm.hugetlb_shm_group = 0
vm.nr_hugepages = 2560
vm.nr_hugepages_mempolicy = 2560
vm.nr_overcommit_hugepages = 0
  3. cat /proc/meminfo | grep -i huge 
    AnonHugePages:         0 kB
HugePages_Total:    2560
HugePages_Free:     2560
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
  4. cat /proc/net/pf_ring/info 
    
PF_RING Version          : 7.5.0 (unknown)
Total rings              : 3

Standard (non ZC) Options Ring slots : 409600 Slot version : 17 Capture TX : No [RX only] IP Defragment : No Socket Mode : Standard

5. cat /proc/net/pf_ring/dev/ens2f0/info

Name: ens2f0 Index: 12 Address: 14:02:EC:96:1D:44 Polling Mode: NAPI/ZC Type: Ethernet Family: Intel ixgbe 82599 TX Queues: 4 RX Queues: 4 Num RX Slots: 32768 Num TX Slots: 32768

6. ethtool -i ens2f0

driver: ixgbe version: 5.3.7 firmware-version: 0x80000897, 1.1446.0 expansion-rom-version: bus-info: 0000:07:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: yes

7. ethtool -l ens2f0

Channel parameters for ens2f0: Pre-set maximums: RX: 0 TX: 0 Other: 1 Combined: 63 Current hardware settings: RX: 0 TX: 0 Other: 1 Combined: 4

8. ethtool -g ens2f0

Ring parameters for ens2f0: Pre-set maximums: RX: 32768 RX Mini: 0 RX Jumbo: 0 TX: 32768 Current hardware settings: RX: 32768 RX Mini: 0 RX Jumbo: 0 TX: 32768

9. ethtool -a ens2f0

Pause parameters for ens2f0: Autonegotiate: off RX: off TX: off

10. ethtool -k ens2f0

Features for ens2f0: rx-checksumming: on tx-checksumming: on tx-checksum-ipv4: off [fixed] tx-checksum-ip-generic: on tx-checksum-ipv6: off [fixed] tx-checksum-fcoe-crc: on [fixed] tx-checksum-sctp: on scatter-gather: on tx-scatter-gather: on tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: on tx-tcp-segmentation: on tx-tcp-ecn-segmentation: off [fixed] tx-tcp6-segmentation: on tx-tcp-mangleid-segmentation: off udp-fragmentation-offload: off [fixed] generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: on rx-vlan-offload: off tx-vlan-offload: off ntuple-filters: off receive-hashing: on highdma: on [fixed] rx-vlan-filter: off vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: on [fixed] tx-gre-segmentation: on tx-ipip-segmentation: off [fixed] tx-sit-segmentation: off [fixed] tx-udp_tnl-segmentation: on fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] busy-poll: on [fixed] tx-gre-csum-segmentation: on tx-udp_tnl-csum-segmentation: on tx-gso-partial: on tx-sctp-segmentation: off [fixed] rx-gro-hw: off [fixed] l2-fwd-offload: off [fixed] hw-tc-offload: off rx-udp_tunnel-port-offload: on

11. ethtool -c ens2f0

Coalesce parameters for ens2f0: Adaptive RX: off TX: off stats-block-usecs: 0 sample-interval: 0 pkt-rate-low: 0 pkt-rate-high: 0

rx-usecs: 50 rx-frames: 0 rx-usecs-irq: 0 rx-frames-irq: 0

tx-usecs: 0 tx-frames: 0 tx-usecs-irq: 0 tx-frames-irq: 256

rx-usecs-low: 0 rx-frame-low: 0 tx-usecs-low: 0 tx-frame-low: 0

rx-usecs-high: 0 rx-frame-high: 0 tx-usecs-high: 0 tx-frame-high: 0

12. Max CPU consumed by proxy application threads is 14%

`top -H -b -p $proxy_app_pid >> $log `

13. ethtool -S ens2f0 | grep -v ': 0'

NIC statistics: rx_packets: 7992471951 tx_packets: 9 rx_bytes: 6127133696805 tx_bytes: 802 multicast: 703762 rx_pkts_nic: 7992537913 tx_pkts_nic: 9 rx_bytes_nic: 6127181001087 tx_bytes_nic: 802 lsc_int: 3 broadcast: 638 hw_rsc_aggregated: 22400 hw_rsc_flushed: 22399 fdir_miss: 7574164553 tx_queue_1_packets: 2 tx_queue_1_bytes: 140 tx_queue_3_packets: 7 tx_queue_3_bytes: 626 rx_queue_0_packets: 1990222904 rx_queue_0_bytes: 1467612165927 rx_queue_1_packets: 1969948736 rx_queue_1_bytes: 1527846066068 rx_queue_2_packets: 2009160152 rx_queue_2_bytes: 1578695928600 rx_queue_3_packets: 2023206120 rx_queue_3_bytes: 1521056688790

14. grep -F Lost /proc/net/pf_ring/*$proxy_app_pid*

[root@CHOIPPROBE06 ~]# grep -F Lost /proc/net/pf_ring/7079-ens2f0.* /proc/net/pf_ring/7079-ens2f0.1:Tot Pkt Lost : 1 /proc/net/pf_ring/7079-ens2f0.2:Tot Pkt Lost : 1 /proc/net/pf_ring/7079-ens2f0.3:Tot Pkt Lost : 2

15. ifconfig ens2f0

ens2f0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500 inet6 fe80::1602:ecff:fe96:1d44 prefixlen 64 scopeid 0x20 ether 14:02:ec:96:1d:44 txqueuelen 1000 (Ethernet) RX packets 8022271170 bytes 6148633203365 (5.5 TiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 9 bytes 802 (802.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device memory 0xf7f00000-f8000000

16.  affinity of IRQs of ens2f0 
     command `watch -td1 'egrep -i 'CPU|ens2f0' /proc/interrupts` 
        CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       CPU8       CPU9       CPU10      CPU11      CPU12      CPU13      CPU14      CPU15      CPU16      CPU17      CPU18      CPU19      CPU20      CPU21      CPU22      CPU23      CPU24      CPU25      CPU26      CPU27      CPU28      CPU29      CPU30      CPU31

84: 15 827752973 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge ens2f0-TxRx-0 85: 17 0 826867424 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge ens2f0-TxRx-1 86: 15 0 0 835615806 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge ens2f0-TxRx-2 87: 20 0 0 0 831170356 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge ens2f0-TxRx-3 88: 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge ens2f0

17. cat /proc/cmdline

/vmlinuz-3.10.0-1160.el7.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet processor.max_cstate=1 intel_idle.max_cstate=0 nosoftlockup mce=ignore_ce isolcpus=1-4 audit=0 idle=poll transparent_hugepage=never


18. numactl -C31 sysctl -a

<< filtered text, refer to attached pfring_packet_lost_details.txt >> net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.core.rmem_default = 1048576 net.core.wmem_default = 1048576 net.ipv4.tcp_rmem = 4096 1048576 16777216 net.ipv4.tcp_wmem = 4096 1048576 16777216 net.ipv4.tcp_sack = 0 net.ipv4.tcp_adv_win_scale = 1 net.ipv4.ipfrag_high_thresh = 134217728 net.ipv4.ipfrag_low_thresh = 100663296 net.ipv4.ipfrag_max_dist = 0 net.ipv4.ipfrag_time = 10 net.core.netdev_max_backlog = 2000 vm.dirty_ratio = 10 vm.dirty_background_ratio = 3 vm.swappiness = 10 kernel.sched_min_granularity_ns = 5000000 kernel.sched_wakeup_granularity_ns = 7000000 kernel.shmall = 12330792 kernel.shmmax = 50506924032 kernel.msgmnb = 65536 net.ipv4.tcp_syncookies = 1 kernel.sysrq = 0 net.ipv4.conf.default.accept_source_route = 0 net.ipv4.conf.default.rp_filter = 1 net.ipv4.ip_forward = 0

19. lscpu

Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Thread(s) per core: 2 Core(s) per socket: 8 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 45 Model name: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz Stepping: 7 CPU MHz: 1995.000 BogoMIPS: 3990.00 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 20480K NUMA node0 CPU(s): 0-7,16-23 NUMA node1 CPU(s): 8-15,24-31 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm epb ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts md_clear spec_ctrl intel_stibp flush_l1d

20. lspci

00:00.0 Host bridge: Intel Corporation Xeon E5/Core i7 DMI2 (rev 07) 00:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1a (rev 07) 00:01.1 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1b (rev 07) 00:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a (rev 07) 00:02.1 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2b (rev 07) 00:02.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2c (rev 07) 00:02.3 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2d (rev 07) 00:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode (rev 07) 00:03.1 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3b (rev 07) 00:03.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3c (rev 07) 00:03.3 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3d (rev 07) 00:04.0 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 0 (rev 07) 00:04.1 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 1 (rev 07) 00:04.2 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 2 (rev 07) 00:04.3 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 3 (rev 07) 00:04.4 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 4 (rev 07) 00:04.5 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 5 (rev 07) 00:04.6 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 6 (rev 07) 00:04.7 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 7 (rev 07) 00:05.0 System peripheral: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management (rev 07) 00:05.2 System peripheral: Intel Corporation Xeon E5/Core i7 Control Status and Global Errors (rev 07) 00:05.4 PIC: Intel Corporation Xeon E5/Core i7 I/O APIC (rev 07) 00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Virtual Root Port (rev 05) 00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 05) 00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 (rev b5) 00:1c.7 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 8 (rev b5) 00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 05) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5) 00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller (rev 05) 00:1f.2 IDE interface: Intel Corporation C600/X79 series chipset 4-Port SATA IDE Controller (rev 05) 01:00.0 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support (rev 05) 01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200EH 01:00.2 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging (rev 05) 01:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (rev 02) 02:00.0 RAID bus controller: Hewlett-Packard Company Smart Array Gen8 Controllers (rev 01) 03:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01) 03:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01) 03:00.2 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01) 03:00.3 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01) 07:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 07:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 0a:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01) 0a:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01) 0a:00.2 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01) 0a:00.3 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01) 1f:08.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0 (rev 07) 1f:08.3 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0 (rev 07) 1f:08.4 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0 (rev 07) 1f:09.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1 (rev 07) 1f:09.3 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1 (rev 07) 1f:09.4 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1 (rev 07) 1f:0a.0 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 0 (rev 07) 1f:0a.1 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 1 (rev 07) 1f:0a.2 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 2 (rev 07) 1f:0a.3 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 3 (rev 07) 1f:0b.0 System peripheral: Intel Corporation Xeon E5/Core i7 Interrupt Control Registers (rev 07) 1f:0b.3 System peripheral: Intel Corporation Xeon E5/Core i7 Semaphore and Scratchpad Configuration Registers (rev 07) 1f:0c.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) 1f:0c.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) 1f:0c.2 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) 1f:0c.3 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) 1f:0c.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 0 (rev 07) 1f:0c.7 System peripheral: Intel Corporation Xeon E5/Core i7 System Address Decoder (rev 07) 1f:0d.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) 1f:0d.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) 1f:0d.2 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) 1f:0d.3 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) 1f:0d.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 1 (rev 07) 1f:0e.0 System peripheral: Intel Corporation Xeon E5/Core i7 Processor Home Agent (rev 07) 1f:0e.1 Performance counters: Intel Corporation Xeon E5/Core i7 Processor Home Agent Performance Monitoring (rev 07) 1f:0f.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers (rev 07) 1f:0f.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller RAS Registers (rev 07) 1f:0f.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 0 (rev 07) 1f:0f.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 1 (rev 07) 1f:0f.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 2 (rev 07) 1f:0f.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 3 (rev 07) 1f:0f.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 4 (rev 07) 1f:10.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0 (rev 07) 1f:10.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 1 (rev 07) 1f:10.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 0 (rev 07) 1f:10.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 1 (rev 07) 1f:10.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 2 (rev 07) 1f:10.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 3 (rev 07) 1f:10.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 2 (rev 07) 1f:10.7 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 3 (rev 07) 1f:11.0 System peripheral: Intel Corporation Xeon E5/Core i7 DDRIO (rev 07) 1f:13.0 System peripheral: Intel Corporation Xeon E5/Core i7 R2PCIe (rev 07) 1f:13.1 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to PCI Express Performance Monitor (rev 07) 1f:13.4 Performance counters: Intel Corporation Xeon E5/Core i7 QuickPath Interconnect Agent Ring Registers (rev 07) 1f:13.5 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 0 Performance Monitor (rev 07) 1f:13.6 System peripheral: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 1 Performance Monitor (rev 07) 20:00.0 PCI bridge: Intel Corporation Xeon E5/Core i7 DMI2 in PCI Express Mode (rev 07) 20:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1a (rev 07) 20:01.1 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1b (rev 07) 20:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a (rev 07) 20:02.1 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2b (rev 07) 20:02.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2c (rev 07) 20:02.3 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2d (rev 07) 20:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode (rev 07) 20:03.1 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3b (rev 07) 20:03.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3c (rev 07) 20:03.3 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3d (rev 07) 20:04.0 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 0 (rev 07) 20:04.1 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 1 (rev 07) 20:04.2 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 2 (rev 07) 20:04.3 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 3 (rev 07) 20:04.4 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 4 (rev 07) 20:04.5 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 5 (rev 07) 20:04.6 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 6 (rev 07) 20:04.7 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 7 (rev 07) 20:05.0 System peripheral: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management (rev 07) 20:05.2 System peripheral: Intel Corporation Xeon E5/Core i7 Control Status and Global Errors (rev 07) 20:05.4 PIC: Intel Corporation Xeon E5/Core i7 I/O APIC (rev 07) 3f:08.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0 (rev 07) 3f:08.3 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0 (rev 07) 3f:08.4 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0 (rev 07) 3f:09.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1 (rev 07) 3f:09.3 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1 (rev 07) 3f:09.4 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1 (rev 07) 3f:0a.0 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 0 (rev 07) 3f:0a.1 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 1 (rev 07) 3f:0a.2 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 2 (rev 07) 3f:0a.3 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 3 (rev 07) 3f:0b.0 System peripheral: Intel Corporation Xeon E5/Core i7 Interrupt Control Registers (rev 07) 3f:0b.3 System peripheral: Intel Corporation Xeon E5/Core i7 Semaphore and Scratchpad Configuration Registers (rev 07) 3f:0c.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) 3f:0c.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) 3f:0c.2 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) 3f:0c.3 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) 3f:0c.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 0 (rev 07) 3f:0c.7 System peripheral: Intel Corporation Xeon E5/Core i7 System Address Decoder (rev 07) 3f:0d.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) 3f:0d.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) 3f:0d.2 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) 3f:0d.3 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 07) 3f:0d.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 1 (rev 07) 3f:0e.0 System peripheral: Intel Corporation Xeon E5/Core i7 Processor Home Agent (rev 07) 3f:0e.1 Performance counters: Intel Corporation Xeon E5/Core i7 Processor Home Agent Performance Monitoring (rev 07) 3f:0f.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers (rev 07) 3f:0f.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller RAS Registers (rev 07) 3f:0f.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 0 (rev 07) 3f:0f.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 1 (rev 07) 3f:0f.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 2 (rev 07) 3f:0f.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 3 (rev 07) 3f:0f.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 4 (rev 07) 3f:10.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0 (rev 07) 3f:10.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 1 (rev 07) 3f:10.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 0 (rev 07) 3f:10.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 1 (rev 07) 3f:10.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 2 (rev 07) 3f:10.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 3 (rev 07) 3f:10.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 2 (rev 07) 3f:10.7 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 3 (rev 07) 3f:11.0 System peripheral: Intel Corporation Xeon E5/Core i7 DDRIO (rev 07) 3f:13.0 System peripheral: Intel Corporation Xeon E5/Core i7 R2PCIe (rev 07) 3f:13.1 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to PCI Express Performance Monitor (rev 07) 3f:13.4 Performance counters: Intel Corporation Xeon E5/Core i7 QuickPath Interconnect Agent Ring Registers (rev 07) 3f:13.5 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 0 Performance Monitor (rev 07) 3f:13.6 System peripheral: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 1 Performance Monitor (rev 07)


21. dmidecode 

<< refer to attached pfring_packet_lost_details.txt >>

22. cat /proc/interrupts

<< refer to attached pfring_packet_lost_details.txt >>

23. cat /etc/redhat-release

Red Hat Enterprise Linux Server release 7.9 (Maipo)

24. numactl -C31 rpm -qa 

<< refer to attached pfring_packet_lost_details.txt >>

25. numactl -C31 systemctl list-unit-files

<< refer to attached pfring_packet_lost_details.txt >>

26. numactl -C31 turbostat --debug run

<< refer to attached pfring_packet_lost_details.txt >>

27. cat /proc/mounts

sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,nosuid,size=32869868k,nr_inodes=8217467,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpuacct,cpu 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_prio,net_cls 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 configfs /sys/kernel/config configfs rw,relatime 0 0 /dev/mapper/rhel-root / ext4 rw,noatime,nodiratime,stripe=128,data=ordered 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=25,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=35038 0 0 debugfs /sys/kernel/debug debugfs rw,relatime 0 0 mqueue /dev/mqueue mqueue rw,relatime 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0 /dev/sda1 /boot xfs rw,relatime,attr2,inode64,logbsize=256k,sunit=512,swidth=1024,noquota 0 0 /dev/mapper/rhel-var /var ext4 rw,noatime,nodiratime,stripe=128,data=ordered 0 0 /dev/mapper/rhel-tmp /tmp ext4 rw,noatime,nodiratime,stripe=128,data=ordered 0 0 /dev/mapper/rhel-opt /opt ext4 rw,noatime,nodiratime,stripe=128,data=ordered 0 0 /dev/mapper/rhel-u01 /u01 ext4 rw,noatime,nodiratime,stripe=128,data=ordered 0 0 /dev/mapper/rhel-opt_Roamware_logs /opt/Roamware/logs ext4 rw,noatime,nodiratime,stripe=128,data=ordered 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 nodev /mnt/huge hugetlbfs rw,relatime,gid=2000,pagesize=2M,size=10G 0 0 tmpfs /run/user/2000 tmpfs rw,nosuid,nodev,relatime,size=6576424k,mode=700,uid=2000,gid=2000 0 0

28. ethtool -x ens2f0

RX flow hash indirection table for ens2f0 with 4 RX ring(s): 0: 0 1 2 3 0 1 2 3 8: 0 1 2 3 0 1 2 3 16: 0 1 2 3 0 1 2 3 24: 0 1 2 3 0 1 2 3 32: 0 1 2 3 0 1 2 3 40: 0 1 2 3 0 1 2 3 48: 0 1 2 3 0 1 2 3 56: 0 1 2 3 0 1 2 3 64: 0 1 2 3 0 1 2 3 72: 0 1 2 3 0 1 2 3 80: 0 1 2 3 0 1 2 3 88: 0 1 2 3 0 1 2 3 96: 0 1 2 3 0 1 2 3 104: 0 1 2 3 0 1 2 3 112: 0 1 2 3 0 1 2 3 120: 0 1 2 3 0 1 2 3 RSS hash key: 27:f1:ed:f0:50:15:d8:9a:cc:8f:3b:62:50:b3:a1:6d:1b:ca:0b:d4:44:f9:64:22:71:46:80:98:be:61:45:af:b3:c8:af:ea:89:e8:a4:73 RSS hash function: toeplitz: on xor: off crc32: off

29. ethtool -u ens2f0

4 RX rings available Total 0 rules

30. lsmem

RANGE SIZE STATE REMOVABLE BLOCK 0x0000000000000000-0x0000000007ffffff 128M online no 0 0x0000000008000000-0x0000000027ffffff 512M online yes 1-4 0x0000000028000000-0x0000000037ffffff 256M online no 5-6 0x0000000038000000-0x00000000b7ffffff 2G online yes 7-22 0x00000000b8000000-0x00000000bfffffff 128M online no 23 0x0000000100000000-0x0000000137ffffff 896M online yes 32-38 0x0000000138000000-0x000000017fffffff 1.1G online no 39-47 0x0000000180000000-0x00000004afffffff 12.8G online yes 48-149 0x00000004b0000000-0x00000004bfffffff 256M online no 150-151 0x00000004c0000000-0x00000004c7ffffff 128M online yes 152 0x00000004c8000000-0x00000004cfffffff 128M online no 153 0x00000004d0000000-0x00000004d7ffffff 128M online yes 154 0x00000004d8000000-0x00000004efffffff 384M online no 155-157 0x00000004f0000000-0x000000056fffffff 2G online yes 158-173 0x0000000570000000-0x0000000577ffffff 128M online no 174 0x0000000578000000-0x00000005f7ffffff 2G online yes 175-190 0x00000005f8000000-0x00000005ffffffff 128M online no 191 0x0000000600000000-0x0000000617ffffff 384M online yes 192-194 0x0000000618000000-0x00000006e7ffffff 3.3G online no 195-220 0x00000006e8000000-0x00000006efffffff 128M online yes 221 0x00000006f0000000-0x00000006f7ffffff 128M online no 222 0x00000006f8000000-0x000000080fffffff 4.4G online yes 223-257 0x0000000810000000-0x000000083fffffff 768M online no 258-263 0x0000000840000000-0x0000000877ffffff 896M online yes 264-270 0x0000000878000000-0x00000008bfffffff 1.1G online no 271-279 0x00000008c0000000-0x0000000fa7ffffff 27.6G online yes 280-500 0x0000000fa8000000-0x0000000fb7ffffff 256M online no 501-502 0x0000000fb8000000-0x0000000fcfffffff 384M online yes 503-505 0x0000000fd0000000-0x000000103fffffff 1.8G online no 506-519

31. cat /etc/sysconfig/modules/pf_ring-3.10.0-1160.el7.x86_64-7.5.0.modules

<< refer to attached pfring_packet_lost_details.txt >>

32. cat /etc/sysconfig/modules/pf_ring-zc-ixgbe-3.10.0-1160.el7.x86_64-7.5.0.modules

<< refer to attached pfring_packet_lost_details.txt >>

33.a. cat /etc/pf_ring/kernel/3.10.0-1160.el7.x86_64/ia_loadModulesixgbe.sh

<< refer to attached pfring_packet_lost_details.txt >>

33.b. cat /etc/pf_ring/kernel/3.10.0-1160.el7.x86_64/ia_loadModulesixgbe.sh

<< refer to attached pfring_packet_lost_details.txt >>

34. cronjob 

<< refer to attached pfring_packet_lost_details.txt >>

35. cat /proc/net/pf_ring/*ens2f0*

Note: 
- I have restarted application software yesterday, so "Tot Pkt Lost" is reflected 0 in below output.
- I have configured application 3 read threads, thus you see stats from 3 sockets.

Bound Device(s) | ens2f0 | ens2f0 | ens2f0 Active | 1 | 1 | 1 Breed | Standard | Standard | Standard Appl. Name | | | Socket Mode | RX only | RX only | RX only Capture Direction | RX+TX | RX+TX | RX+TX Sampling Rate | 1 | 1 | 1 Filtering Sampling Rate| 0 | 0 | 0 IP Defragment | No | No | No BPF Filtering | Enabled | Enabled | Enabled Sw Filt Hash Rules | 0 | 0 | 0 Sw Filt WC Rules | 0 | 0 | 0 Sw Filt Hash Match | 0 | 0 | 0 Sw Filt Hash Miss | 0 | 0 | 0 Sw Filt Hash Filtered | 0 | 0 | 0 Hw Filt Rules | 0 | 0 | 0 Poll Pkt Watermark | 128 | 128 | 128 Num Poll Calls | 32070003 | 30920505 | 33174337 Poll Watermark Timeout | 0 | 0 | 0 Channel Id Mask | 0xFFFFFFFFFFFFFFFF | 0xFFFFFFFFFFFFFFFF | 0xFFFFFFFFFFFFFFFF VLAN Id | 65535 | 65535 | 65535 Cluster Id | 1 | 1 | 1 Slot Version | 17 [7.5.0] | 17 [7.5.0] | 17 [7.5.0] Min Num Slots | 409602 | 409602 | 409602 Bucket Len | 1600 | 1600 | 1600 Slot Len | 1648 [bucket+header] | 1648 [bucket+header] | 1648 [bucket+header] Tot Memory | 675033088 | 675033088 | 675033088 Tot Packets | 2120655076 | 2041063899 | 2195874271 Tot Pkt Lost | 0 | 0 | 0 Tot Insert | 2120655076 | 2041063899 | 2195874271 Tot Read | 2120654994 | 2041063778 | 2195874263 Insert Offset | 657178360 | 589047504 | 620717696 Remove Offset | 657123944 | 588950184 | 620712672 Num Free Slots | 409520 | 409481 | 409594 Reflect: Fwd Ok | 0 | 0 | 0 Reflect: Fwd Errors | 0 | 0 | 0



### Attachment

**Attaching following details** for reference and check;
1). detailed output of above commands, pfring_packet_lost_details.txt
2). analysis of packet drop observed on 2022/7/1, analysis_20220701.txt
3). analysis of packet drop observed on 2022/7/3, analysis_20220703.txt

[pfring_packet_lost_details.txt](https://github.com/ntop/PF_RING/files/9051521/pfring_packet_lost_details.txt)
[analysis_20220701.txt](https://github.com/ntop/PF_RING/files/9051522/analysis_20220701.txt)
[analysis_20220703.txt](https://github.com/ntop/PF_RING/files/9051524/analysis_20220703.txt)
cardigliano commented 2 years ago

@dharmen8 I have a few questions:

  1. what application are you running? Are you able to reproduce the same with pfcount?
  2. are you running ZC (using zc:xxx as interface name)? However please note we support latest pf_ring dev (8.3) or stable (8.2)
dharmen8 commented 2 years ago

@cardigliano , please find my comments

  1. I am running proxy application that receives the GTP packets from pf_ring. Application get the packet from below pf-ring call back. rcvLen = pfring_loop(tData->m_pcapHandle, CGTPPFRingHandler::ReceivePFRingPacket, (u_char*)tData, 1);

    I am not able to successfully run pfcount on the server pfring_open error [Invalid argument] (pf_ring not loaded or interface ens2f0 is down ?). I had build pfcount in my vm setup (running same kernel version as the staging server) and migrated pfcount/zcount to the staging server.

    Kindly note, I had received pf_ring & zc source rpms from our partner along with their proxy application. And they did not ship pfcount/zcount utilities in shared source rpms.

  2. We are not opening socket in ZC mode.

    
[root@CHOIPPROBE06 Roamware]# cat /proc/net/pf_ring/info
PF_RING Version          : 7.5.0 (unknown)
Total rings              : 3

Standard (non ZC) Options Ring slots : 409600 Slot version : 17 Capture TX : No [RX only] IP Defragment : No Socket Mode : Standard


Furthermore, after spending some time in past 1 week on going through open/closed issues on pf_ring on github, I have learnt that our current version of pf_ring (7.5) is EOS. I have initiated discussion on this matter with our partner to move their proxy application on current supported stable release (8.2). However, same is going to take time and not going to be available in current project delivery. 

Until proxy application support is available for latest pf_ring sable release, we will have to stick to version 7.5.
dharmen8 commented 2 years ago

@cardigliano ,

Yesterday around 2022/07/07 00:09:29, we pumped 3 x feed triplicate feed into staging Server in an attempt to make sure issue is not caused due to low traffic throughput.

As soon as the 3x feed was open we saw a burst in traffic 900 Mbps -> 2500 Mbps and the system was able to handle that event, no packet drops/lost were observed.

However, at later timestamps, 4 drops are observed, and further details as below.

070722-001501: Drop-1 by FC
TotPktLost:0
TotPktLost:1
TotPktLost:0

070722-010201: Drop-2 by FC
TotPktLost:0
TotPktLost:1
TotPktLost:1

070722-015901: Drop-3 by FC
TotPktLost:0 
TotPktLost:1
TotPktLost:2

070722-080801: Drop-4 by FC
TotPktLost:0 
TotPktLost:2
TotPktLost:2

image image image image image

cardigliano commented 2 years ago

@dharmen8 packet loss at this stage happens when the application is not fast enough to dequeue packets from the receive queue. Since the CPU load is not that high, and the application seems to handle the rate, it seems that for some reason the application from time to time is busy in some activity and stops capturing traffic for a moment, leading to packet loss.

dharmen8 commented 2 years ago

@cardigliano , Thank you for your quick reply.

On RHEL 6.7, with PF_RING 6.6.0, zc ixgbe 4.1.5 and the same application (built on rhel 6.x and lower release), is not experiencing this issue.

Per your experience, is it possible to improve on this issue with configuration change in pf_ring 7.5.0, zc ixgbe 5.3.7 or OS, or the only way ahead is to look into & tune application polling and packet processing logic, please advise?

cardigliano commented 2 years ago

You can try increasing the min_num_slots as explained at https://www.ntop.org/guides/pf_ring/get_started/packages_installation.html

dharmen8 commented 2 years ago

@cardigliano , Thank you for your quick response on raised query.

We have currently configured min_num_slots=409600 both on RHEL 6.7 & RHEL 7.9, can you help to understand:

  1. What is the maximum value supported by min_num_slots. Referring to u_int32_t min_num_slots, would it be 2147483647, please advise?
  2. After knowing current value, would you still recommend to increase min_num_slots, please advise?
  3. Is there any side effect, if I increase size to maximum supported value. Kindly clarify?
cardigliano commented 2 years ago

@dharmen8 the real limit is the amount of memory the system is able to allocate for the ring, the default is 4096, I usually recommend setting up to 100K slots which is a reasonable number to handle spikes. Anyway there is no side effect besides memory utilization.