cardigliano
commented
3 years ago Hi I have a few questions:
- what application are you running? it is a packet forwarding application?
- how did you measure latency?
- are you opening all queues? (I see you have RSS set to 4)
- are you opening the socket in ZC mode? (using the 'zc:' prefix in the interface name)
Regards Alfredo
On 25 Jun 2021, at 18:47, MobileumTech @.***> wrote:
Hi Team,
Hope you are doing good.
We facing issue while using pf-ring zc for data packet processing.
PF-Ring ZC is adding latency while processing GTP GPD-U packets processing from network. We are observing the latency at pf-ring level when it handle the packets at lower throughput.
It looks like PF-Ring is doing some internal buffering before giving the packets to application for processing and as per our observation is only when we have lower throughput.
Please help us to investigate the issue further.
Do we have any configurable param setting at PF-Ring level which control the buffering of packets internally. How we can detect why PF-Ring is introducing the latency before giving the packet to application. For latency check, does PF-Ring provides any KPI and Traps , etc. Please find the system level output and stats from server where we are observing the latency problem.
++++++++++++++++ ens1f0
numactl --hardware
available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23 node 0 size: 65211 MB node 0 free: 16149 MB node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31 node 1 size: 65535 MB node 1 free: 8944 MB node distances: node 0 1 0: 10 21 1: 21 10 @.*** ~]#
sysctl -a | grep huge
@. net]# sysctl -a | grep huge vm.hugepages_treat_as_movable = 0 vm.hugetlb_shm_group = 0 vm.nr_hugepages = 0 vm.nr_hugepages_mempolicy = 0 vm.nr_overcommit_hugepages = 0 @. net]#
cat /proc/meminfo | grep -i huge
@. net]# cat /proc/meminfo | grep -i huge AnonHugePages: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB @. net]#
cat /proc/net/pf_ring/info
@.*** net]# cat /proc/net/pf_ring/info PF_RING Version : 7.5.0 (unknown) Total rings : 17
Standard (non ZC) Options Ring slots : 409600 Slot version : 17 Capture TX : Yes [RX+TX] IP Defragment : Yes Socket Mode : Standard @.*** net]#
cat /proc/net/pf_ring/dev//info
@. net]# cat /proc/net/pf_ring/dev/ens1f0/info Name: ens1f0 Index: 14 Address: 48:DF:37:A5:5E:A0 Polling Mode: NAPI/ZC Type: Ethernet Family: Intel i40e TX Queues: 4 RX Queues: 4 Num RX Slots: 4096 Num TX Slots: 512 @. net]#
ethtool -l
@.*** net]# ethtool -l ens1f0 Channel parameters for ens1f0: Pre-set maximums: RX: 0 TX: 0 Other: 1 Combined: 4 Current hardware settings: RX: 0 TX: 0 Other: 1 Combined: 4
ethtool -g
@.*** net]# ethtool -g ens1f0 Ring parameters for ens1f0: Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096 Current hardware settings: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 512
ethtool -a
@.*** net]# ethtool -a ens1f0 Pause parameters for ens1f0: Autonegotiate: off RX: off TX: off
9 ethtool -k
@.*** net]# ethtool -k ens1f0 Features for ens1f0: rx-checksumming: on tx-checksumming: on tx-checksum-ipv4: on tx-checksum-ip-generic: off [fixed] tx-checksum-ipv6: on tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: on scatter-gather: off tx-scatter-gather: off tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: off tx-tcp-segmentation: off tx-tcp-ecn-segmentation: off tx-tcp6-segmentation: off tx-tcp-mangleid-segmentation: off udp-fragmentation-offload: off [fixed] generic-segmentation-offload: off generic-receive-offload: off large-receive-offload: off [fixed] rx-vlan-offload: off tx-vlan-offload: on ntuple-filters: on receive-hashing: on highdma: on rx-vlan-filter: on [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: off [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: on tx-ipip-segmentation: on tx-sit-segmentation: on tx-udp_tnl-segmentation: on fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] busy-poll: off [fixed] tx-gre-csum-segmentation: on tx-udp_tnl-csum-segmentation: on tx-gso-partial: on tx-sctp-segmentation: off [fixed] rx-gro-hw: off [fixed] l2-fwd-offload: off [fixed] hw-tc-offload: off [fixed] rx-udp_tunnel-port-offload: on ethtool -c
@.*** net]# ethtool -c ens1f0 Coalesce parameters for ens1f0: Adaptive RX: on TX: on stats-block-usecs: 0 sample-interval: 0 pkt-rate-low: 0 pkt-rate-high: 0
rx-usecs: 50 rx-frames: 0 rx-usecs-irq: 0 rx-frames-irq: 256
tx-usecs: 50 tx-frames: 0 tx-usecs-irq: 0 tx-frames-irq: 256
rx-usecs-low: 0 rx-frame-low: 0 tx-usecs-low: 0 tx-frame-low: 0
rx-usecs-high: 0 rx-frame-high: 0 tx-usecs-high: 0 tx-frame-high: 0
11 a) Run a script with the below command to store the output of top command in a file. For ex - create a script with the name top.sh having below contents
cat top.sh
!/usr/bin top -H -b -p $1 >> $2
To run this script
. top.sh
let it run for a day and execute the below command on the generated file to get the max cpu utilization by any of the gtp probe thread.
grep -i | sed 's/^/ /' | tr -s ' ' | cut -d' ' -f10 | sort -n
If cpu thread consumption is more than 70 percent. Then contact the engineering and share the file generated above along with pstack on probe pid with root user.
Maximum CPU comsumption is 0.4%
b) Observe if there is any packet loss in pf_ring stats
cat /proc/net/pf_ring/
* | grep Lost Output of above command will appear like this - Tot Pkt Lost : 0 <--- This number should be zero. Tot Pkt Lost : 0 <--- This number should be zero. ***@***.*** support]# cat /proc/net/pf_ring/32499* | grep Lost Tot Pkt Lost : 0 Tot Pkt Lost : 0 Tot Pkt Lost : 0 Tot Pkt Lost : 0 Tot Pkt Lost : 0 Tot Pkt Lost : 0 Tot Pkt Lost : 0 Tot Pkt Lost : 0 Tot Pkt Lost : 0 Tot Pkt Lost : 0 Tot Pkt Lost : 0 Tot Pkt Lost : 0 Tot Pkt Lost : 0 ***@***.*** support]#c) Observe if there are any drop/errors in interface stats.
ethtool -S
| grep -i error # ifconfig <interface>@. support]# ethtool -S ens1f0 | grep -i error rx_errors: 1 tx_errors: 0 rx_length_errors: 0 rx_crc_errors: 1 port.tx_errors: 0 port.rx_crc_errors: 1 port.rx_length_errors: 0 @. support]# ifconfig ens1f0 ens1f0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500 inet 80.10.181.87 netmask 255.255.255.240 broadcast 80.10.181.95 ether 48:df:37:a5:5e:a0 txqueuelen 1000 (Ethernet) RX packets 57694105343 bytes 34604747261659 (31.4 TiB) RX errors 1 dropped 0 overruns 0 frame 1 TX packets 57678422256 bytes 34371727492276 (31.2 TiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
@.*** support]# Verification of affinity of IRQs of Ethernet interface Observe the output of command "watch -td1 'cat /proc/interrupts | grep -i '" and traffic should be increased in column associated to core to which we have set the affinity in the script.
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 CPU23 CPU24 CPU25 CPU26 CPU27 CPU28 CPU29 CPU30 CPU31126: 29645 2140537503 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge i40e-ens1f0-TxRx-0 127: 1 0 2123367377 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge i40e-ens1f0-TxRx-1 128: 29615 0 0 4103604021 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge i40e-ens1f0-TxRx-2 129: 59358 0 0 0 3021043582 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge i40e-ens1f0-TxRx-3 Share the output of all verification step from 8a to 8e for audit along with output of commands "numactl --hardware ; lscpu ; lspci ; ; cat /proc/interrupts; dmidecode ; cat /etc/redhat-release ; crontab -l from root and any other user on the same server; start script of GTPProxy and config file of GTPProxy"
***@***.*** net]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Thread(s) per core: 2 Core(s) per socket: 8 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6244 CPU @ 3.60GHz Stepping: 7 CPU MHz: 2253.735 CPU max MHz: 4400.0000 CPU min MHz: 1200.0000 BogoMIPS: 7200.00 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 25344K NUMA node0 CPU(s): 0-7,16-23 NUMA node1 CPU(s): 8-15,24-31 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 intel_ppin intel_pt ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_pkg_req pku ospke avx512_vnni spec_ctrl intel_stibp flush_l1d arch_capabilities ***@***.*** net]# ***@***.*** net]# lspci 00:00.0 Host bridge: Intel Corporation Sky Lake-E DMI3 Registers (rev 07) 00:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 07) 00:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 07) 00:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 07) 00:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 07) 00:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 07) 00:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 07) 00:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 07) 00:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 07) 00:05.0 System peripheral: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers (rev 07) 00:05.2 System peripheral: Intel Corporation Device 2025 (rev 07) 00:05.4 PIC: Intel Corporation Device 2026 (rev 07) 00:08.0 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 07) 00:08.1 Performance counters: Intel Corporation Sky Lake-E Ubox Registers (rev 07) 00:08.2 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 07) 00:11.0 Unassigned class [ff00]: Intel Corporation C620 Series Chipset Family MROM 0 (rev 09) 00:11.1 Unassigned class [ff00]: Intel Corporation C620 Series Chipset Family MROM 1 (rev 09) 00:14.0 USB controller: Intel Corporation C620 Series Chipset Family USB 3.0 xHCI Controller (rev 09) 00:14.2 Signal processing controller: Intel Corporation C620 Series Chipset Family Thermal Subsystem (rev 09) 00:16.0 Communication controller: Intel Corporation C620 Series Chipset Family MEI Controller #1 (rev 09) 00:16.4 Communication controller: Intel Corporation C620 Series Chipset Family MEI Controller #3 (rev 09) 00:1c.0 PCI bridge: Intel Corporation C620 Series Chipset Family PCI Express Root Port #1 (rev f9) 00:1c.4 PCI bridge: Intel Corporation C620 Series Chipset Family PCI Express Root Port #5 (rev f9) 00:1f.0 ISA bridge: Intel Corporation C621 Series Chipset LPC/eSPI Controller (rev 09) 00:1f.2 Memory controller: Intel Corporation C620 Series Chipset Family Power Management Controller (rev 09) 01:00.0 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support (rev 07) 01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eH3 (rev 02) 01:00.2 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging (rev 07) 01:00.4 USB controller: Hewlett-Packard Company iLO5 Virtual USB Controller 11:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port A (rev 07) 11:02.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port C (rev 07) 11:05.0 System peripheral: Intel Corporation Device 2034 (rev 07) 11:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 07) 11:05.4 PIC: Intel Corporation Device 2036 (rev 07) 11:08.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:08.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:08.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:08.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:08.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:08.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:08.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:08.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:09.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:09.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:09.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:09.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:09.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:09.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:09.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:09.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0a.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0a.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0a.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0a.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0a.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0a.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0a.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0a.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0b.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0b.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0b.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0b.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0e.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0e.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0e.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0e.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0e.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0e.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0e.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0e.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0f.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0f.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0f.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0f.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0f.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0f.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0f.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:0f.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:10.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:10.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:10.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:10.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:10.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:10.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:10.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:10.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:11.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:11.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:11.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:11.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:1d.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:1d.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:1d.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:1d.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 11:1e.0 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 07) 11:1e.1 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 07) 11:1e.2 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 07) 11:1e.3 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 07) 11:1e.4 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 07) 11:1e.5 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 07) 11:1e.6 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 07) 12:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) 12:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) 13:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01) 13:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01) 13:00.2 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01) 13:00.3 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01) 36:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port A (rev 07) 36:05.0 System peripheral: Intel Corporation Device 2034 (rev 07) 36:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 07) 36:05.4 PIC: Intel Corporation Device 2036 (rev 07) 36:08.0 System peripheral: Intel Corporation Device 2066 (rev 07) 36:09.0 System peripheral: Intel Corporation Device 2066 (rev 07) 36:0a.0 System peripheral: Intel Corporation Device 2040 (rev 07) 36:0a.1 System peripheral: Intel Corporation Device 2041 (rev 07) 36:0a.2 System peripheral: Intel Corporation Device 2042 (rev 07) 36:0a.3 System peripheral: Intel Corporation Device 2043 (rev 07) 36:0a.4 System peripheral: Intel Corporation Device 2044 (rev 07) 36:0a.5 System peripheral: Intel Corporation Device 2045 (rev 07) 36:0a.6 System peripheral: Intel Corporation Device 2046 (rev 07) 36:0a.7 System peripheral: Intel Corporation Device 2047 (rev 07) 36:0b.0 System peripheral: Intel Corporation Device 2048 (rev 07) 36:0b.1 System peripheral: Intel Corporation Device 2049 (rev 07) 36:0b.2 System peripheral: Intel Corporation Device 204a (rev 07) 36:0b.3 System peripheral: Intel Corporation Device 204b (rev 07) 36:0c.0 System peripheral: Intel Corporation Device 2040 (rev 07) 36:0c.1 System peripheral: Intel Corporation Device 2041 (rev 07) 36:0c.2 System peripheral: Intel Corporation Device 2042 (rev 07) 36:0c.3 System peripheral: Intel Corporation Device 2043 (rev 07) 36:0c.4 System peripheral: Intel Corporation Device 2044 (rev 07) 36:0c.5 System peripheral: Intel Corporation Device 2045 (rev 07) 36:0c.6 System peripheral: Intel Corporation Device 2046 (rev 07) 36:0c.7 System peripheral: Intel Corporation Device 2047 (rev 07) 36:0d.0 System peripheral: Intel Corporation Device 2048 (rev 07) 36:0d.1 System peripheral: Intel Corporation Device 2049 (rev 07) 36:0d.2 System peripheral: Intel Corporation Device 204a (rev 07) 36:0d.3 System peripheral: Intel Corporation Device 204b (rev 07) 5b:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port A (rev 07) 5b:02.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port C (rev 07) 5b:05.0 System peripheral: Intel Corporation Device 2034 (rev 07) 5b:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 07) 5b:05.4 PIC: Intel Corporation Device 2036 (rev 07) 5b:0e.0 Performance counters: Intel Corporation Device 2058 (rev 07) 5b:0e.1 System peripheral: Intel Corporation Device 2059 (rev 07) 5b:0f.0 Performance counters: Intel Corporation Device 2058 (rev 07) 5b:0f.1 System peripheral: Intel Corporation Device 2059 (rev 07) 5b:10.0 Performance counters: Intel Corporation Device 2058 (rev 07) 5b:10.1 System peripheral: Intel Corporation Device 2059 (rev 07) 5b:12.0 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 07) 5b:12.1 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 07) 5b:12.2 System peripheral: Intel Corporation Sky Lake-E M3KTI Registers (rev 07) 5b:12.4 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 07) 5b:12.5 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 07) 5b:15.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 07) 5b:15.1 Performance counters: Intel Corporation Device 2088 (rev 07) 5b:16.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 07) 5b:16.1 Performance counters: Intel Corporation Device 2088 (rev 07) 5b:16.4 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 07) 5b:16.5 Performance counters: Intel Corporation Device 2088 (rev 07) 5b:17.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 07) 5b:17.1 Performance counters: Intel Corporation Device 2088 (rev 07) 5c:00.0 Serial Attached SCSI controller: Adaptec Smart Storage PQI 12G SAS/PCIe 3 (rev 01) 5d:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) 5d:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) 5d:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) 5d:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) 80:04.0 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 07) 80:04.1 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 07) 80:04.2 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 07) 80:04.3 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 07) 80:04.4 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 07) 80:04.5 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 07) 80:04.6 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 07) 80:04.7 System peripheral: Intel Corporation Sky Lake-E CBDMA Registers (rev 07) 80:05.0 System peripheral: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers (rev 07) 80:05.2 System peripheral: Intel Corporation Device 2025 (rev 07) 80:05.4 PIC: Intel Corporation Device 2026 (rev 07) 80:08.0 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 07) 80:08.1 Performance counters: Intel Corporation Sky Lake-E Ubox Registers (rev 07) 80:08.2 System peripheral: Intel Corporation Sky Lake-E Ubox Registers (rev 07) 85:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port A (rev 07) 85:05.0 System peripheral: Intel Corporation Device 2034 (rev 07) 85:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 07) 85:05.4 PIC: Intel Corporation Device 2036 (rev 07) 85:08.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:08.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:08.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:08.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:08.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:08.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:08.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:08.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:09.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:09.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:09.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:09.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:09.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:09.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:09.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:09.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0a.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0a.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0a.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0a.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0a.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0a.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0a.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0a.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0b.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0b.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0b.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0b.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0e.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0e.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0e.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0e.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0e.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0e.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0e.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0e.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0f.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0f.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0f.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0f.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0f.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0f.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0f.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:0f.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:10.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:10.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:10.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:10.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:10.4 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:10.5 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:10.6 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:10.7 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:11.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:11.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:11.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:11.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:1d.0 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:1d.1 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:1d.2 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:1d.3 System peripheral: Intel Corporation Sky Lake-E CHA Registers (rev 07) 85:1e.0 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 07) 85:1e.1 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 07) 85:1e.2 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 07) 85:1e.3 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 07) 85:1e.4 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 07) 85:1e.5 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 07) 85:1e.6 System peripheral: Intel Corporation Sky Lake-E PCU Registers (rev 07) 86:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) 86:00.1 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) ae:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port A (rev 07) ae:05.0 System peripheral: Intel Corporation Device 2034 (rev 07) ae:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 07) ae:05.4 PIC: Intel Corporation Device 2036 (rev 07) ae:08.0 System peripheral: Intel Corporation Device 2066 (rev 07) ae:09.0 System peripheral: Intel Corporation Device 2066 (rev 07) ae:0a.0 System peripheral: Intel Corporation Device 2040 (rev 07) ae:0a.1 System peripheral: Intel Corporation Device 2041 (rev 07) ae:0a.2 System peripheral: Intel Corporation Device 2042 (rev 07) ae:0a.3 System peripheral: Intel Corporation Device 2043 (rev 07) ae:0a.4 System peripheral: Intel Corporation Device 2044 (rev 07) ae:0a.5 System peripheral: Intel Corporation Device 2045 (rev 07) ae:0a.6 System peripheral: Intel Corporation Device 2046 (rev 07) ae:0a.7 System peripheral: Intel Corporation Device 2047 (rev 07) ae:0b.0 System peripheral: Intel Corporation Device 2048 (rev 07) ae:0b.1 System peripheral: Intel Corporation Device 2049 (rev 07) ae:0b.2 System peripheral: Intel Corporation Device 204a (rev 07) ae:0b.3 System peripheral: Intel Corporation Device 204b (rev 07) ae:0c.0 System peripheral: Intel Corporation Device 2040 (rev 07) ae:0c.1 System peripheral: Intel Corporation Device 2041 (rev 07) ae:0c.2 System peripheral: Intel Corporation Device 2042 (rev 07) ae:0c.3 System peripheral: Intel Corporation Device 2043 (rev 07) ae:0c.4 System peripheral: Intel Corporation Device 2044 (rev 07) ae:0c.5 System peripheral: Intel Corporation Device 2045 (rev 07) ae:0c.6 System peripheral: Intel Corporation Device 2046 (rev 07) ae:0c.7 System peripheral: Intel Corporation Device 2047 (rev 07) ae:0d.0 System peripheral: Intel Corporation Device 2048 (rev 07) ae:0d.1 System peripheral: Intel Corporation Device 2049 (rev 07) ae:0d.2 System peripheral: Intel Corporation Device 204a (rev 07) ae:0d.3 System peripheral: Intel Corporation Device 204b (rev 07) af:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01) af:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01) af:00.2 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01) af:00.3 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5719 Gigabit Ethernet PCIe (rev 01) d7:00.0 PCI bridge: Intel Corporation Sky Lake-E PCI Express Root Port A (rev 07) d7:05.0 System peripheral: Intel Corporation Device 2034 (rev 07) d7:05.2 System peripheral: Intel Corporation Sky Lake-E RAS Configuration Registers (rev 07) d7:05.4 PIC: Intel Corporation Device 2036 (rev 07) d7:0e.0 Performance counters: Intel Corporation Device 2058 (rev 07) d7:0e.1 System peripheral: Intel Corporation Device 2059 (rev 07) d7:0f.0 Performance counters: Intel Corporation Device 2058 (rev 07) d7:0f.1 System peripheral: Intel Corporation Device 2059 (rev 07) d7:10.0 Performance counters: Intel Corporation Device 2058 (rev 07) d7:10.1 System peripheral: Intel Corporation Device 2059 (rev 07) d7:12.0 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 07) d7:12.1 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 07) d7:12.2 System peripheral: Intel Corporation Sky Lake-E M3KTI Registers (rev 07) d7:12.4 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 07) d7:12.5 Performance counters: Intel Corporation Sky Lake-E M3KTI Registers (rev 07) d7:15.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 07) d7:15.1 Performance counters: Intel Corporation Device 2088 (rev 07) d7:16.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 07) d7:16.1 Performance counters: Intel Corporation Device 2088 (rev 07) d7:16.4 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 07) d7:16.5 Performance counters: Intel Corporation Device 2088 (rev 07) d7:17.0 System peripheral: Intel Corporation Sky Lake-E M2PCI Registers (rev 07) d7:17.1 Performance counters: Intel Corporation Device 2088 (rev 07) ***@***.*** net]# cat /proc/interrupts; dmidecode ; cat /etc/redhat-release ; crontab -l CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 CPU23 CPU24 CPU25 CPU26 CPU27 CPU28 CPU29 CPU30 CPU31 0: 74 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-edge timer 1: 5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-edge i8042 4: 1241 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-edge serial 8: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-edge rtc0 9: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi acpi 12: 6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-edge i8042 17: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-IO-APIC-fasteoi ehci_hcd:usb1, hpilo 24: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 25: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 27: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 29: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 31: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 33: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 35: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 37: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 39: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 41: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge PCIe PME 42: 24 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge xhci_hcd 43: 138231031 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 44: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 45: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 46: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 47: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 48: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 49: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 50: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 51: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 52: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 53: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 54: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 55: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 56: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 57: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 58: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 59: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 60: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 61: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 62: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 63: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 64: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 65: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 66: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 67: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 68: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 69: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 70: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 71: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 72: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 73: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 74: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge smartpqi 75: 22 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge mei_me 77: 13 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eno5 78: 46750907 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eno5-TxRx-0 79: 45041743 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eno5-TxRx-1 80: 56291180 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eno5-TxRx-2 81: 46784740 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eno5-TxRx-3 82: 73099357 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eno5-TxRx-4 83: 47311582 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eno5-TxRx-5 84: 52994115 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eno5-TxRx-6 85: 44672904 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eno5-TxRx-7 88: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar6 89: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar5 90: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar4 91: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar3 92: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar2 93: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar1 94: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar0 95: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 DMAR_MSI-edge dmar7 96: 13 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI-edge eno6 97: 154418022 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
MobileumTech
Hi Team,
Hope you are doing good.
We facing issue while using pf-ring zc for data packet processing.
PF-Ring ZC is adding latency while processing GTP GPD-U packets processing from network. We are observing the latency at pf-ring level when it handle the packets at lower throughput.
It looks like PF-Ring is doing some internal buffering before giving the packets to application for processing and as per our observation is only when we have lower throughput.
Please help us to investigate the issue further.
Please find the system level output and stats from server where we are observing the latency problem.
++++++++++++++++ ens1f0
numactl --hardware
available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23 node 0 size: 65211 MB node 0 free: 16149 MB node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31 node 1 size: 65535 MB node 1 free: 8944 MB node distances: node 0 1 0: 10 21 1: 21 10 [root@oprrsr3m2msprs07 ~]#
sysctl -a | grep huge
[root@oprrsr3m2msprs07 net]# sysctl -a | grep huge vm.hugepages_treat_as_movable = 0 vm.hugetlb_shm_group = 0 vm.nr_hugepages = 0 vm.nr_hugepages_mempolicy = 0 vm.nr_overcommit_hugepages = 0 [root@oprrsr3m2msprs07 net]#
cat /proc/meminfo | grep -i huge
[root@oprrsr3m2msprs07 net]# cat /proc/meminfo | grep -i huge AnonHugePages: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB [root@oprrsr3m2msprs07 net]#
cat /proc/net/pf_ring/info
[root@oprrsr3m2msprs07 net]# cat /proc/net/pf_ring/info PF_RING Version : 7.5.0 (unknown) Total rings : 17
Standard (non ZC) Options Ring slots : 409600 Slot version : 17 Capture TX : Yes [RX+TX] IP Defragment : Yes Socket Mode : Standard [root@oprrsr3m2msprs07 net]#
cat /proc/net/pf_ring/dev//info
[root@oprrsr3m2msprs07 net]# cat /proc/net/pf_ring/dev/ens1f0/info Name: ens1f0 Index: 14 Address: 48:DF:37:A5:5E:A0 Polling Mode: NAPI/ZC Type: Ethernet Family: Intel i40e TX Queues: 4 RX Queues: 4 Num RX Slots: 4096 Num TX Slots: 512 [root@oprrsr3m2msprs07 net]#
ethtool -l
[root@oprrsr3m2msprs07 net]# ethtool -l ens1f0 Channel parameters for ens1f0: Pre-set maximums: RX: 0 TX: 0 Other: 1 Combined: 4 Current hardware settings: RX: 0 TX: 0 Other: 1 Combined: 4
ethtool -g
[root@oprrsr3m2msprs07 net]# ethtool -g ens1f0 Ring parameters for ens1f0: Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096 Current hardware settings: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 512
ethtool -a
[root@oprrsr3m2msprs07 net]# ethtool -a ens1f0 Pause parameters for ens1f0: Autonegotiate: off RX: off TX: off
9 ethtool -k
ethtool -c
[root@oprrsr3m2msprs07 net]# ethtool -c ens1f0 Coalesce parameters for ens1f0: Adaptive RX: on TX: on stats-block-usecs: 0 sample-interval: 0 pkt-rate-low: 0 pkt-rate-high: 0
rx-usecs: 50 rx-frames: 0 rx-usecs-irq: 0 rx-frames-irq: 256
tx-usecs: 50 tx-frames: 0 tx-usecs-irq: 0 tx-frames-irq: 256
rx-usecs-low: 0 rx-frame-low: 0 tx-usecs-low: 0 tx-frame-low: 0
rx-usecs-high: 0 rx-frame-high: 0 tx-usecs-high: 0 tx-frame-high: 0
11 a) Run a script with the below command to store the output of top command in a file. For ex - create a script with the name top.sh having below contents
cat top.sh
12) Verification of affinity of IRQs of Ethernet interface Observe the output of command "watch -td1 'cat /proc/interrupts | grep -i'" and traffic should be increased
in column associated to core to which we have set the affinity in the script.
Share the output of all verification step from 8a to 8e for audit along with output of commands "numactl --hardware ; lscpu ; lspci ; ; cat /proc/interrupts; dmidecode ; cat /etc/redhat-release ; crontab -l from root and any other user on the same server; start script of GTPProxy and config file of GTPProxy"
++++++++++++++++++++++++
PFRing_troubleshooting_20210625.txt