andre-richter / pcie-lat

Generic x86_64 PCIe latency measurement module for the Linux kernel
GNU General Public License v2.0
56 stars 15 forks source link

network card latency overly optimistic? #9

Open vossm opened 1 week ago

vossm commented 1 week ago

Hello the latency observed on a RHEL9 host to a low latency network card appeared to more likely be the one-way latency versus the roundtrip latency. Do you have any idea what could cause this? Perhaps an incorrect TSC or some other setting? Processor is a Intel(R) Xeon(R) Gold 6250 CPU with 8 cores (no hyperthreading) at 4.5ghz across all cores per i7z (no power savings/etc.) on a generic whitebox Supermicro X11SPG-TF motherboard.

ruby measure.rb -p b3:00.0 -l 1000000 -b 0 -o 0x0

TSC freq: 3899999000.0 Hz TSC overhead: 27 cycles Device: b3:00.0 BAR: 0 Offset: 0x0 Loops: 1000000

   | Results (1000000 samples)

Mean | 1652.43 cycles | 423.70 ns Stdd | 15.16 cycles | 3.89 ns

   | 3σ Results (994957 samples, 0.005% discarded)

Mean | 1652.01 cycles | 423.59 ns Stdd | 13.87 cycles | 3.56 ns

ruby measure.rb -p b3:00.0 -l 1000000 -b 2 -o 0x0

TSC freq: 3899999000.0 Hz TSC overhead: 27 cycles Device: b3:00.0 BAR: 2 Offset: 0x0 Loops: 1000000

   | Results (1000000 samples)

Mean | 1586.32 cycles | 406.75 ns Stdd | 12.57 cycles | 3.22 ns

   | 3σ Results (985469 samples, 0.015% discarded)

Mean | 1585.41 cycles | 406.51 ns Stdd | 9.58 cycles | 2.46 ns