Observe different latency results at various Tx rates

[Aniurm]

What's the problem

I have observed varying latency results at different transmission (Tx) rates. I ran pktgen for sufficient time to generate a large number of packets, aiming to minimize the chance of error.

set 0 rate 0.01

Latency:            :
  Rate (us)         :                10,000
  Entropy           :                     0
  Total Pkts        :                27,830
  Skipped Pkts      :                     0
  Cycles/Minimum(us):       21,468/    9.33
  Cycles/Average(us):       27,236/   11.84
  Cycles/Maximum(us):       64,300/   27.96
Jitter:             :
  Threshold (us)    :                    50
  Count/Percent     :              0/  0.00
                    :
Pattern Type        :               abcd...
Tx Count/% Rate     :        Forever /0.01%
PktSize/Rx:Tx Burst :           64 /  1:  1

set 0 rate 0.001

Latency:            :
  Rate (us)         :                10,000
  Entropy           :                     0
  Total Pkts        :                30,301
  Skipped Pkts      :                     1
  Cycles/Minimum(us):       18,104/    7.87
  Cycles/Average(us):       21,538/    9.36
  Cycles/Maximum(us):      917,084/  398.73
Jitter:             :
  Threshold (us)    :                    50
  Count/Percent     :              4/  0.01
                    :
Pattern Type        :               abcd...
Tx Count/% Rate     :       Forever /0.001%
PktSize/Rx:Tx Burst :           64 /  1:  1

set 0 rate 0.0001

Latency:            :
  Rate (us)         :                10,000
  Entropy           :                     0
  Total Pkts        :                32,863
  Skipped Pkts      :                     0
  Cycles/Minimum(us):       12,864/    5.59
  Cycles/Average(us):       15,866/    6.90
  Cycles/Maximum(us):       44,268/   19.25
Jitter:             :
  Threshold (us)    :                    50
  Count/Percent     :              0/  0.00
                    :
Pattern Type        :               abcd...
Tx Count/% Rate     :      Forever /0.0001%
PktSize/Rx:Tx Burst :           64 /  1:  1

set 0 rate 0.00001

Latency:            :
  Rate (us)         :                10,000
  Entropy           :                     0
  Total Pkts        :                43,280
  Skipped Pkts      :                     0
  Cycles/Minimum(us):       12,584/    5.47
  Cycles/Average(us):       17,755/    7.72
  Cycles/Maximum(us):    1,457,560/  633.72
Jitter:             :
  Threshold (us)    :                    50
  Count/Percent     :              2/  0.00
                    :
Pattern Type        :               abcd...
Tx Count/% Rate     :       Forever /1e-05%
PktSize/Rx:Tx Burst :           64 /  1:  1 

• Rate 0.01 - Average 11.84 us
• Rate 0.001 - Average 9.36 us
• Rate 0.0001. - Average 6.90 us
• Rate 0.00001 - Average 7.72 us

I am unsure why latency varies with the Tx rate. I would expect latency to remain consistent regardless of the Tx rate.

By the way, what does Tx rate mean?

I've read the source code:

static void
calculate_rate(port_info_t *info)
{
    rate_info_t *rate = &info->rate;

    rate->bytes_per_vframe = (rate->vlines * rate->pixels * rate->color_bits) / 8;
    rate->bits_per_sec     = (rate->bytes_per_vframe * rate->fps) * 8;

    rate->pkts_per_vframe    = rate->bytes_per_vframe / rate->payload;
    rate->total_pkts_per_sec = rate->pkts_per_vframe * rate->fps;
    rate->pps_rate           = 1.0 / (double)rate->total_pkts_per_sec;

    rate->cycles_per_pkt = (pktgen.hz / rate->total_pkts_per_sec);

    info->tx_cycles = info->rate.cycles_per_pkt;
}

From this code, info->tx_cycles controls the frequency of Tx bursts in pktgen.c. However, how rate affects tx_cycles is somewhat confusing.

My current understanding: Pktgen knows the maximum speed of my NIC (but how?), and we can use the rate to control the actual transmission speed (actual speed = max speed * rate). Is this correct?

My configuration

Testpmd

sudo ./dpdk-testpmd -l 1,3,5 -n 4 --file-prefix pg-receive --proc-type auto -a 82:00.0,class=eth,rx_vec_en=0 -- -i --portlist=0 --nb-cores=2 --port-topology=loop

set fwd macswap
set burst 1
start

Pktgen

 sudo ./pktgen -l 0,2 -n 4 --proc-type auto -a 03:00.0,class=eth,rx_vec_en=0 
 -- -P -m "[2].0" -T -j

set 0 rxburst 1
set 0 txburst 1
set 0 size 64
enable 0 latency
start 0

[KeithWiles]

I see the problem, the calculate_rate() routine in app/pktgen-rate.c is not used for latency testing. This routine is used for packet-pacing which is different from rate. In app/pktgen.c look at the pktgen_packet_rate() routine for the calculation of rate used for the other modes.

The only reason I can see the latency to change is the number of packets being handled by testpmd and testpmd could be doing something like batching packets or using rte_eth_tx_buffer_xxxx routines to maintainer better performance. The code in pktgen sends a stream of packets at a given rate then is injects latency packet at 10,000us intervals or whatever the inject rate is set to.

Latency is tricky when trying to use software to measure the values.

[KeithWiles]

I have a performance-update branch, which removes the pktgen-rate.c code and attempts to give better performance along with handling injecting packets into a stream. Have a look at it if you it is still very much a Work-In-Progress.

[Aniurm]

Thank you for your help, now I can understand the code, I'm very grateful to you.

Since rate is an irrelevant variable in my experiment, I'll just use a fixed rate.