Closed doraeric closed 6 months ago
NicNull
commented
1 year ago I've seen the same, to me it looks like the Tx rate is not being fulfilled. I'm running 200Gbps load tests. Try the "dbg tx_rate all" command to see the target rate for Tx and compare it to the actual rate. The latest 23.10.0 is a little bit better with some changes in the Tx timing and better precision in the tx_cycles calculation. You'll need the latest dpdk 23.11-rc1 for that.
NicNull
commented
1 year ago If you are using ranges for the test this issue was also affecting the Tx rate produced. Substandard 'range' performance #213
NicNull
commented
1 year ago I also found this commit but haven't looked at it yet with the latest version of PktGen if it affects my Tx rate profile. https://github.com/pktgen/Pktgen-DPDK/commit/bd3e615c98aaef2aac6c75a31371c9b5fe72ee67
edit I tested reverting this but it looks worse. If the commit is aimed to better take spent cycles into account it doesn't find the actual target or the precision is off somehow.
KeithWiles
commented
12 months ago I am in the process of figuring out the performance drop so I created a simpler application in the txpkts branch under pktgen/examples/pktperf directory. If you do a make rebuild it should build this application and if you have the time to give it a try it would be great. The README.md file is not great, but I hope it helps you get it running.
I spent some time on this application to make sure I could get the best Tx and Rx performance. The application is a WiP.
doraeric
commented
12 months ago I have checkouted to the branch and tested pktperf. pktperf seems working better than pktgen. I tried different combinations for pktgen. Here's the result:
| tx | rx | tx (pps) | rx (pps) | tx (%) | rx (%) |
|---|---|---|---|---|---|
| testpmd | pktgen | 14,880,954 | 14,853,063 | 100.00% | 99.81% |
| pktgen | testpmd | 14,880,952 | 14,296,037 | 100.00% | 96.07% |
| pktgen (pg0) | pktgen (pg1) | 14,880,947 | 14,858,916 | 100.00% | 99.85% |
| pktgen (pg) | pktgen (pg) | 13,908,176 | 13,876,717 | 93.46% | 93.25% |
I tried to run pktgen several times, and found something special. If I start 2 pktgen processes and assign different cores and prefixes, they can reach the maximum speed. If I start 1 pktgen process and let it control 2 ports, there is an upper limit per run. Sometimes it can reach 14,87x,xxx pps. Sometimes the upper bound is much lower, and it will never exceed the limit for that round. It looks like some kind of bug?
KeithWiles
commented
12 months ago Thank you for testing pktperf.
doraeric
commented
11 months ago I tried some different combination for pktperf recently with version 5d32180. Here's the result.
| tx | rx | tx (pps) | rx (pps) | tx (%) | rx (%) | Note |
|---|---|---|---|---|---|---|
| pktperf (same proc) | pktperf (same proc) | 14,926,224 | 14,926,235 | 100.30% | 100.30% | average on 2 port |
| pktperf (pp0) | pktperf (pp1) | 14,931,707 | 14,927,999 | 100.34% | 100.32% | average on 2 port |
| pktperf | pktgen | 15,053,387 | 14,880,961 | 101.16% | 100.00% | |
| pktgen | pktperf | 14,880,950 | 15,128,012 | 100.00% | 101.66% | pktperf running |
Although the number from pktperf seems a bit higher as rx (pktperf) shouldn't be greater than tx (pktgen), the performance is much better than running one pktgen process with 2 ports.
KeithWiles
commented
6 months ago Some performance changes have been done in the latest release 24.05.2
Hello, I am doing some benchmarks with pktgen. I notice there is a performance issue after updating pktgen, so I am here for help. I run pktgen multiple times on the same hardware, same operating system (ubuntu 22.04), the only difference is the version of pktgen and dpdk. I want to saturate a 10 GbE link. For pktgen <= 22.07.1, there is no problem, it can reach 10 gbps. For pktgen >= 23.03.0, it only reaches about 8,600 mbps. Here is the experiment data.
The speed is just a rough number, but there is a big difference between the specific version.
I also tried to write a small dpdk app myself to forward packets with new dpdk version. The result is that my app can saturate the link, so I think there is no problem with dpdk.
Any ideas or help would be appreciated.