search

emmericp / MoonGen

MoonGen is a fully scriptable high-speed packet generator built on DPDK and LuaJIT. It can saturate a 10 Gbit/s connection with 64 byte packets on a single CPU core while executing user-provided Lua scripts for each packet. Multi-core support allows for even higher rates. It also features precise and accurate timestamping and rate control.
MIT License
1.04k stars 235 forks source link

Rate limiter above 30 Gbit/s #258

Closed Assar-Westman closed 4 years ago

Assar-Westman commented 4 years ago

Hi Have been now using MoonGen to generate packets initially with 25 Gbit/s interface cards. Now I have upgraded to a Mellanox ConnectX 5 card with dual 100 Gbit/s ports. This is a PCIe3 card so of course I will not get a total data rate of 200 Gbit/s.

I have now been doing some test and using the software limiter (limiter:new(txDevs … ) and have achieved a data rate of almost 34 Gbit/s for each of the ports so a total of 68 Gbit/s. I can also set the data rate to what ever speed I want below 34 Gbit/s. This is almost half of the PCie3 (16 lanes with 8 GT/s). Then I tried to send the data without any limiter (queu:send(bufs)) and got almost 57 Gbits/s on each of the port (a total of 114 Gbit/s).

I would really liked to be able to do some rate limiting even above 34 Gbit/s per port but not succeeded. I have tried queue:setRate(speed in Mbit/s) but nothing changes, the port speed is still 57 Gbit/s. Then I tried the queue:sendWithDelay( than I get the error "Driver does not support disabling the CRC flag. This feature requires a patched driver.”

Does this means that a patched driver for the Mellanox cards is needed and if so does this patch exist?

Best regards Assar Westman

Assar-Westman commented 4 years ago

Hi It seems like the problem for me was that I used the poisson rate limiter. I changed it to cbr instead and now I can set the data rate all the way up to a maximum data rate of around 55 Gbits/s for each port.

Thanks to all that have been involved in the MoonGen project and especially Paul Emmerich for a very useful piece of software.

Assar Westman

emmericp commented 4 years ago

Hmm. The poisson rate limiter should scale better than the CBR limiter. The idea is that the sum of several poisson processes is a poisson process, and you can therefore start as many poisson rate limiters as you want without affecting the distribution.