search

emmericp / MoonGen

MoonGen is a fully scriptable high-speed packet generator built on DPDK and LuaJIT. It can saturate a 10 Gbit/s connection with 64 byte packets on a single CPU core while executing user-provided Lua scripts for each packet. Multi-core support allows for even higher rates. It also features precise and accurate timestamping and rate control.
MIT License
1.03k stars 234 forks source link

Ubuntu and Mellanox OFED version that works with Moongen? #301

Open Assar-Westman opened 3 years ago

Assar-Westman commented 3 years ago

Thanks for a really nice high speed packet generator. I used the Moongen packet generator a lot a year ago on Mellanox mlx5 100 Gbe cards. Now I needed to do some more benchmarks using Moongen using these 100 Gbe cards. During last year I updated the computer to latest version of ubuntu and also the Mellanox OFED was updated to latest version. Now when I try to use Moongen I get this error

PMD: net_mlx5: ibv_exp_query_device() failed

So I wonder what is the latest version of Ubuntu and Mellanox OFED that are supported by Moongen?

Best regards Assar W.

emmericp commented 3 years ago

sorry, our support for mellanox is somwhat bad due to their terrible driver model. Can you try to use the branch dpdk-19.05?

Assar-Westman commented 3 years ago

Thanks for the suggestion. When I using the 19.05 branch I get a lot of compile error like

libmoon/deps/dpdk/drivers/net/mlx5/mlx5_glue.c:754: undefined reference to `mlx5dv_open_device' I have added the preferred deb packages, except libmlx5[...].deb libmlx5-dev[...].deb which are missing in the used Mellanox OFED 5.2 distribution It feels like some Mellanox library is missing maybe it is supplied with libmlx5...

Best regards Assar W.

Assar-Westman commented 3 years ago

Now things looks much better. I have been able to compile MoonGen on my box using the DPDK 19.05 branch. What It did was to edit libmoon/CMakeLists.txt and add to add two lines after

Add Mellanox libraries if options are specified

if(USE_MLX5 OR USE_MLX4) SET(DPDK_LIBS ${DPDK_LIBS} ibverbs)

so it looks like this

Add Mellanox libraries if options are specified

if(USE_MLX5 OR USE_MLX4) SET(DPDK_LIBS ${DPDK_LIBS} ibverbs) SET(DPDK_LIBS ${DPDK_LIBS} mnl) SET(DPDK_LIBS ${DPDK_LIBS} mlx5)

I really needed the LUA jit stuff as I have to attach current time to to the crafted data part of the UDP package.

Again thanks for a really nice software package. Assar W.

michael-kaplan commented 3 years ago

Did you succeed? I'm curious if you got DPDK to include the Mellanox drivers, --mlx5 flag to build.sh not seem to be working for me. have very similar setup.

Assar-Westman commented 3 years ago

My moongen setup now works with Mellanox OFED driver version 5.2-2.2.0.0, Ubuntu version 20.04.2 LTS and kernel 5.8.0-44-generic. For me the crucial thing was to use branch 19.05, install the whole Mellanox OFED driver package and also to install libmnl-dev package, and of course edit libmoon/CMakeLists.txt as seen above AW

michael-kaplan commented 3 years ago

Fascinating. I am using Mellanox OFED driver 5.3-OFED.5.3.1.0.0.1, Ubuntu 20.04.2 LTS, and kernel 5.4.0-26-generic. I changed the MoonGen branch to 19.05 and built with ./build.sh --mlx5 (after updating CMAKELists.txt, built successfully).

Yet for whatever reason I continue to get a 0 device error. Arg! I have a MT28880, do you know which you have? I also find some of the complaints concerning, but do not know what to make of them. I suppose there's also updating the Ubuntu kernel.

[INFO]  Initializing DPDK. This will take a few seconds...
EAL: Detected 64 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /run/user/10002/dpdk/rte/mp_socket
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: Cannot obtain physical addresses: Success. Only vfio will function.
EAL: Couldn't get fd on hugepage file
EAL: Couldn't get fd on hugepage file
error allocating rte services array
EAL: FATAL: rte_service_init() failed
EAL: rte_service_init() failed
[INFO]  Found 0 usable devices:
[FATAL] Lua error in task master
...me/mkaplan/prod3/MoonGen/build/../libmoon/lua/device.lua:100: there are only 0 ports, tried to configure port id 0
Assar-Westman commented 3 years ago

I have a MT28800 Family ConnectX-5 PCie 4.0 I see in my notes that I only compile Moongen with GCC version 7.5 Of course you have run both bind-interfaces.sh and setup-hugetlbfs.sh AW

liqia commented 2 years ago

Now things looks much better. I have been able to compile MoonGen on my box using the DPDK 19.05 branch. What It did was to edit libmoon/CMakeLists.txt and add to add two lines after

Add Mellanox libraries if options are specified if(USE_MLX5 OR USE_MLX4) SET(DPDK_LIBS ${DPDK_LIBS} ibverbs)

so it looks like this

Add Mellanox libraries if options are specified if(USE_MLX5 OR USE_MLX4) SET(DPDK_LIBS ${DPDK_LIBS} ibverbs) SET(DPDK_LIBS ${DPDK_LIBS} mnl) SET(DPDK_LIBS ${DPDK_LIBS} mlx5)

I really needed the LUA jit stuff as I have to attach current time to to the crafted data part of the UDP package.

Again thanks for a really nice software package. Assar W.

thanks！！！！

Ramla-I commented 2 years ago

Hi, have you found a solution for this? I'm also able to to build MoonGen with the mlx5 flag, but when I try to run it I get the 0 usable devices error.

tom-wegener commented 2 years ago

I hope Ramla-I solved his problem but for people looking into this later: because of the special driver model of the mlx-devices the devices do not have to bind by the script and the script reports 0 devices anyway. I could just my script after following the steps from Assar-Westman .