mOS with mlx5

[tbarbette]

Hi,

Is it possible to configure mOS to use Mellanox NICs (using the mlx5 driver)? --run-dpdk complains igb_uio is not loaded (because we don't need it with MLX).

Thanks, Tom

[tbarbette]

Oh actually I see things changed since the last time I built mOS ;) I see dpdk-iface is failing to build, I guess that may be part of the problem : "CHRDEV "dpdk-iface" major requested (1110) is greater than the maximum (512)". Is that a known problem? EDIT: I solved that temporarily by setting the major number to 500. I have a very "usual" Ubuntu server, maybe you should change the default? Nevertheless I still can't make it work (I could like one year ago with i40e though).

[tbarbette]

Ok, I could make it work for 1 core, because mlx5 does not support 52 bytes RSS key (max is 40 bytes), so I disabled the configuration of the RSS key. It will break multi-core for sure... But it works with one.

[tbarbette]

Well, simply resizing the hash key to 40 bytes worked. Also, I had the same problem of endianess for the key, so I tried to change this in config.c :

• if (!strcmp(*argp, "net_i40e"))
• if (!strcmp(argp, "net_i40e") || !strcmp(argp, "net_mlx5") ) But the performance with 3 cores is still somehow heratic. Exactly as with the problem this solved for net_i40e.

I'll leave you close the post. If you want to support mlx5 you may just do that, and latest ubuntu by changing the major number.

[ygmoon]

Hi Tom,

First of all, thanks for the detailed reports and comments.

Regarding the RSS issue, I suspect that this is closely related to #17 . mOS precalculates the expected Rx queue number for receiving the returning packets for the given 4-tuple (GetRSSCPUCore() in core/src/util.c).

• As you might notice in https://github.com/mos-stack/mOS-networking-stack/commit/d88f3b1df1fbe5f88a4201c9af0205ca2006fd30, our current code only cares ixgbe and i40e.
• Although i40e NICs and mlx5 NICs use the same RSS key (and the same endianess), the way to translate the hash value to the Rx queue number can be different.
• However, as opposed to Intel NICs, the datasheet of Mellanox NICs does not explain those details. I will take a look on this issue, but it might take some time to resolve.

If you have any chance, can you please check whether mOS works fine with mlx5 for 2 and/or 4 cores?

Thanks, YoungGyoun

[ajamshed]

@tbarbette,

Apologies for the delayed response. The dpdk-iface module is reporting a run-time issue for newer kernels as they are unable to assign a major device number (above a certain threshold, 511) to newly registered character devices. I patched the code for mTCP a few weeks ago (https://github.com/mtcp-stack/mtcp/commit/2d308b588675065497a95b558cbac02ad7e3f8ff) but forgot to apply it to mOS. Please give me a a few days to apply similar changes to mOS core stack.

Also, you are right. mlx4/mlx5 devices don't really need neither igb_uio.ko nor dpdk_iface.ko drivers. They are only used for Intel-based Ethernet NICs. The setup.sh script is written for Intel-specific setups which seems to be one of the few problems that you have faced in your setup. We need a revision for setup.sh as well. We plan to update it in the coming days.

[tbarbette]

@ygmoon Yes, as a researcher, Mellanox NICs are a pain when it comes to documentation It works great for 1, 2 and 4 cores. So yes, it looks like the old problem...

@ajamshed even a better fix!

Thanks, both of you for the help. I can live with 1/2/4 for now, no rush ;)

[ajamshed]

Oh actually I see things changed since the last time I built mOS ;) I see dpdk-iface is failing to build, I guess that may be part of the problem : "CHRDEV "dpdk-iface" major requested (1110) is greater than the maximum (512)". Is that a known problem? EDIT: I solved that temporarily by setting the major number to 500. I have a very "usual" Ubuntu server, maybe you should change the default? Nevertheless I still can't make it work (I could like one year ago with i40e though).

This part is fixed. Please see the devel branch.