is it possible to use PF's RDMA device once SFs are enabled for it?

[htariq9436]

Hi,

I am working with ConnectX-6 card and Linux kernel v5.13.0. As it has 2 PFs, output of _ibvdevices lists two RDMA devices, namely _mlx50 & _mlx51. Once the SFs are enabled in firmware and the eswitch mode is changed to switchdev for PF1, _mlx51 loses its GIDs and hence can not be used for communication. But the following snippet from the Wiki suggests otherwise:

When scalable function is RDMA capable, it has its own QP1, GID table and rdma resources neither shared nor stolen from the parent PCI function.

Am I misinterpreting the above statement and seeing the expected behavior? If not do you have any idea as to why I am seeing this?

Following are the commands run before losing GIDs for _mlx51:

• mlxconfig -d 0000:04:00.1 s PF_BAR2_ENABLE=0 PER_PF_NUM_SF=1 PF_TOTAL_SF=236 PF_SF_BAR_SIZE=10
• sudo devlink dev eswitch set pci/0000:04:00.1 mode switchdev

Thanks

[paravmellanox]

@htariq9436 ,

Even before creating the SFs, when you move the device to switchdev mode, mlx5_0 and mlx5_1 both will lose the GID table. This is because PF is a switching device once you move to the switchdev mode. In this mode, RDMA (RoCE) cannot operate. Hence there is no GID table.

you should use one of the SF or VF as communication instead of switchdev PF.

[htariq9436]

@paravmellanox,

Makes sense. Thanks for the quick response.

[htariq9436]

@paravmellanox

Just had a little follow up to the above query. As you mentioned above, once PF is in switchdev mode it cannot be used for communication as an RDMA device. Does this hold true for the PF's net device?

Thanks

[paravmellanox]

@htariq9436 I am not lot familiar with the setups where someone assigns IP to the PF netdevice when using as the switchdev mode. but I think it should be possible somehow, I do not know the trick for such setup.

[htariq9436]

@paravmellanox Once again thanks for the information.