Mellanox / scalablefunctions

All about Scalable functions
39 stars 2 forks source link

devlink device naming for more than 100 SFs #1

Closed bessa closed 3 years ago

bessa commented 3 years ago

Hi, Is it possible to add a rename option upon device creation, as the default behaviour the devlink with add suffix 1-99 to PF, and then auto generate ethxxx randomly, and it would hard to track, configure, delete after.

tested on Connect-x6 FW: 20.31.0356

 grep -i pretty /etc/os-release
PRETTY_NAME="Ubuntu 20.10"

 uname -r
5.12.0-051200-generic

 devlink -V
devlink utility, iproute2-5.12.
paravmellanox commented 3 years ago

@bessa ,

sorry for the late response. What is the netdevice name of the PF device? Can you please share the output of below command before creating the SF port?

$ devlink port show

yes, I will share the little different udev naming script, that overcomes the limitation of 15 characters limit by renaming them differently. Alteratively, you can also use $ devlink port show -jp to get the JSON output format that contains the netdevice name of the port to reuse later.

Every created port has unique port index, so you can use that as well to find out its representor netdevice name.

bessa commented 3 years ago

Hi @paravmellanox , Sorry for the delay, I have set uncorrectly PF_SF_BAR_SIZE, which made the cards undetectable, took me sometime to restore them back.

Here you go:

# devlink port show
pci/0000:85:00.0/1: type eth netdev enp133s0 flavour physical port 0
pci/0000:41:00.0/1: type eth netdev enp65s0 flavour physical port 0

example, creating 200 SFs on each, not yet attaching them, which will create other interfaces.

Attached is the list.

Many thanks SF-list.txt

paravmellanox commented 3 years ago

hi @bessa

Kernel is limited to only 15 characters for the eth netdevice name. So udev/systemd is unable to rename them beynd 15 characters. To overcome this limitation, alternative name support was added. You might have to update your kernel and user space systemd to make use of alternative naming scheme. I do not have list of those patches, but you can possibly find them on mailing list or use most upto date kernel and simplify.

Alternatively ( I do not recommend) but kernel and user space upgrade is hard, you can setup your on udev rule by adding /etc/udev/rules.d/84.rules file for shorter names.

Please find the attach files for udev naming.

  1. /etc/udev/rules.d/83-net-setup-link.rules

  2. /etc/infiniband/sf-rep-netdev-rename.sh 83-net-setup-link.rules.txt auxdev-sf-netdev-rename.sh.txt sf-rep-netdev-rename.sh.txt

  3. /etc/infiniband/auxdev-sf-netdev-rename.sh

Do not forget to have execution rights on it and remove the .txt extension from it.

paravmellanox commented 3 years ago

hi @bessa,

By now I assume it is working. Can you please close the issue, if its resolved.

Parav

paravmellanox commented 3 years ago

hi @bessa ,

Since this issue is more than 30+ days old. I am closing it. Please reopen if the issue exists. you can also use the Nvidia support channel to get it resolved.