All vhca resources can be seen in the container even only one rdma/vhca is requested.

[flymark2010]

I 've create a test pod from test-sriov-pod.yaml, in which only one rdma/vhca is requested, and the get in the container with kubectl exec . When I run the command ibv_devinfo in the container, I find that all the vhca devices cab be seen in the container. Is that normal?

[paravmellanox]

@flymark2010. Yes, this fine for now. Even if container sees it, RDMA QPs won't be able to send/receive from those devices. They will be hidden in future releases. You should use ibdev2net or show_gids script to use rdma device for that container.

[flymark2010]

In the test pod, I can see all the vhca devices, but with ifconfig command , I can only see 2 Ethernet interface:

# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.0.251.73  netmask 255.255.255.255  broadcast 0.0.0.0
        ether e6:71:29:34:60:1f  txqueuelen 0  (Ethernet)
        RX packets 69  bytes 10725 (10.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

And nothing with show_gids:

# show_gids
DEV PORT    INDEX   GID                 IPv4        VER DEV
--- ----    -----   ---                 ------------    --- ---

n_gids_found=0

Actually, I'm not familiar with rdma device using, do you have more detail tutorials? Thank you very much!

[paravmellanox]

@flymark2010 two device seems correct. device plugin also seems to work. However show_gids should have shown GID entries (which maps to this IP addresses).

so one possibility is the eth0 device is not the sriov vf but some other plugin netdevice. This typically happens if you have cni conf file of other plugin such as weave or flannel or others. If that plugin is used, it will use create the eth0 device of it.

So make sure you have only one cni conf file in /etc/cni/net.d directory as /etc/cni/net.d//10-sriov-cni.conf.

If you have ethtool than you can see vendor and driver of the eth0 like ethtool -i eth0 to get information about it.

We have one reference document like below, but you have done most steps of it. https://community.mellanox.com/docs/DOC-3151

There is also a MOFED user space image prebuild with MOFED and tools installed here. https://hub.docker.com/r/mellanox/centos_7_2_mofed_4_4_0_1_9_0/ (not for this problem, but in general to make use of).

[flymark2010]

Actually we've calico cni in ours k8s cluster, and I think the eth0 should be created by calico. And now ours k8s cluster is running with calico, I'm afraid of stop the calico plugin. I've to think a way to try what you've said. Thanks a lot!

[paravmellanox]

@flymark2010 you dont need to stop the calico plugin, There is multus plugin which allows using multiple other cni plugin such as calico and sriov at same time. By using that you will get eth0 of calico and net0 from sriov. I don't have exact config steps but it is tested in our lab too that way.

[flymark2010]

Thanks for your reminder, and I found that the kubernetes documen mentioned that "If there are multiple CNI configuration files in the directory, the first one in lexicographic order of file name is used." . I tried to use calico and sriov the same time, and just rename the calico cni configuration file on the test node to make sure the sriov cni configuration file to be the first order. Then the kubelet use the sriov cni to create network for the test Pod.

In the Pod, there's only a eth0. Here is the result for command ethtool -i eth0:

driver: mlx5_core
version: 4.4-1.0.0
firmware-version: 14.23.1000 (MT_2420110034)
expansion-rom-version: 
bus-info: 0000:82:00.2
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

I think this should be right. Thanks very much!

[paravmellanox]

@flymark2010 . Great. Thanks for using it. happy to help.