can't sniffer packet in sr-iov

[flyhigao]

hi , I follow the readme , and running "4. Test it out - SRIOV mode" chapter fine ,it's working . now i want to run tcpdump in container with sr-iov in privileged mode:

my command:

docker pull rdma/sriov-plugin
docker run -v /run/docker/plugins:/run/docker/plugins -v /etc/docker:/etc/docker -v /var/run:/var/run --net=host --name sriov-plugin-docker --privileged rdma/sriov-plugin

docker network create -d sriov --subnet=192.168.70.0/24 -o netdevice=enp7s0f2  -o privileged=1 enp7s0f2
docker run --net=enp7s0f2 -itd --name=busybox busybox
docker run --net=enp7s0f2 -itd --name=busybox1 busybox
docker run --net=enp7s0f2 -itd --name=tcpdump corfr/tcpdump 

When busybox ping tcpdump container , then the tcpdump can sniffer the incoming ping packet. But when I ping between the 2 busybox container, the tcpdump container can NOT sniffer the ping traffic .

I also try pktgen to host nic enp7s0f2 , the tcpdump container also can NOT sniffer the traffic .

In tcpdump container , i try set up promisc in eth0 ,but it fail:

/ # ifconfig eth0 promisc 
ifconfig: ioctl 0x8914 failed: Operation not permitted

do I make something wrong ? this is my host setup:

[root@localhost ~]# cat /etc/redhat-release 
CentOS Linux release 7.6.1810 (Core) 
[root@localhost ~]# uname -a
Linux localhost.localdomain 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# ls -l /sys/class/net/enp7s0f2/device/virtfn*
lrwxrwxrwx 1 root root 0 7月  17 06:09 /sys/class/net/enp7s0f2/device/virtfn0 -> ../0000:07:10.2
lrwxrwxrwx 1 root root 0 7月  17 06:09 /sys/class/net/enp7s0f2/device/virtfn1 -> ../0000:07:10.6
lrwxrwxrwx 1 root root 0 7月  17 06:09 /sys/class/net/enp7s0f2/device/virtfn2 -> ../0000:07:11.2
lrwxrwxrwx 1 root root 0 7月  17 06:09 /sys/class/net/enp7s0f2/device/virtfn3 -> ../0000:07:11.6
lrwxrwxrwx 1 root root 0 7月  17 06:09 /sys/class/net/enp7s0f2/device/virtfn4 -> ../0000:07:12.2
lrwxrwxrwx 1 root root 0 7月  17 06:09 /sys/class/net/enp7s0f2/device/virtfn5 -> ../0000:07:12.6
lrwxrwxrwx 1 root root 0 7月  17 06:09 /sys/class/net/enp7s0f2/device/virtfn6 -> ../0000:07:13.2
[root@localhost ~]# lspci
....
07:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
07:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
07:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
07:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
07:10.0 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)
07:10.1 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)
07:10.2 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)
07:10.3 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)
07:10.4 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)
07:10.5 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)
07:10.6 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)
07:10.7 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)
07:11.0 Ethernet controller: Intel Corporation I350 Ethernet Controller Virtual Function (rev 01)
.....

thank you for your work!

[paravmellanox]

@flyhigao as by design containers cannot sniff traffic unless given a permission to do so. -o privileged gives privilege to netdevice to receive packets which are not destined to it. But you still need to give permission to container using CAP_NET_RAW, CAP_NET_ADMIN to allow tcpdump to put the netdevice in promiscuous mode.

So when you run container using docker run --net=enp7s0f2 -itd --name=tcpdump corfr/tcpdump

add either --privileged or do --caps=NET_RAW --cap=NET_ADMIN.

[flyhigao]

@paravmellanox thanks for the super fast reply :)
but curious , even I add privilied to tcpdump container , and set container eth0 promisc successful .the tcpdump still can't sniffer the promisc packet ,while the host nic RX is increasing(pktgen from other machine).

now my command is :

docker pull rdma/sriov-plugin
ifconfig enp7s0f2 promisc
docker run -v /run/docker/plugins:/run/docker/plugins -v /etc/docker:/etc/docker -v /var/run:/var/run --net=host --name sriov-plugin-docker --privileged rdma/sriov-plugin

docker network create -d sriov --subnet=192.168.70.0/24 -o netdevice=enp7s0f2  -o privileged=1  enp7s0f2
docker run --net=enp7s0f2    --name=tcpdump --privileged --cap-add=NET_RAW --cap-add=NET_ADMIN --rm corfr/tcpdump

in my host:

[root@localhost ~]# ifconfig enp7s0f2
enp7s0f2: flags=4419<UP,BROADCAST,RUNNING,**PROMISC**,MULTICAST>  mtu 1500
        ether ec:d6:8a:2c:7d:f9  txqueuelen 1000  (Ethernet)
        RX packets **13368953**  bytes 814420040 (776.6 MiB)

in my tcpdump container:

/ # ifconfig eth0 
eth0      Link encap:Ethernet  HWaddr A2:69:20:54:9C:AD  
          inet addr:192.168.70.2  Bcast:192.168.70.255  Mask:255.255.255.0
          UP BROADCAST RUNNING **PROMISC** MULTICAST  MTU:1500  Metric:1
          RX packets:**711** errors:0 dropped:0 overruns:0 frame:0
          TX packets:660 errors:0 dropped:0 overruns:0 carrier:0

my host nic is intel I350 , do i need specific nic card to enable the packet sniffer? or do i still miss sth ?

[paravmellanox]

@flyhigao , packet counter should increment on eth0 if you are sending it to this VF. That will ensure that packets are targeted to the VF. You could as well run a basic ping test from VF to some other system or VF to ensure that VF is currently doing packet tx/rx. While that is going on, you run tcpdump.

I do not know about I350 nics. We have tested it using Mellanox ConnectX4, ConnectX5 NICs.

[flyhigao]

I want tcpdump container eth0 run in Promiscuous Mode which can sniffer all the traffic on the host NIC. can sr-iov support this promiscuous mode ?

[paravmellanox]

@flyhigao, VF cannot sniff of host NIC. VF can sniff its own traffic. It doesn't matter container/no-container.

[flyhigao]

i search the Mellanox doc . VF Promiscuous Rx Modes And in VF Promiscuous Rx Modes chapter,it says :

VFs can enter a promiscuous mode that enables receiving the unmatched traffic and all the multicast traffic that reaches the physical port in addition to the traffic originally targeted to the VF. The unmatched traffic is any traffic's DMAC that does not match any of the VFs' or PFs' MAC addresses. Note: Only privileged/trusted VFs can enter the VF promiscuous mode.

So , I think maybe some nic card support this function . and intel i350 doesn't suppot it . thank you @paravmellanox

[paravmellanox]

@flyhigao , notice the word "unmatched". It cannot sniff matched traffic of PF. Hope you got the point.

Can you please close the case as its not related to container or this plugin.

[paravmellanox]

@flyhigao thanks.