Plugin wanted: SR-IOV Virtual Function counters collector

[EvgenyAGRO]

After a research on Intel SR-IOV, I've found that there is no convenient way to get the VF counters while VF interface is being attached to a VM for example(I've tried it on QEMU/KVM VMs using Libvirt). Furthermore, I didn't find even an inconvenient way to do it, because as soon as VM is being created on a HOST and it is bound in configuration to some VF interface(this is the main use case of SRI-OV VFs), the interface is automatically being detached from the HOST and attached to the VM (I think that it is done on purpose in order to allow passthrough to bypass the kernel or something), and there is actually no way that I found yet to find the counters of the VM on the HOST(I'm using Ubuntu 16.04 and I've tried looking for counters on VF ports, in ethtool -S on the PF interface but with no avail), those counters may be actually very helpful in various situations. It would be really helpful if someone who has some experience in dealing with SRIOV could shade some light on this problem, and if there is a way to get those counters somehow on the HOST, I would like to try and implement a SNAP plugin collecting them. Thanks!

[andrzej-k]

Hi @EvgenyAGRO I don't think there is an easy way to gather those metrics on the host level. I know one example when someone was using DPPD application from Intel DPDK library, and in this case application was configured to expose NIC stats on TCP port. And someone here suggested using router level netflow/sflow to get insight into VM metrics.

[EvgenyAGRO]

Hi @andrzej-k Unfortunately using router level collection is problematic in my environment but I'll try to experiment with DPPD app. Thanks for the advice.

[crab86]

It seems as if as least IBM has found a way to monitor SR-IOV interfaces on their PowerVM platform. But unfortunately i couldn't find any more details about it. Anyway just thought it might help somehow.

"Is there a way to monitor SR-IOV network activity?

Yes, there are a number of ways to monitor SR-IOV network activity, here are a few options.

• The hardware management console Performance and Capability Monitor (PCM) exposes SR-IOV adapter utilization with per Virtual Function & Partition breakdown. PCM provides both GUI and REST APIs support.

• The hardware management console exposes SR-IOV adapter physical port statistics and SR-IOV logical port statistics. GUI, CLI, and REST APIs are provided.

• Partition network performance/monitoring tools may also provide logical port performance statistics. SR-IOV logical ports appear as physical devices to a partition, therefore traditional network performance tools typically include SR-IOV logical port statistics. The specific tools are operating system dependent."

Source: IBM

[markseger]

I think I cracked the code and even wrote a plugin for collectl to get/display per-vm net stats like this, here I was scp-ing a 600MB file to it and am running my display on the compute host:

PID  THRD S   VSZ   RSS CP  SysT  UsrT Pct  N   AccumTim DskI DskO NetI NetO Instance
1609     4 S    7G    1G  8  0.00  0.00   0  2   01:26.79    0    0    0    0 000001dc
 1609     4 S    7G    1G  8  0.01  0.25  13  2   01:27.05    0    0 3667 4543 000001dc
 1609     4 S    7G    1G  0  0.04  0.80  42  2   01:27.89    0    0 371K 178M 000001dc
 1609     4 S    7G    1G  0  0.09  0.77  43  2   01:28.75    0    0 378K 196M 000001dc
 1609     4 S    7G    1G  6  0.08  0.81  44  2   01:29.64    0  76K 400K 199M 000001dc
 1609     4 S    7G    1G  7  0.05  0.38  21  2   01:30.07    0  87K 187K  94M 000001dc
 1609     7 S    7G    1G  6  0.01  0.04   2  2   01:30.12   60  84K  126  126 000001dc
 1609     7 S    7G    1G  6  0.00  0.01   0  2   01:30.13    0  16K    0    0 000001dc
 1609     7 S    7G    1G  6  0.00  0.00   0  2   01:30.13    0    0    0    0 000001dc```

what I did was find the mac address in the qemu command string, then map that to a network tap device name by looking at all the tap device mac addresses in /sys/devices/virtual/net. Then finally using that tap name to look here - /sys/devices/virtual/net/${tap}/statistics/rx_bytes. There's another entry for tx bytes and a whole lot of other stats but this is all I wanted for now.

comments/questions?

-mark

[chittaranjanpandamavenir]

Hi Mark,

Is it working for SRIOV VF? I am struggling for the same.

Thanks, -Chitta

[satishdotpatel]

I am also looking for SR-IOV VF stats ( we are seeing packet drops and i want to see where and how many)

[shitou520]

I am also want to know the drop packet stats for VF 。

[nikolakis89]

Do you have news on how to show the statistics of SRIOV interfaces through collectd?

[satishdotpatel]

I don't think SR-IOV support from OS level to obtain that counter for VF, i believe it required some driver level interface to get that data out, I heard some NIC company provide external software to extract that data.

[nikolakis89]

Which company @satishdotpatel ?

[satishdotpatel]

https://community.mellanox.com/s/article/openstack-ceilometer-sr-iov-counters-support