BSDRP + sr-iov and bhyve ?

[benoitc]

I'm thinking to try bsdrp with sr-iov and bhyve. Has it been already tested? What about installing it with vm-bhyve?

[ocochard]

It should work: I've tried it years ago, but I was disappointed by the number of limited queues assigned to the virtual interfaces (it was one or two queues, so very bad performances). I didn't tried it since, but the first step should be to check those number and increase them. My old notes about, bhyve host setup:

# cat /etc/rc.conf
(...)
vm_enable="YES"
vm_dir="/usr/local/VMs"
iovctl_files="/etc/iovctl.cxl1.conf"
# cat /etc/iovctl.cxl1.conf
PF {
device : "cxl1";
num_vfs : 4;
}
DEFAULT {

passthrough : true;

}
# pciconf -l | grep ppt
ppt0@pci0:179:0:9:      class=0x020000 rev=0x00 hdr=0x00 vendor=0x1425 device=0x5810 subvendor=0x1425 subdevice=0x0000
ppt1@pci0:179:0:13:     class=0x020000 rev=0x00 hdr=0x00 vendor=0x1425 device=0x5810 subvendor=0x1425 subdevice=0x0000
ppt2@pci0:179:0:17:     class=0x020000 rev=0x00 hdr=0x00 vendor=0x1425 device=0x5810 subvendor=0x1425 subdevice=0x0000
ppt3@pci0:179:0:21:     class=0x020000 rev=0x00 hdr=0x00 vendor=0x1425 device=0x5810 subvendor=0x1425 subdevice=0x0000
# cat /usr/local/VMs/fbsd/fbsd.conf
(...)
passthru0="179/0/9"
passthru1="179/0/13"

And the VM config:

# kldload if_cxgbev
t5vf0: <Chelsio T580-LP-CR VF> mem 0xc0018000-0xc0018fff,0xc0000000-0xc0007fff,0xc0014000-0xc0015fff at device 6.0 on pci0
t5vf0: 1 ports, 2 MSI-X interrupts, 4 eq, 2 iq
t5vf1: <Chelsio T580-LP-CR VF> mem 0xc0019000-0xc0019fff,0xc0008000-0xc000ffff,0xc0016000-0xc0017fff at device 6.1 on pci0
t5vf1: 1 ports, 2 MSI-X interrupts, 4 eq, 2 iq
cxlv0: <port 0> on t5vf0
cxlv0: 2 txq, 1 rxq (NIC)
cxlv1: <port 0> on t5vf1
cxlv1: 2 txq, 1 rxq (NIC)

[benoitc]

I tried it on the mellanox connect4x-lx 2x25G I have good performances once you increase the number of channels. Using 8 cpus I get this:

[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.73 GBytes  23.4 Gbits/sec    0   3.01 MBytes
[  5]   1.00-2.00   sec  2.74 GBytes  23.5 Gbits/sec    0   3.01 MBytes
[  5]   2.00-3.00   sec  2.72 GBytes  23.4 Gbits/sec    0   3.01 MBytes
[  5]   3.00-4.00   sec  2.67 GBytes  22.9 Gbits/sec    0   3.01 MBytes
[  5]   4.00-5.00   sec  2.70 GBytes  23.2 Gbits/sec    0   3.01 MBytes
[  5]   5.00-6.00   sec  2.70 GBytes  23.2 Gbits/sec    0   3.01 MBytes
[  5]   6.00-7.00   sec  2.66 GBytes  22.8 Gbits/sec    0   3.01 MBytes
[  5]   7.00-8.00   sec  2.64 GBytes  22.7 Gbits/sec    0   3.01 MBytes
[  5]   8.00-9.00   sec  2.65 GBytes  22.7 Gbits/sec    0   3.01 MBytes
[  5]   9.00-10.00  sec  2.63 GBytes  22.6 Gbits/sec    0   3.01 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  26.8 GBytes  23.1 Gbits/sec    0             sender
[  5]   0.00-10.12  sec  26.8 GBytes  22.8 Gbits/sec                  receiver

Once thing I am not sure is how to configure the vm for the filesystem. Should I give it 2 disks to have the same features for the backup?

[benoitc]

closing the issue as answered. I think the lst step I need to figure is how perform netgraph. Such thing slike ng_patch are quite handy.