vhost-user design

[sboeuf]

We need a clear story on how vhost-user is going to be supported through rust-vmm crates. There are three main areas which need to be addressed in order to get it working with current hypervisors (crosvm and firecracker):

• vhost-user protocol support
• VM memory sharing support
• virtio devices consuming those crates

vhost-user protocol

There is some very nice work started by @jiangliu here. This crate development looks almost ready. He implemented both master and slave side of the protocol, even if we care only about the master side for vhost-user support in any hypervisor. The slave implementation is useful for unit testing though. There are a few tests missing to validate the right behavior of this crate regarding the expected vhost-user protocol.

I think if that's fine with everybody, we would create a crate vhost-user under rust-vmm, and @jiangliu could submit his code there, where some real code reviews could happen. What do you all think?

VM memory sharing

Again, @jiangliu submitted some code to the firecracker codebase here, but everybody agreed during the rust-vmm meeting that it should be part of a separate crate that could be leveraged from every hypervisor. We agreed on the creation of a memory-model crate that would hold the code responsible for the way the guest memory should/could be managed.

Such crate will allow any virtio device using vhost-user to share some memory regions of the guest address space corresponding to the virtqueues that need to be accessed from the vhost-user backend running on the host.

@jiangliu has just created the crate on rust-vmm organization, but I would suggest that we remove the initial code to submit proper PRs that everybody could review. Any objection here?

virtio devices

The last piece of the puzzle is to try the new vhost-user crate for a real use case, and we need to write devices for that. Having a vhost-user-net device to be used with DPDK or a vhost-user-fs device to be used with the libfuse daemon, as discussed during rust-vmm meeting, would be good starting points for such devices.

Feeback

/cc @andreeaflorescu @mcastelino @sameo @rbradford @jiangliu @zachreizner

Please let me know if I missed anything here, and cc more people for who this could be relevant!

[dagrh]

I'm gently looking at using @jiangliu's crate for writing the daemon side of a virtio-fs daemon; but it's pretty low down my list but I'll try and make some more time to look.

[andreeaflorescu]

I would only have one mention. We should make sure that the memory-model crate is suitable for both CrosVM and Firecracker usecases. I will do some tests with the memory-model crate next week. I think @alexandruag had some concerns with the memfd. I'll let him chime in.

[sboeuf]

@andreeaflorescu

We should make sure that the memory-model crate is suitable for both CrosVM and Firecracker usecases

Yes I agree with that, that's why I was mentioning we should erase what's in the repo, start from scratch, and submit PRs that everybody can look at.

[jiangliu]

@andreeaflorescu @mcastelino @sameo @rbradford @jiangliu @zachreizner @sboeuf @dagrh I'm thinking about the relationship between vhostuser_rs and vhost in crosvm/firecracker. The crosVM and firecracker projects already have a kernel-based(ioctl) vhost implementation. And there are vhost drivers for net/vsock. So should we merge them into one unified crate with an architecture as below:

[andreeaflorescu]

It looks to me that with this architecture you are assuming that the backend for all virtio devices is a vhost. This is not the case in Firecracker as we use Tap devices as a backend for virtio-net. I can also look into this to see how we can accommodate backends that are not vhost. I am not sure how much the implementation changes when using vhost as backend, but one option would be to use Rust features to switch between vhost vs non-vhost backends.

[jiangliu]

It looks to me that with this architecture you are assuming that the backend for all virtio devices is a vhost. That may not be true. Per my understanding, the relationship among device, virtio and vhost looks like this:

So vhost may be positioned as an accelerator for virtio devices.

[sboeuf]

All devices are virtio devices (unless we talk about real devices using VFIO for instance), since they all uses virtqueues to communicate with the driver in the guest. But the main difference between all those devices is the backend, and we can classify the backends into 2 categories, vhost or non-vhost. And in case of vhost, we can have several sub-categories such as vhost-kernel or vhost-user. Another aspect of virtio devices, that is completely orthogonal from the backend is the transport layer being chosen, that can be either virtio-pci or virtio-mmio.

[jiangliu]

Exactly! So I propose to combine vhost-user and vhost-kernel into a unified vhost crate and provide a common interface trait for them.

On Feb 12, 2019, at 12:03 AM, Sebastien Boeuf <notifications@github.com mailto:notifications@github.com> wrote:

All devices are virtio devices, since they all uses virtqueues to communicate with the driver in the guest. But the main difference between all those devices is the backend, and we can classify the backends into 2 categories, vhost or non-vhost. And in case of vhost, we can have several sub-categories such as vhost-kernel or vhost-user. Another aspect of virtio devices, that is completely orthogonal from the backend is the transport layer being chosen, that can be either virtio-pci or virtio-mmio.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rust-vmm/community/issues/13#issuecomment-462383636, or mute the thread https://github.com/notifications/unsubscribe-auth/AB14_PYfkcwBMQgMNQcZQvPIEW4HDHeWks5vMZRpgaJpZM4axTLp.

[sboeuf]

SGTM

[dagrh]

a good chunk of vhost user is packing/unpacking the messages on the vhost-user socket and doing something useful with them 9Like mapping memory); which I assume is something that none of the other crates cases care about.