The vhost user protocol supports live migration, and during live migration, the vhost-user frontend needs to track the modifications the vhost-user backend makes to the memory mapped regions, marking the dirty pages in a log (i.e., a bitmap).
If the backend has the VHOST_USER_PROTOCOL_F_LOG_SHMFD protocol feature it will receive the VHOST_USER_SET_LOG_BASE message with a file descriptor of the dirty-pages log memory region. This log covers all known guest addresses, and must be manipulated atomically.
This commit adds support for creating an atomic-memory-mapped bitmap, and being able to replace it in runtime. The vhost user protocol does not specify whether the previous bitmap is still active after replying to the VHOST_USER_SET_LOG_BASE message, so we use an RwLock to be sure that the in-flight requests are using the new bitmap after the message reply.
This PR is intended to be as unobtrusive as possible, but in a future PR, I think we can make some improvements to eliminate code duplication:
1- Adding a parameter to NewBitmap::with_len() with the starting address of the memory region, Thus the region_start_addr: GuestAddress and region_len: GuestUsize parameters of BitmapMmap::from_file() could be removed.
2- move Bitmap::split_at() to its own trait because we cannot implement it for AtomicBitmapMmap (and I can get rid of BitmapReplace::InnerBitmap)
3- Make AtomicBitmap::new() take the bitmap's memory as parameter with a type bounded to Index (and a len() method). So, the code in AtomicBitmapMmap would be minimum.
(there is more duplicate code that could be unified, since all the mmap code here, and in mmap_unix and mmap_xen do similar things, but I think that is beyond the scope of this PR.)
Summary of the PR
The vhost user protocol supports live migration, and during live migration, the vhost-user frontend needs to track the modifications the vhost-user backend makes to the memory mapped regions, marking the dirty pages in a log (i.e., a bitmap).
If the backend has the
VHOST_USER_PROTOCOL_F_LOG_SHMFD
protocol feature it will receive theVHOST_USER_SET_LOG_BASE
message with a file descriptor of the dirty-pages log memory region. This log covers all known guest addresses, and must be manipulated atomically.For further info please see https://qemu-project.gitlab.io/qemu/interop/vhost-user.html#migration
This commit adds support for creating an atomic-memory-mapped bitmap, and being able to replace it in runtime. The vhost user protocol does not specify whether the previous bitmap is still active after replying to the VHOST_USER_SET_LOG_BASE message, so we use an
RwLock
to be sure that the in-flight requests are using the new bitmap after the message reply.This version is currently being used in a version of virtiofsd that supports migration: https://gitlab.com/virtiofsd-live-migration
and is intended to be used as: https://github.com/rust-vmm/vhost/blob/05c4efa6b04a44a23092d9439fab8385b9148394/crates/vhost-user-backend/src/handler.rs#L616
Note to reviewers
This PR is intended to be as unobtrusive as possible, but in a future PR, I think we can make some improvements to eliminate code duplication: 1- Adding a parameter to
NewBitmap::with_len()
with the starting address of the memory region, Thus theregion_start_addr: GuestAddress
andregion_len: GuestUsize
parameters ofBitmapMmap::from_file()
could be removed.2- move
Bitmap::split_at()
to its own trait because we cannot implement it forAtomicBitmapMmap
(and I can get rid ofBitmapReplace::InnerBitmap
)3- Make
AtomicBitmap::new()
take the bitmap's memory as parameter with a type bounded toIndex
(and alen()
method). So, the code inAtomicBitmapMmap
would be minimum.(there is more duplicate code that could be unified, since all the mmap code here, and in mmap_unix and mmap_xen do similar things, but I think that is beyond the scope of this PR.)