Crate Addition Request: vmm-serde to build serde instrastructure for rust-vmm

[jiangliu]

vmm-serde

Short Description

Build serde infrastructure to support snapshot, live migration and live upgrade

Why is this crate relevant to the rust-vmm project?

A common requirement for virtual machine snapshot, live migration and live upgrading is to serialize and deserialize VMM internal objects and data structures. The rust community has the excellent serde/serde_derive framework for data serialization/deserialization. But the serde/serde_derive framework have very heavy dependency chain, and it's a common concern of the rust-vmm project to reduce dependency chain as small as possible. So this helper crate uses rust features to opt-in/opt-out data serialization/deserialization code and dependencies.

Currently following features are supported:

• Feature export_as_pub. The data serializer often needs to access private data structures or private fields in data structures. But rust has no support for friend visibility yet, though there's proposals for C++ like friend relationship. So a proc macro export_as_pub() is introduced to rewrite private data structs/fields as pub on demand when the feature export_as_pub is enabled. Otherwise export_as_pub() becomes a null operation.

• Feature serde_derive. When this feature is enabled, the proc_macro_derive for De/Serialize is reexported from the serde_derive crate, otherwise #[derive(Serialize, Deserialize)] becomes a null operation.

• Feature serde_derive_ffi. Often the bindgen utiltily is used to auto-generate FFI bindings for Linux syscalls and system libraries. And data structures with flexible array member are often used for input and/or output parameters. So three derive macros, SerializeFfi, DeserializeFfi and DeserializeFfiFam, are introduced to serialize data structures generated by bindgen.

[aghecenco]

Couple of questions:

• is the purpose of this crate to replace serde? If so, why reinvent the wheel (serde has a mature, stable community and is extensively used)? And if not, how would this new crate reduce the dependencies if it pulls in serde anyway?
• re:field access: isn't it easier to add serialization logic near the definition of the data structure? What I mean is: having a rust-vmm crate that defines a struct Foo, we can write impl Serialize for Foo / #[derive(Serialize)] in the same crate code. What does the dedicated serialization crate gain?
• re:ffi: I'm no longer a fan of this approach for reasons listed here https://github.com/rust-vmm/kvm-bindings/pull/7#issuecomment-567521792, I'd rather not expose it as I find it to be a non-negligible attack surface
• re:custom serialization itself: with so many customizable bits and pieces, I fear that it would become increasingly difficult for this crate to achieve feature parity with plain serde - imagine a complex struct that has a bunch of members to be skipped, some with defaults, others with custom serializers. Again, if said struct is in rust-vmm and we come up with a standard feature name (we want to turn serialization on and off), we can add the logic right near the struct.

[jiangliu]

Couple of questions:

• is the purpose of this crate to replace serde? If so, why reinvent the wheel (serde has a mature, stable community and is extensively used)? And if not, how would this new crate reduce the dependencies if it pulls in serde anyway?

It's not aimed to replace serde, but conditionally enable serde on demand. One of the crate's purpose is to implement

[cfg_attr(feature = "with-serde", derive(Deserialize, Serialize))]

with just

[derive(Deserialize, Serialize)]

• re:field access: isn't it easier to add serialization logic near the definition of the data structure? What I mean is: having a rust-vmm crate that defines a struct Foo, we can write impl Serialize for Foo / #[derive(Serialize)] in the same crate code. What does the dedicated serialization crate gain?

Different subsystems have different requirements on the way to serialize/deserialize an object. Taking a KVM vcpu object as an example, snapshot/live migration needs to serialize/deserialize the vCPU, but live-upgrading only needs to serialize/deserialize the underline fd. This crate doesn't implement Serialize/Deserialize for data structs, but provides a way for external crate to access struct's private data fields.

/// Struct for the vhost-user master endpoint.
#[vmm_serde::export_as_pub()]
#[derive(Clone)]
pub struct Master {
    node: Arc<Mutex<MasterInternal>>,
}

The field node is private when feature "export_as_pub" is disabled, and it becomes pub when "export_as_pub" is enabled.

• re:ffi: I'm no longer a fan of this approach for reasons listed here rust-vmm/kvm-bindings#7 (comment), I'd rather not expose it as I find it to be a non-negligible attack surface

The ffi serde implementation has limitations for live migration/snapshot, but it's OK for live upgrading. I'm work to switch to the new serialize/deserialize for kvm-binding.

• re:custom serialization itself: with so many customizable bits and pieces, I fear that it would become increasingly difficult for this crate to achieve feature parity with plain serde - imagine a complex struct that has a bunch of members to be skipped, some with defaults, others with custom serializers. Again, if said struct is in rust-vmm and we come up with a standard feature name (we want to turn serialization on and off), we can add the logic right near the struct.

So to sumarize, this crate aims to: 1) conditionally enable #[derive(Serialize, Deserialize)] relying on those popular serde crates. 2) provides a mechanism to implement different serialization/deserialization for a single data structure. And doesn't aim to implement serialization/deserialization for any data struct.