Cannot load library if new fields are added in sabi_trait trait

[SilverMira]

Hello, I'm currently trying out this crate to implement a plugin system.

Although the docs seem to suggest that I can load a library that's compiled against a previous minor version of the shared interface, I'm getting a AbiInstability error when trying to load.

I'm not sure whether I'm doing anything wrong which made it not work.

// interface: 0.1.0
#[repr(u8)]
#[derive(StableAbi, Debug)]
#[sabi(kind(WithNonExhaustive(size = [usize; 8], traits(Debug))))]
#[non_exhaustive]
pub enum PluginError {
    NotSupported,
}

#[sabi_trait]
pub trait Plugin {
    #[sabi(last_prefix_field)]
    fn hello(&self, value: RStr<'_>) -> RResult<RString, NonExhaustiveFor<PluginError>> {
        RResult::RErr(NonExhaustiveFor::new(PluginError::NotSupported))
    }
}

pub type PluginBox = Plugin_TO<'static, RBox<()>>;

#[repr(C)]
#[derive(StableAbi)]
#[sabi(kind(Prefix(prefix_ref = PluginLibRef)))]
#[sabi(missing_field(panic))]
pub struct PluginLib {
    #[sabi(last_prefix_field)]
    pub new_plugin: extern "C" fn() -> PluginBox,
}

impl RootModule for PluginLibRef {
    abi_stable::declare_root_module_statics! {PluginLibRef}

    const BASE_NAME: &'static str = "plugin-interface";

    const NAME: &'static str = "plugin-interface";

    const VERSION_STRINGS: VersionStrings = package_version_strings!();
}

With the above interface crate, I'm able to compile and run hello() without issues. However if I update the Plugin trait as such and try to rebuild & run the main executable without recompiling the library, I'm getting a AbiInstability error "Too many fields" while loading the library.

// interface: 0.1.1 
#[sabi_trait]
pub trait Plugin {
    #[sabi(last_prefix_field)]
    fn hello(&self, value: RStr<'_>) -> RResult<RString, NonExhaustiveFor<PluginError>> {
        RResult::RErr(NonExhaustiveFor::new(PluginError::NotSupported))
    }
    fn not_implemented_yet(&self) -> RResult<RString, NonExhaustiveFor<PluginError>> {
        RResult::RErr(NonExhaustiveFor::new(PluginError::NotSupported))
    }
}

Logging the AbiInstability error

```sh AbiInstability(Compared : --- Type Layout --- type:PrefixRef<'a, PluginLib> size:8 align:8 package:'abi_stable' version:'0.11.3' line:392 mod:abi_stable::prefix_type::prefix_ref data: Struct with Fields: Phantom fields: field_name:0 type:PluginLib size:8 align:8 package:'interface' version:'0.1.1'Tag: null Extra checks: Repr attribute:Transparent Module reflection mode:DelegateDeref { layout_index: 0 } To : --- Type Layout --- type:PrefixRef<'a, PluginLib> size:8 align:8 package:'abi_stable' version:'0.11.3' line:392 mod:abi_stable::prefix_type::prefix_ref data: Struct with Fields: Phantom fields: field_name:0 type:PluginLib size:8 align:8 package:'interface' version:'0.1.1'Tag: null Extra checks: Repr attribute:Transparent Module reflection mode:DelegateDeref { layout_index: 0 } 0 error(s). 0 error(s)inside: field_name:0 type:PluginLib size:8 align:8 package:'interface' version:'0.1.1' Layout of expected type: --- Type Layout --- type:PluginLib size:8 align:8 package:'interface' version:'0.1.1' line:12 mod:interface data: Prefix type: first_suffix_field:1 conditional_prefix_fields: 0 fields: field_name:new_plugin type:AFunctionPointer size:8 align:8 package:'abi_stable' version:'0.11.3' fn pointer(s): fn()->Plugin_TO<'lt> accessible_fields: [Yes] Tag: null Extra checks: Repr attribute:C Module reflection mode:Module Layout of found type: --- Type Layout --- type:PluginLib size:8 align:8 package:'interface' version:'0.1.1' line:12 mod:interface data: Prefix type: first_suffix_field:1 conditional_prefix_fields: 0 fields: field_name:new_plugin type:AFunctionPointer size:8 align:8 package:'abi_stable' version:'0.11.3' fn pointer(s): fn()->Plugin_TO<'lt> accessible_fields: [Yes] Tag: null Extra checks: Repr attribute:C Module reflection mode:Module 0 error(s)inside: field_name:0 type:PluginLib size:8 align:8 package:'interface' version:'0.1.1' field_name:new_plugin type:AFunctionPointer size:8 align:8 package:'abi_stable' version:'0.11.3' fn pointer(s): fn()->Plugin_TO<'lt> fn()->Plugin_TO<'lt> field_name:__returns type:Plugin_TO<'lt> size:24 align:8 package:'interface' version:'0.1.1' Layout of expected type: --- Type Layout --- type:Plugin_TO<'lt> size:24 align:8 package:'interface' version:'0.1.1' line:38 mod:interface::Plugin_trait data: Struct with Fields: field_name:obj type:RObject<'lt> size:24 align:8 package:'abi_stable' version:'0.11.3' lifetime indices:Array(ArrayLen { len: 1, array: [[Param(0), NONE], [NONE, NONE], [NONE, NONE]] }) field_name:_marker type:UnsafeIgnoredType size:0 align:1 package:'abi_stable' version:'0.11.3' Tag: null Extra checks: Repr attribute:Transparent Module reflection mode:Module Layout of found type: --- Type Layout --- type:Plugin_TO<'lt> size:24 align:8 package:'interface' version:'0.1.1' line:38 mod:interface::Plugin_trait data: Struct with Fields: field_name:obj type:RObject<'lt> size:24 align:8 package:'abi_stable' version:'0.11.3' lifetime indices:Array(ArrayLen { len: 1, array: [[Param(0), NONE], [NONE, NONE], [NONE, NONE]] }) field_name:_marker type:UnsafeIgnoredType size:0 align:1 package:'abi_stable' version:'0.11.3' Tag: null Extra checks: Repr attribute:Transparent Module reflection mode:Module 0 error(s)inside: field_name:0 type:PluginLib size:8 align:8 package:'interface' version:'0.1.1' field_name:new_plugin type:AFunctionPointer size:8 align:8 package:'abi_stable' version:'0.11.3' fn pointer(s): fn()->Plugin_TO<'lt> fn()->Plugin_TO<'lt> field_name:__returns type:Plugin_TO<'lt> size:24 align:8 package:'interface' version:'0.1.1' field_name:obj type:RObject<'lt> size:24 align:8 package:'abi_stable' version:'0.11.3' lifetime indices:Array(ArrayLen { len: 1, array: [[Param(0), NONE], [NONE, NONE], [NONE, NONE]] }) Layout of expected type: --- Type Layout --- type:RObject<'lt> size:24 align:8 package:'abi_stable' version:'0.11.3' line:77 mod:abi_stable::sabi_trait::robject data: Struct with Fields: field_name:vtable type:PrefixRef<'a, VTable> size:8 align:8 package:'abi_stable' version:'0.11.3' field_name:ptr type:ManuallyDrop size:16 align:8 package:'std' version:'1.0.0' field_name:_marker type:PhantomData<&'a () , AFunctionPointer> size:0 align:1 package:'std' version:'1.0.0' fn pointer(s): fn()->Plugin_Interface lifetime indices:Array(ArrayLen { len: 1, array: [[Param(0), NONE], [NONE, NONE], [NONE, NONE]] }) Tag: null Extra checks: RequiredTraits Auto traits: Impld traits: Repr attribute:C Module reflection mode:Opaque Layout of found type: --- Type Layout --- type:RObject<'lt> size:24 align:8 package:'abi_stable' version:'0.11.3' line:77 mod:abi_stable::sabi_trait::robject data: Struct with Fields: field_name:vtable type:PrefixRef<'a, VTable> size:8 align:8 package:'abi_stable' version:'0.11.3' field_name:ptr type:ManuallyDrop size:16 align:8 package:'std' version:'1.0.0' field_name:_marker type:PhantomData<&'a () , AFunctionPointer> size:0 align:1 package:'std' version:'1.0.0' fn pointer(s): fn()->Plugin_Interface lifetime indices:Array(ArrayLen { len: 1, array: [[Param(0), NONE], [NONE, NONE], [NONE, NONE]] }) Tag: null Extra checks: RequiredTraits Auto traits: Impld traits: Repr attribute:C Module reflection mode:Opaque 0 error(s)inside: field_name:0 type:PluginLib size:8 align:8 package:'interface' version:'0.1.1' field_name:new_plugin type:AFunctionPointer size:8 align:8 package:'abi_stable' version:'0.11.3' fn pointer(s): fn()->Plugin_TO<'lt> fn()->Plugin_TO<'lt> field_name:__returns type:Plugin_TO<'lt> size:24 align:8 package:'interface' version:'0.1.1' field_name:obj type:RObject<'lt> size:24 align:8 package:'abi_stable' version:'0.11.3' lifetime indices:Array(ArrayLen { len: 1, array: [[Param(0), NONE], [NONE, NONE], [NONE, NONE]] }) field_name:vtable type:PrefixRef<'a, VTable> size:8 align:8 package:'abi_stable' version:'0.11.3' Layout of expected type: --- Type Layout --- type:PrefixRef<'a, VTable> size:8 align:8 package:'abi_stable' version:'0.11.3' line:392 mod:abi_stable::prefix_type::prefix_ref data: Struct with Fields: Phantom fields: field_name:0 type:VTable size:16 align:8 package:'interface' version:'0.1.1'Tag: null Extra checks: Repr attribute:Transparent Module reflection mode:DelegateDeref { layout_index: 0 } Layout of found type: --- Type Layout --- type:PrefixRef<'a, VTable> size:8 align:8 package:'abi_stable' version:'0.11.3' line:392 mod:abi_stable::prefix_type::prefix_ref data: Struct with Fields: Phantom fields: field_name:0 type:VTable size:16 align:8 package:'interface' version:'0.1.1'Tag: null Extra checks: Repr attribute:Transparent Module reflection mode:DelegateDeref { layout_index: 0 } 1 error(s)inside: field_name:0 type:PluginLib size:8 align:8 package:'interface' version:'0.1.1' field_name:new_plugin type:AFunctionPointer size:8 align:8 package:'abi_stable' version:'0.11.3' fn pointer(s): fn()->Plugin_TO<'lt> fn()->Plugin_TO<'lt> field_name:__returns type:Plugin_TO<'lt> size:24 align:8 package:'interface' version:'0.1.1' field_name:obj type:RObject<'lt> size:24 align:8 package:'abi_stable' version:'0.11.3' lifetime indices:Array(ArrayLen { len: 1, array: [[Param(0), NONE], [NONE, NONE], [NONE, NONE]] }) field_name:vtable type:PrefixRef<'a, VTable> size:8 align:8 package:'abi_stable' version:'0.11.3' field_name:0 type:VTable size:16 align:8 package:'interface' version:'0.1.1' Layout of expected type: --- Type Layout --- type:VTable size:16 align:8 package:'interface' version:'0.1.1' line:38 mod:interface::Plugin_trait data: Prefix type: first_suffix_field:3 conditional_prefix_fields: 0 fields: field_name:_sabi_tys type:PhantomData<(), RBox> size:0 align:1 package:'std' version:'1.0.0' field_name:_sabi_vtable type:PrefixRef<'a, RObjectVtable> size:8 align:8 package:'abi_stable' version:'0.11.3' field_name:hello type:AFunctionPointer size:8 align:8 package:'abi_stable' version:'0.11.3' fn pointer(s): unsafe fn(_self,param_0,: RRef<'a>, param_1: RStr<'a>)->RResult lifetime indices:Array(ArrayLen { len: 1, array: [[ANONYMOUS, ANONYMOUS], [NONE, NONE], [NONE, NONE]] }) field_name:not_implemented_yet type:AFunctionPointer size:8 align:8 package:'abi_stable' version:'0.11.3' fn pointer(s): unsafe fn(_self,: RRef<'a>)->RResult lifetime indices:Array(ArrayLen { len: 1, array: [[ANONYMOUS, NONE], [NONE, NONE], [NONE, NONE]] }) accessible_fields: [Yes, Yes, Yes, Yes] Tag: null Extra checks: Repr attribute:C Module reflection mode:Module Layout of found type: --- Type Layout --- type:VTable size:16 align:8 package:'interface' version:'0.1.1' line:38 mod:interface::Plugin_trait data: Prefix type: first_suffix_field:3 conditional_prefix_fields: 0 fields: field_name:_sabi_tys type:PhantomData<(), RBox> size:0 align:1 package:'std' version:'1.0.0' field_name:_sabi_vtable type:PrefixRef<'a, RObjectVtable> size:8 align:8 package:'abi_stable' version:'0.11.3' field_name:hello type:AFunctionPointer size:8 align:8 package:'abi_stable' version:'0.11.3' fn pointer(s): unsafe fn(_self,param_0,: RRef<'a>, param_1: RStr<'a>)->RResult lifetime indices:Array(ArrayLen { len: 1, array: [[ANONYMOUS, ANONYMOUS], [NONE, NONE], [NONE, NONE]] }) accessible_fields: [Yes, Yes, Yes] Tag: null Extra checks: Repr attribute:C Module reflection mode:Module Error:too many fields Expected: 4 Found: 3 ) ```

[thorio]

It seems to me like extensibility is there to support loading newer libraries than expected, essentially working in the wrong direction to what's needed for a plugin system.

Try compiling the plugin with interface 0.1.1, then loading it with interface 0.1.0. This works in my experience, but is really the opposite of what I wanted.

---

EDIT: If instead of loading the plugin like this:

let lib = PluginLibRef::load_from_file(&PathBuf::from("tests/data/libexample_provider.so"))
    .expect("plugin must be loadable");

You check the version and type layout yourself, reversing the expected and actual layouts:

let header = lib_header_from_path(&PathBuf::from("tests/data/libexample_provider.so"))
    .expect("plugin library header must be loadable");

// TODO check version strings manually

if let IsLayoutChecked::Yes(layout) = header.root_mod_consts().layout() {
    // Note the full path here, the abi_checking module is hidden from documentation
    // Also note the arguments have been reversed, passing the plugin's layout first
    abi_stable::abi_stability::abi_checking::check_layout_compatibility(layout, PluginLibRef::LAYOUT)
        .expect("interface must be compatible");
};

let lib = unsafe {
    header
        .unchecked_layout::<PluginLibRef>()
        .expect("plugin broke while loading")
};

Then everything seems to work like you would want for a plugin system. I can add new methods/nonexhaustive variants to the interface and still load older libraries, but not the other way around. No clue whether this is still safe to use, but it seems to behave correctly for now.

[SilverMira]

@thorio, that's a good idea for validating the layout, in my testing I did managed to use the unsafe functions to load the library which ignored layout validation, and everything seems to work as expected. My version of ensuring compatibility was a bit different, I used the same check_layout_compatibility function without reversing the arguments but instead permissively passes the validation if the error detected is just AbiInstability::FieldCountMismatch where expected count is more than actual count

fn ensure_compatibility(
    interface: &'static TypeLayout,
    implementation: &'static TypeLayout,
) -> Result<(), AbiInstabilityErrors> {
    let compatibility = abi_stable::abi_stability::abi_checking::check_layout_compatibility(
        interface,
        implementation,
    );
    if let Err(err) = compatibility {
        let incompatibilities = err.errors.iter().filter(|e| !e.errs.is_empty());
        let fatal_incompatibilities = incompatibilities.filter(|err| {
            err.errs.iter().any(|err| {
                !matches!(
                    err,
                    AbiInstability::FieldCountMismatch(assert) if assert.expected > assert.found
                )
            })
        });
        if fatal_incompatibilities.count() > 0 {
            return Err(err);
        }
    }
    Ok(())
}

Checking the generated code for the Trait_TO, this part of the docs does work as expected (ie: default implementation does get invoked if there's no corresponding VTable entry in the library layout).

Accidentally calling newer methods on trait objects from older versions of a library will cause a panic at runtime, unless it has a default implementation (within the trait definition that #[sabi_trait] can see).

It's just the layout checking logic that seems to be contradicting what the docs say.

A library will not load (through safe means) if methods are added anywhere but the end.

[SilverMira]

For people who have stumbled upon this same plugin use case, I have since changed my approach to implementing a plugin system.

Check out remoc's rtc feature which can create the same sort of RPC capability like sabi_trait as long as there is a binary pipe between 2 ends (even across networks). In fact, you can even combine sabi with remoc, with sabi providing the abi stable pipe to a dylib using the channels feature. Since remoc is based on serde, types with Serialize and Deserialize can just work easily without plumbing StableAbi, though of course, serializing/deserializing will have more overhead over just doing pure FFI