RFC: Support traits for interface macro

I would like to be able to implement the interface macro on traits, so that I can inject the concrete implementation into an interface struct that is generic on the trait I gave. This is useful for several things:

Mocking the zbus interface by passing in a custom implementation during testing.
Breaking the entire interface along with its proxy into a separate crate, to allow other services to consume without creating cyclic dependencies between crates

Sorry if this is too much detail. Hoping to know your thoughts.

Example of how it could work

Does not compile today, because interface macro doesn't support traits

#[interface(
    name = "org.foo.AuthTokenManager1",
    proxy(
        default_service = "org.foo.AuthTokenManager1",
        default_path = "/org/foo/AuthTokenManager1",
    )
)]
#[derive(Debug, derive_more::From)]
trait AuthTokenManagerT {
    #[zbus(property)]
    fn token(&self) -> zbus::fdo::Result<String>;

    fn force_token_refresh(
        &mut self,
        #[zbus(signal_context)] ctxt: zbus::SignalContext<'_>,
    );
}

This would then transform into the following code, which does compile today:

#[derive(Debug, derive_more::From)]
struct AuthTokenManager<T: AuthTokenManagerT>(pub T);

#[interface(
    name = "org.foo.AuthTokenManager1",
    proxy(
        default_service = "org.foo.AuthTokenManager1",
        default_path = "/org/foo/AuthTokenManager1",
    )
)]
impl<T: AuthTokenManagerT> AuthTokenManagerT for AuthTokenManager<T> {
    #[zbus(property)]
    fn token(&self) -> zbus::fdo::Result<String> {
        self.0.token()
    }

    fn force_token_refresh(
        &mut self,
        #[zbus(signal_context)] ctxt: zbus::SignalContext<'_>,
    ) {
        self.0.force_token_refresh(ctxt)
    }
}

This now means that users can simply do the following to bring their own behavior:


type AuthTokenManagerIface = AuthTokenManager<MyAuthTokenManager>;

/// Concrete implementation
struct MyAuthTokenManager {
    token: String,
}

impl MyAuthTokenManager {
    fn new(token: String) -> Self { Self { token } }
}

impl AuthTokenManagerT for MyAuthTokenManager {
    #[zbus(property)]
    fn token(&self) -> zbus::fdo::Result<String> {
        self.0.clone()
    }

    fn force_token_refresh(
        &mut self,
        #[zbus(signal_context)] ctxt: zbus::SignalContext<'_>,
    ) {
        todo!("some more stuff here, idk")
    }
}

fn create_my_interface {
    // Create our concrete implementation
    let my_auth_token_manager = MyAuthTokenManager::new(String::from("foobar"));
    // Construct the interface (dependency injection)
    let iface = AuthTokenManagerIface::from(my_auth_token_manager);
    let dbus = ConnectionBuilder::session()?
        .name("org.worldcoin.AuthTokenManager1")?
        .serve_at("/org/worldcoin/AuthTokenManager1", iface)?
        .build()
        .await?;
}

The first code block above is what this RFC is proposing. The second code block is one possible way to implement it - this one compiles today. In other words, this RFC exists to reduce the amount of boilerplate needed to implement the second code block. Here is a real-world example of the second code block in action.

Specific examples of where it could be useful in practice

Use Case: Breaking proxies into their own crates

Suppose I have:

Server in the server crate, runs a zbus::interface
Client in the client crate, talks to the server over a zbus::proxy defined in the client crate.

The naive way to do this is for client to declare its own zbus::proxy, basically copy-pasting the function sigs from the server codebase. This is very brittle however. Duplicating proxies like this means the compiler won't tell us if the server's interface changed and the proxy is now out of date.

Instead, it would be better if we had:

server's dbus interface and proxy in server-dbus crate, with the proxy generated directly from the zbus::interface macro.
Server implementation in server crate, has a dependency on server-dbus
Client has a dependency on server-dbus

This would allow us to deduplicate our proxy stub, and start getting compilation errors when the interface changes.

Note however that because server depends on server-dbus, to avoid cyclic depencencies in crates server-dbus cannot depend on any types or functionality from server.

A Naive solution

To avoid the cyclic dependency, we might naively try to pull some fuctionality in server back into server-dbus.

Unfortunately this approach is often doomed for failure. Best case, we end up with an awkward bifurcation of the codebase, with business logic and types weirdly split across server and server-dbus. Compile times of all consumers of the server-dbus crate are bloated, and you have more dependencies than you need (especially a problem when the implementation of server-dbus is now platform specific or requires physical hardware, and you are not running your tests on that machine!). Worst case scenario, you find that its actually totally impossible to split it up, because the implementation of the interface itself depends on types in client.

Dependency injection to the rescue!

A better solution than trying to bring the implementation logic into server-dbus is to make the interface support dependency injection! server-dbus defines a trait for the dbus interface, applies zbus::interface(..., proxy(...)) to it, and gets out a concrete struct that is generic over the trait. Then server simply instantiates this struct with its own concrete implementation. Ta Da!

Use Case: Mocking services

Suppose that I am testing the business logic of a dbus client. It uses a proxy to some other service. If the service's interface was defined on a trait, we could mock the external service by providing an artificial implementation of the service. We can then spin up the service using a crate like dbus_launch.

Any time we make RPC calls to the mocked service, the test can actually fully inject whatever behavior it wants. For example, instead of doing some complicated logic that requires hardware in the loop (for example, talking to a microcontroller), the mocked implementation can just return some fake responses, as well as collect some info to be used at the end of the test to check the number of times the function was called (or other nice things that you would expect from a mock).

The benefit of using the approach in this RFC is that this gives you the ability to mock on the client side for free. You don't need to make the client more complex by having a trait or Box that acts as the client's way of talking to the external service. Instead, you get to use the existing dbus proxy, and talk over the real dbus protocol. But instead of talking to the real server, you talk to a server running different business logic.

In other words, its more like the wiremock crate (where you create a real http server that has mocked business logic) than it is the mockall crate (where the client would talk to a Box<dyn T> instead of performing HTTP calls).

Mocking the zbus interface by passing in a custom implementation during testing.

Yeah, this does sound very useful. User can easily impl two different versions of the same interface in separate mod already though, but a common interface would ensure that they've the exact same D-Bus interface.

The naive way to do this is for client to declare its own zbus::proxy, basically copy-pasting the function sigs from the server codebase. This is very brittle however. Duplicating proxies like this means the compiler won't tell us if the server's interface changed and the proxy is now out of date.

But we already support generation of proxies from interface directly (see https://github.com/dbus2/zbus/issues/236) and I just added support for changing the visibility of the generated proxy types separately, so you can have your interface types private while proxies could be public. This already allows you to have them both in the same cargo project, while splitting the lib and bin parts. In fact, we're already doing this at work. So this use case, I'm not sold on.

Suppose that I am testing the business logic of a dbus client. It uses a proxy to some other service. If the service's interface was defined on a trait, we could mock the external service by providing an artificial implementation of the service.

If you create the proxy through interface, you can have your interface be the mock impl you require, while the proxy can be used against both the real impl. and your mocked impl. i-e this is already very much possible. Also, since the actual service is external to you, having a trait would not help you since that service is unlikely to use this trait.

In any case, I think this would be useful but it may require API break. Fortunately we're in the middle of one so if you can implement this soon, we can look into getting it into zbus 5.0. Otherwise, we'll have to wait for 6.0 (and there is no plan for it yet).

But we already support generation of proxies from interface directly (see https://github.com/dbus2/zbus/issues/236) and I just added support for changing the visibility of the generated proxy types separately, so you can have your interface types private while proxies could be public.

Yep, I used zbus::interface(..., proxy(...)) in the example code :) The reason why the naive solution cannot use that approach is that if you attempt to use the proxy argument to the zbus::interface proc macro, the proxy and interfaces get placed into the same module. Its not possible to have the proxy get generated into a different crate. So visibility is not the issue, the issue is that there is no way to separate the zbus code into its own, minimal crate. Without that, a client cannot consume the api without also having a dependency on the crate that contains the main business logic.

If client has a dependency on the crate that contains the main business logic, you get all the same issues I described in the RFC:

Compile times of all consumers of the crate are bloated, and you have more dependencies than you need (especially a problem when the implementation of server-dbus is now platform specific or requires physical hardware, and you are not running your tests on that machine!).

At work, this issue becomes quite serious. Many of our dbus interfaces talk to microcontrollers over CAN, manipulate EFI variables, etc. This means that the server crate can only compile to linux (making macos devs unable to run tests) and often we cannot actually run it in containers for testing, since it needs physical hardware present. This makes testing impossible unless we:

Have a solution that supports mocking
Doesn't pull in the platform-specific crates

If the proxy lives in the same crate as all the business logic, we proliferate this non-portability to all consumers of the proxy, since they depend on the non-portable crate. It makes writing cross platform tests or tests that don't need physical hardware impossible without doing the naive solution above (or doing the second code block in the RFC).

This is why in the naive solution, you would be forced to crate a separate zbus::proxy and not use the proxy argument to zbus::interface. Because you don't want your client to depend on the entire business logic and all platform specific dependencies of the main server crate.

A better solution is what is described in the RFC: something that allows you to have a server-dbus crate that has both the interface and the proxy, uses zbus::interface(..., proxy(...)), and no other dependencies or business logic. So that the server and the client can both consume it.

If you create the proxy through interface, you can have your interface be the mock impl you require, while the proxy can be used against both the real impl. and your mocked impl. i-e this is already very much possible.

I'm not sure I understand. Can you give an example where the client code doesn't need to change, but the server code is mocked?

Also, since the actual service is external to you, having a trait would not help you since that service is unlikely to use this trait.

I also didn't understand this 😅

I think this would be useful but it may require API break

Sorry I'm unfamiliar with how zbus is implemented so maybe I'm missing something - can you elaborate why it would cause an API break? My assumption was that because the first codeblock I listed doesn't compile today (the macro throws an error if it sees a trait), there shouldn't be any existing code that would break.

But we already support generation of proxies from interface directly (see #236) and I just added support for changing the visibility of the generated proxy types separately, so you can have your interface types private while proxies could be public.

Yep, I used zbus::interface(..., proxy(...)) in the example code :) The reason why the naive solution cannot use that approach is that if you attempt to use the proxy argument to the zbus::interface proc macro, the proxy and interfaces get placed into the same module. Its not possible to have the proxy get generated into a different crate.

Ah ok, gotcha. Sorry, I didn't read your RFC too thoroughly. :facepalm:

If client has a dependency on the crate that contains the main business logic, you get all the same issues I described in the RFC:

Compile times of all consumers of the crate are bloated, and you have more dependencies than you need (especially a problem when the implementation of server-dbus is now platform specific or requires physical hardware, and you are not running your tests on that machine!).

Can't you resolve this through cargo features and/or platform-specific impls? I understand this does not solve the issue of being able to share the interface but just a workaround till there is a way to do that as well (probably through traits, like you are proposing)?

At work, this issue becomes quite serious. Many of our dbus interfaces talk to microcontrollers over CAN, manipulate EFI variables, etc.

Interestingly, we're doing the same at work, except that we're not communicating over CAN but rather USB.

This means that the server crate can only compile to linux (making macos devs unable to run tests) and often we cannot actually run it in containers for testing, since it needs physical hardware present.

Interesting. Is it cause CAN support in Mac is not present/good? :thinking: I'm asking cause my colleague (and team lead) using Mac exclusively and he's able to run the D-Bus services on his Mac, talking to the MCs just fine.

This is why in the naive solution, you would be forced to crate a separate zbus::proxy and not use the proxy argument to zbus::interface. Because you don't want your client to depend on the entire business logic and all platform specific dependencies of the main server crate.

Sure but don't forget also the naive solution through #cfg and cargo features I mentioned above here.

A better solution is what is described in the RFC: something that allows you to have a server-dbus crate that has both the interface and the proxy, uses zbus::interface(..., proxy(...)), and no other dependencies or business logic. So that the server and the client can both consume it.

I agree. Although I recall I did attempt to make this happen before and it turned out to be harder than I thought. I only wish I remembered what problems I faced exactly before giving up and going for the interface-generated proxy approach.

If you create the proxy through interface, you can have your interface be the mock impl you require, while the proxy can be used against both the real impl. and your mocked impl. i-e this is already very much possible.

I'm not sure I understand. Can you give an example where the client code doesn't need to change, but the server code is mocked?

I was referring to the case of external (as in existing/out of your control). For those you don't need to create the actual interface but only a mock one for testing your proxy. Examples would be various systemd interfaces.

Also, since the actual service is external to you, having a trait would not help you since that service is unlikely to use this trait.

I also didn't understand this 😅

Same here. You can't share your trait with systemd's service impl. :)

I think this would be useful but it may require API break

Sorry I'm unfamiliar with how zbus is implemented so maybe I'm missing something - can you elaborate why it would cause an API break?

I'm not sure this will require an API break, it's more a feeling based on how every new feature I have implemented (or have tried to implement) in the macros, ended up requiring a break to be implemented in a nice way.

My assumption was that because the first codeblock I listed doesn't compile today (the macro throws an error if it sees a trait), there shouldn't be any existing code that would break.

You're correct that the goal/feature itself is very unlikely to break anything itself but implementing it in the macro, might require us to do so, depending on what attributes we may need to add to the macro. Let's see. :)

Interesting. Is it cause CAN support in Mac is not present/good?

Yep, on linux we use the socketcan api and that doesn't exist on MacOS (to my knowledge)

Can't you resolve this through cargo features

Maybe, but I think refactoring my project's codebase to use code block #2 is a better approach.

I was referring to the case of external (as in existing/out of your control)

Most of the cases im interested in are first-party code talking to first party code over dbus, since these are the apis that are most brittle and change most frequently.

dbus2 / zbus