zbus pulls in async-io

[jeff-hiner]

A lot of us are using tokio instead. There's a feature flag to build zbus with Tokio internals, but we would need to transitively expose this as a feature on secret-service.

[Noah-Kennedy]

The dependency bloat here also is kinda prohibitive for a lot of users. I'd be willing to contribute towards a fix for this.

[complexspaces]

Hi everyone,

Would you be willing to try out this branch with your projects to see if it cleans everything up as expected?

Edit: As pointed out below, this doesn't get rid of most of the dependency bloat.

This feature should be easy to get into the upcoming 3.0 release.

[jeff-hiner]

After some review, this is actually far worse than I thought. The dbus crate was pretty small, but zbus even in "tokio mode" pulls in async-std for a bunch of internal executors. The dependency list winds up still being 130+ crates.

ed: Tried with the branch above, and I'm still getting async-io, async-executor, async-lock and a bunch of other stuff pulled in.

[complexspaces]

@jeff-hiner I raised a similar concern on the zbus issue tracker a while ago, but it was closed as intentional. I don't know if the scenario has changed at all since I asked now that they have a dedicated Tokio feature.

[Noah-Kennedy]

I suspect that the fix here would be to use dbus by default and have the async features pull in zbus.

[Noah-Kennedy]

@complexspaces I would be willing to collaborate and contribute to this.

[complexspaces]

I'm personally against that being the default because of the extra C dependency and the build complexity it punts to downstream users. For my current employer, this is one of the reasons we picked secret-service instead of rolling something of our own.

I would like to see if we could make changes to zbus to support minimizing the dependencies since the attitude towards Tokio has changed some before looking at adding other backends here.

If time permits, I plan to see what a changeset would look like that makes all of the async-* crates optional when using Tokio. My goal is to see how good or bad the changeset would be as part of a potential request for this upstream.

[Noah-Kennedy]

@complexspaces I've checked the code in zbus. It's too tightly coupled and enormous for us to easily do this. I do agree with your point though regarding dbus. I would say that we should have 3 features: async-std, blocking, and tokio. I would probably make blocking be the one in default-features, but I don't think this should be an issue for users since they can disable this default feature.

[Noah-Kennedy]

One thing we should try really hard to avoid here is making the blocking code spawn an executor. This is a pretty bad anti-pattern because it can cause huge issues for anyone downstream who depends on a library that only uses the blocking API. In general, blocking code from libraries should always be capable of being called from an async runtime, although there are exceptions where this isn't practical.

[zeenix]

zbus dev here. 👋

but zbus even in "tokio mode" pulls in async-std

That's not correct. We never depended on async-std. It's only a build dep.

: Tried with the branch above, and I'm still getting async-io, async-executor, async-lock

if you build with default features disabled and tokio enabled, there's no async-io dep. As for the other two, those are small crates that allow us to be runtime-agnostic.

Keeping in mind zbus provides a much friendlier API than dbus-rs and helps you avoid a dep on a notoriously bad C library, I think that tiny bit of "bloat" is well worth it. YMMV of course. 🙂

[complexspaces]

if you build with default features disabled and tokio enabled, there's no async-io dep. As for the other two, those are small crates that allow us to be runtime-agnostic.

It is not necessarily two crates, however @zeenix. IIUC, all of these dependencies are technically unnecessary if a consumer has chosen to use Tokio instead:

async-lock = "2.3.0"
async-broadcast = "0.3.1"
async-channel = "1.6.1"
async-executor = "1.4.1"
async-task = "4.0.3"

I ran a few builds locally making slight tweaks to zbus's Cargo.toml and found these numbers:

Dep Count	"Runtime Flavor"	Tweaked manifest
84	Tokio Integration	No
78	Tokio Integration	Making all the async-* crates optional
97	Default Features	No

That's ~13 extra crates in the dependency tree that a Tokio user wouldn't be using.

One idea, if abstracting runtime details further is unreasonable, is to look at replacing async-lock, async-broadcast, and async-channel with Tokio's sync module. IIRC, everything in that module is compatible with any async runtime.

[zeenix]

if you build with default features disabled and tokio enabled, there's no async-io dep. As for the other two, those are small crates that allow us to be runtime-agnostic.

It is not necessarily two crates,

I was referring to the other 2 you mentioned. :) The main points there were:

1. there are only a handful of those.
2. they are quite small with a few deps (with those deps themselves being super tiny w/ very specific API only and depending in turn on either nothing or again some very specific small dep).

async-runtime doesn't even have any deps at all and is super dupper tiny (< 1kB) so unless you need to run your binary in extremely (resource) constrained environments, is that really something to even think about?

however @zeenix. IIUC, all of these dependencies are technically unnecessary if a consumer has chosen to use Tokio instead:

No necessarily true. async-broadcast for example has a different behavior than the broadcast api in tokio-sync so it's about what behaviour you need/want. I actually created (more like finished) async-broadcast myself specifically for our usecase in zbus. async-channels IIRC also has different behaviour than the corrsponding tokio API.

One idea, if abstracting runtime details further is unreasonable,

Nothing would make me happier than that happening but:

1. It'd be best if these abstractions lived outside zbus as zbus isn't alone in this dilemma. There are folks trying to make that happen as part of std itself but tokio devs don't seem thrilled about it so not sure how far that dream will go. Having said that, if some simple not-so-complicated abstractions in zbus can help you reduce some of the deps, I'd be very welcoming to any such contributions.
2. You can't abstract away all API because of the different behaviour I mentioned above.

Having said all that, some deps should be easy to drop with tokio, e.g async-task and async-executor. I was even planning on look into that myself soon.

I ran a few builds locally making slight tweaks to zbus's Cargo.toml and found these numbers:

I think those numbers don't necessarily mean much on their own. If a dep is only 600 bytes and has 0 deps of its own, should that really be counted? I think what's important is how many bytes do these deps add to the (release) binary size.

IIRC, everything in that module is compatible with any async runtime.

That is correct but keep in mind that you can easily disable that in tokio and use async- yourself as well if the behavior of async- works for you. Just because an app depends on tokio, doesn't mean it has to use tokio for everything. Not saying that you should (I don't know anything about your code), just saying that it's an option to consider.

[complexspaces]

I apologize for the long delay between responses. This fell off my priority list at work.

they are quite small with a few deps (with those deps themselves being super tiny w/ very specific API only and depending in turn on either nothing or again some very specific small dep). I think those numbers don't necessarily mean much on their own. If a dep is only 600 bytes and has 0 deps of its own, should that really be counted? I think what's important is how many bytes do these deps add to the (release) binary size.

In general, this increases the bus trust factor for a lot of consumers. It doesn't necessarily matter how small a crate is but how many extra crates.io publishers it adds to the list of those who can affect someones lockfile. Its very hard to manually vet every change made in every upstream dependency you use with a reasonably large project.

More broadly on the topic of using many more crates, this just seems to be an area that will disagree on, which is perfectly fine.

The one thing that I would like to additionally bring up is @Noah-Kennedy's point that currently trying to use this crate in a blocking fashion will spawn an entire async runtime under the hood silently. I don't believe this is entirely zbus's fault/responsibility though. I think that it may have better ways to be handled, but is ultimately a responsibility of this crate.

Given the above, @Noah-Kennedy, I would be willing to review and merge changes here to separate out the blocking/non-blocking APIs using dbus and zbus respectively before we publish 3.0. Is this something you're still interested in?

[Noah-Kennedy]

Yes. I'm going on vacation next week. I can get back to this then.

[complexspaces]

Great, thank you. Enjoy your vacation 😄.

[zeenix]

how many extra crates.io publishers it adds to the list of those who can affect someones lockfile. Its very hard to manually vet every change made in every upstream dependency you use with a reasonably large project.

I could see your point if we were talking about a lot of crates here. The point you're making is quite vague and could be easily applied to all other deps of zbus (or any crate really).

More broadly on the topic of using many more crates, this just seems to be an area that will disagree on, which is perfectly fine.

Sure thing.

The one thing that I would like to additionally bring up is @Noah-Kennedy's point that currently trying to use this crate in a blocking fashion will spawn an entire async runtime under the hood silently. I don't believe this is entirely zbus's fault/responsibility though. I think that it may have better ways to be handled, but is ultimately a responsibility of this crate.

• If you use the blocking wrappers, they have to run a runtime. In case of tokio, zbus already caches the tokio runtime instead of creating one for each blocking call.
• the blocking wrappers are just for convenience and mostly for simple/quick code writing. I'd suggest using the async API.

[zeenix]

Anyway, I think I've provided enough opinions and straightened out the facts here already and ultimately your crate, your call. :) So I'll unsubscribe now. Best of luck!

[complexspaces]

Thank you for the suggestions and feedback again this time around, its greatly appreciated.

[zeenix]

Hi, just fyi I have been using quite a bit of my spare time lately to add better (more fuller) integration with tokio. There were a lot of problems on the way but I believe I should be done soon and allow you to drop some more of the deps (at least async-executor and async-task, maybe also async-lock in the near future).

However, this requires an API break so I'll likely have to do a 3.0 release. I hope that's not an issue.

[zeenix]

However, this requires an API break so I'll likely have to do a 3.0 release. I hope that's not an issue.

Just to clarify, the API break is most theoretical. It's unlikely to require any changes from users other than updating the required version in Cargo.toml line, especially folks building with tokio feature already.

[zeenix]

As promised. If folks could give a test, that'd be great!

[zeenix]

As promised. If folks could give a test, that'd be great!

I merged that and already made the async-lock optional as well in the same merge-request. Yesterday and today, I spent many hours making async-channel optional. It turned out to be a lot more challenging than I thought (mainly cause async-channel is MPMC and `tokio's impl is MPSC) but I managed to get it working I believe. :partying_face:

So the only one remaining is async-broadcast but as I explained before, we rely on the behavior of async-broadcast and it's very different from that of tokio::sync::broadcast API.

I spent a significant amount of my spare time on making this happen so if one of you folks could just help me test it out, that would be great. :pray:

[zeenix]

I spent a significant amount of my spare time on making this happen so if one of you folks could just help me test it out, that would be great.

By "you folks" I mean the folks here who have been asking for this. :) secret-service seems to currently not proxy the tokio feature of zbus and disabling default features of it.

[complexspaces]

Hey again @zeenix. It's on my TODO list this weekend to point with secret-service's master branch to zbus's head and play around. I didn't have the time last week while it was ongoing due to work, but I am, personally at least, very grateful that you've put the time and effort into this. I'll let you know how this goes 🎉.

Re: not proxying the tokio feature, that's currently in the tokio-support branch, which I'll likely end up merging in as well.

[Noah-Kennedy]

I'll try this out next week.

[zeenix]

Hey again @zeenix. It's on my TODO list this weekend to point with secret-service's master branch to zbus's head and play around.

:+1:

I am, personally at least, very grateful that you've put the time and effort into this.

You're welcome. :)

Re: not proxying the tokio feature, that's currently in the tokio-support branch, which I'll likely end up merging in as well.

Oh cool, that looks great.

I'll try this out next week.

:+1: :pray:

[complexspaces]

Overall, version 3 of zbus does exactly what you said :) For this crate, updating was almost entirely a drop-in process. The only thing that I noticed was different is that a test failed to compile with zbus 3 but worked with zbus 2, due to an apparent lifetime issue. I left some more details in #50 about the other changes.

[zeenix]

Overall, version 3 of zbus does exactly what you said :) For this crate, updating was almost entirely a drop-in process.

Great, thanks for testing it.

a test failed to compile with zbus 3 but worked with zbus 2, due to an apparent lifetime issue

Interesting, what was the issue? https://github.com/hwchen/secret-service-rs/pull/50/commits/078af62351b609147a6dcaf7a9298beab923b758 is not caused by zbus, is it? :thinking:

[complexspaces]

AFAICT, it seems to be. Nothing else changed on that branch or in my local configuration. The CI failure is unrelated to this issue. I can reproduce by doing the following:

1. Checkout https://github.com/hwchen/secret-service-rs/commit/f7af766e69598dc69e1857637e06f1e3dc648bb6
2. Run cargo test and observe that the crate builds fine.
3. Checkout https://github.com/hwchen/secret-service-rs/commit/2ce3dd9f397e044efe7a557258642c89ec75efaf
4. Observe that one test now fails to compile with lifetime issues.
5. Checkout the head of the tokio-support branch and observe it compiles again.

[zeenix]

AFAICT, it seems to be. Nothing else changed on that branch or in my local configuration. The CI failure is unrelated to this issue. I can reproduce by doing the following:

Ah yes, this is a known issue.

Do I understand correctly that you currently only have blocking API? tokio integration isn't super relevant then, if so.

[complexspaces]

That is correct, yeah. It is on my TODO (probably this upcoming weekend) to convert this crate's API to be properly async on the master branch instead of using the blocking wrappers. I don't think its impossible that there would be a blocking module in this crate later on though. Its useful for very simple use cases and experimentation.

[zeenix]

That is correct, yeah.

Interesting. I don't understand then why folks were so outraged by lack of tighter tokio integration then. :thinking: Blocking API is for non-async code. If you call blocking API from async context, you'll get even a panic with tokio (and a hang in other runtimes).

Because of this limitation of async machinery combined with the tighter tokio integration, you also would get a panic from tokio if you drop a blocking wrapper type (e.g proxy) in an async context.

It is on my TODO (probably this upcoming weekend) to convert this crate's API to be properly async on the master branch instead of using the blocking wrappers.

Cool, no pressure from my side. I'm not one of the users of your API (yet). :)

I don't think its impossible that there would be a blocking module in this crate later on though. Its useful for very simple use cases and experimentation.

For sure, hence the reason zbus also provides blocking wrappers. I'd suggest keeping the blocking wrappers under a blocking mode though.

[zeenix]

Interesting. I don't understand then why folks were so outraged by lack of tighter tokio integration then. thinking Blocking API is for non-async code. If you call blocking API from async context, you'll get even a panic with tokio (and a hang in other runtimes).

However, since you'll soon be enabling async API, this becomes a real problem so I'm still glad I fixed it in zbus.

[Sytten]

Any progress on using zbus 3?

[zeenix]

Any progress on using zbus 3?

Was released more than a week ago.

[zeenix]

2 weeks ago actually: https://crates.io/crates/zbus/versions

[complexspaces]

Just to check in here, this is blocked on either myself or another contributor adding a proper async fn interface to the crate which can correctly utilize the async interface zbus 2/3 provides. Beyond that, updating the dependency should be no work at all, like what was done in #50.

I have been on leave and (intentionally) not doing much of anything related to code the last few weeks. I can, and still plan to, work on this update once I'm back, but that will be around Sep 15th. I apologize that this is taking a while, but I needed a break from looking at an IDE for a while 😅.

[Sytten]

Do you want to replace the sync api with an async one or provide both? Considering this will impact keyring crate we might want to bring him in the discussion.

The zbus sync seems to spawn a tokio executor so thats not great, but kicking down the can isn't always the nicest solution.

I am willing to put the work in if the interface is agreed upon.

[zeenix]

The zbus sync seems to spawn a tokio executor so thats not great,

You mean a runtime and "spawn" is very misleading here. The runtime is created once (on the first blocking call) and used to run the underlying async call (the future) to completion and it's a single-threaded runtime so it doesn't even spawn any threads. This is a common way to add blocking API on top of async one.

but kicking down the can isn't always the nicest solution.

I'm not sure what that means.

[Sytten]

Yeah I meant a runtime. Yes I am aware this is the general practice, but when you already have one it is usually a bit wasteful. Even further when you use the blocking crate since it will never be the same thread that handles the request thus you create a new runtime every time.

Well if keyring only wants to have a sync api they will create this async runtime (instead of zbus), so you are just kicking the problem down the line. Usually it's ok since I am a big async proponent but async do "infect" all pieces of downstream code.

[zeenix]

Yeah I meant a runtime. Yes I am aware this is the general practice, but when you already have one it is usually a bit wasteful.

Why would you already have one? blocking code is not meant to be used in async context (and must not be) and tokio isn't very relevant to blocking code.

What can happen is you using multiple crates that all do this but if that happens, it means you really should consider making your code to be async. Blocking wrappers are mainly targeted for simple cases, where a runtime behind the scene doesn't matter. Still, assuming all these deps use a single-threaded runtime and they cache it (like zbus does), there is no performance issue (at least not a significant enough to worry about).

Even further when you use the blocking crate since it will never be the same thread that handles the request thus you create a new runtime every time.

zbus is not using that crate so don't think this information is relevant here.

Well if keyring only wants to have a sync api they will create this async runtime (instead of zbus), so you are just kicking the problem down the line.

While I agree with you that it'd be great for secret-service to provide blocking wrappers, I really don't think it's a must. If keyring wants to use blocking code for I/O, that's their decision of course but a single-threaded runtime completing the futures under the hood, is hardly a big price to pay.

[Sytten]

Sorry that my comments really seem to confuse you. My code is already async but if I use keyring or this crate I have to use a thread pool since they don't offer an async interface and thus it will spawn runtimes uselessly. I can assure you I do know what I am doing here.

Anyway lets not continue the discussion, I just want to know from the author if they want to keep the sync interface or not for the transition period.

[zeenix]

Sorry that my comments really seem to confuse you.

Well if you use incorrect terminology ("spawning" for example), it's not only me who will get confused. I fear that people will read your comments and think that zbus does something bad/wrong and end up deciding not to use it. As someone who has put a huge amount of (mostly my own) time into the project, it's hard for me to sit idly by and not respond to get the record straight.

I can assure you I do know what I am doing here.

I apologize if I gave the impression that I want to contradict that fact.

Anyway lets not continue the discussion, I just want to know from the author if they want to keep the sync interface or not for the transition period.

:+1:

[complexspaces]

@jeff-hiner @Sytten Can I ask you to try the tokio-support branch again? #50 has been updated to use zbus 3 and I'd appreciate someone else smoke testing it for me before I merge it and a make a new secret-service release.

[Sytten]

@complexspaces Observations:

• Readme example is missing a dyn in the box, making rust grumpy
• Default example works fine, still the annoying misleading error (Zbus(Io(Os { code: 2, kind: NotFound, message: "No such file or directory" }))) on headless linux when the keyring is not started/unlocked but this can be improved later
• Not convinced an async new is really the best word for the SS creation, connect or similar might be better

It looks really good, great job :) Can't wait for it to ship so we can start refactoring keyring-rs to async

[zeenix]

• Default example works fine, still the annoying misleading error (Zbus(Io(Os { code: 2, kind: NotFound, message: "No such file or directory" }))) on headless linux when the keyring is not started/unlocked but this can be improved later

Without a D-Bus session, there is nothing to connect to, so yeah that's what you'll get. I agree that better error could be shown. This is just the debug representation of the error, that's just a wrapper around the source std::io::Error coming from tokio/std on trying to open the in-existent socket file for us.

It looks really good, great job :)

:+1:

[Sytten]

I am wondering if it could be possible to reduce the zbus dependencies footprint. It does pull a lot of stuff right now. Not a big deal but could be nice for future version!

[zeenix]

I am wondering if it could be possible to reduce the zbus dependencies footprint. It does pull a lot of stuff right now. Not a big deal but could be nice for future version!

This issue was first created for this and after a lot of hard work, I managed to drop 5-6 deps in favor of appropriate tokio api where possible. If it still feels too many deps, I'd be more than happy to hear ideas on how to further reduce them.

[complexspaces]

Thanks for the feedback, @Sytten 😃. I fixed the missing dyn in the README and renamed all the new methods to connect since I agree its better reflecting whats really happening when you call them.

I'm going to see if I can find any time to try updating to zbus 3 in my codebase at work this week, as a final test run before moving forward with merging.

[complexspaces]

The missing session bus error can be improved for sure. I'll try and open a PR for that to give it a dedicated error variant once the zbus 3 update merges.