Continuous (push) sync

[9ary]

After using rbw for a couple years, I just noticed for the first time that it doesn't actually sync the database automatically.

It would be nice if to have an option for the agent to keep everything in sync via the push channel, just like the official clients (it doesn't look like the CLI client does this though).

If this isn't possible then it's already possible to run rbw sync on a timer, so I do not believe it'd be necessary to include that here.

[quexten]

If this isn't possible

It's definetly possible, as long as someone has the time to implement it. The web server exposes a websocket endpoint on /hub?access_token=...

More specifically, it uses the signalr protocol. There are keepalive messages every few seconds, and a few messages on folder / collection / cipher change.

The rbw-agent could connect to this websocket endpoint, listen for these messages and do a sync when a change notification comes in.

[doy]

i did go ahead and add background sync based on a timer, because it was pretty easy and i think it is still somewhat useful, but yeah, if anyone would like to add push functionality here, i would be open to pull requests.

[9ary]

@quexten thanks a lot for the technical details! I'm sure that'll save some time digging around in case I or someone else decide to give this a try.

By signalr I suppose you mean the microsoft asp.net thing rather than a typo of signal.org's e2ee protocol?

i did go ahead and add background sync based on a timer, because it was pretty easy and i think it is still somewhat useful

Definitely good enough for now. I also went ahead and added a shortcut to my fzf script to do a sync manually: https://github.com/9ary/dotfiles/commit/314cfe1f601e99784818c22a16fbf13becb2026f.

My point was that it's trivial to set up a systemd timer/cronjob/whatever to do the same, so I didn't feel like it was necessary to add this to rbw itself. A welcome addition regardless, thanks!

[quexten]

By signalr I suppose you mean the microsoft asp.net thing rather than a typo of signal.org's e2ee protocol?

Yep. Though the protocol is fairly simple, and it's likely not needed to implement the entire protocol. To be honest it might be enough to just scan for the one byte that shows the message type, and then always to a full sync (regardless of the push messages contents).

[9ary]

That's definitely the easiest way to go for an initial implementation. Considering how small the cache is (129k for 169 entries in my case), it shouldn't be too much of a concern most of the time, but implementing delta updates properly would be good for cellular and other limited connections.

[9ary]

(this post is turning into my research notes :P)

So there is a signalr implementation in rust, but it's based on actix client, whereas rbw uses reqwest. The whole thing however is only a ~500 line file, so it should work as a reference for a basic implementation.

https://github.com/Igosuki/signalr-rs/blob/master/src/hub/client.rs

Also here's the protocol documentation. https://github.com/dotnet/aspnetcore/blob/main/src/SignalR/docs/specs/HubProtocol.md

And vaultwarden's implementation which might be helpful too. It's extremely simple. https://github.com/dani-garcia/vaultwarden/blob/main/src/api/notifications.rs

The notification service in the JS-based clients: https://github.com/bitwarden/clients/blob/master/libs/common/src/services/notifications.service.ts

https://crates.io/crates/tokio-tungstenite seems like a good option for websockets, unfortunately reqwest doesn't seem to have support at the moment.

[GRbit]

@9ary If you are going to implement this thing, I kindly ask you to implement a timeout sync solution. I use self-hosted vaultwarden under a proxy that doesn't support websockets, and it would be just marvelous to have timeout sync. Firefox browser extension, for instance, doesn't have this functionality, and it's a shame.

And in case you'll change your mind about working on this, please, leave a note here. I'll try to do it by myself.

Thank you for your efforts, this topic is already pretty informative making it a good place for any developer to start work on it.

[9ary]

@GRbit: timer-based sync was already implemented in 5eab3c4b33f2b0b594993a095eae86f88828827d.

I've done a bit of preliminary research on implementing push sync, but at the moment it's a low priority item for me so I won't be working on it yet.

It seems like implementing the push protocol would open the door for other features though, such as authorizing passwordless logins via rbw, and I think that'd be pretty neat.

[GRbit]

@9ary Wow. Good news for me. That means I can just update my rbw version and that's it. Thank you!

[quexten]

@9ary I have started work on this and basic connecting to the service / parsing of the messages is implemented. Turns out a signalr library is not needed, just like in Vaultwarden, tungstenite + rmpv (for messagepack parsing) is enough.

The PR is still far from ready but in case you want to join in on this, let me know so I can add you as a collaborator on the repository.

[9ary]

I have started work on this and basic connecting to the service / parsing of the messages is implemented.

Nice!

Turns out a signalr library is not needed, just like in Vaultwarden, tungstenite + rmpv (for messagepack parsing) is enough.

That was also my conclusion after skimming the spec. 👍

[quexten]

I cleaned up the PR a bit. The PR implements a full sync when an item changes and logout (when de-authorizing all sessions in the web UI), though I believe refining this to do partial sync in another PR should not be too difficult, but is out of scope for my PR.

Regardless of whether the Websocket subscription is available, it will still do the timer based sync, so if the connection does not work (f.e due to not configuring a self-hosted installation for Websockets), it will still work like before.

I marked the PR as ready for review now, but could definitely use someone with more Rust experience to look over it as the proper patterns for handling some things, especially w.r.t async/await in Rust are not familiar to me.

[9ary]

I cleaned up the PR a bit. The PR implements a full sync when an item changes and logout (when de-authorizing all sessions in the web UI), though I believe refining this to do partial sync in another PR should not be too difficult, but is out of scope for my PR.

This is good enough for now. Having the basics laid out is already a great help to implement other features that rely on the hub service.

Regardless of whether the Websocket subscription is available, it will still do the timer based sync, so if the connection does not work (f.e due to not configuring a self-hosted installation for Websockets), it will still work like before.

Sounds good to me. I'm not sure how the official server implementation works, but wouldn't an unconfigured websocket service simply result in it being unreachable?

I marked the PR as ready for review now, but could definitely use someone with more Rust experience to look over it as the proper patterns for handling some things, especially w.r.t async/await in Rust are not familiar to me.

I'm not familiar with async in rust either (only ever used it in python), but I can give your patch a skim.

[doy]

i think #115 should resolve this, but feel free to let me know if any other features are desired!

[fabian-thomas]

@quexten Thank you very much for your adding support for this. This was the only thing that annoyed me for the last two weeks after switching to bitwarden. Also thanks for this amazing project @doy.

I have one question regarding the setup of live sync. It seems like I had to set notifications_url manually, while my setup shouldn't vary too much from the excpected one. I host vaultwarden at vaultwarden.mydomain.de and websocket proxying is enabled and works in the app out of the box. Nevertheless I had to do the following to get rbw to work:

rbw config set notifications_url wss://vaultwarden.mydomain.de/notifications
rbw stop-agent
rbw sync

Is that excpected? (base_url is set to https://vaultwarden.mydomain.de)