search

bus1 / dbus-broker

Linux D-Bus Message Broker
https://github.com/bus1/dbus-broker/wiki
Apache License 2.0
677 stars 79 forks source link

enable dbus-broker reexecute #310

Open Geass-LL opened 1 year ago

Geass-LL commented 1 year ago

This add a new feature to dbus-broker. With this patch, we can reexecute dbus-broker by running dbus-broker-launch --reexec.

The main idea behind this commit is:

  1. Serialize: when broker gets the Reexecute message, it will serialize the main information (socket fd, id, uid, pid, match rule, request name, sasl) to the anonymous file,
  2. Reexecute: launcher and broker reexecute.
  3. AddListener: when launcher has reexecuted successfully, it will send AddListener to broker.
  4. Recover: broker recovers all peers when processing AddListener message, and it uses the socket fd saved in anonymous file to create new peers, and recover id/uid/pid/...
  5. Ready: launcher will broadcast Ready signal, when the dbus-broker-launch --reexec gets this signal, the reexecuting process is over.

I'll split this commit to several small commits after your first review. Please take a look, thank a lot.

teg commented 1 year ago

Thank you for working on this! It is our most requested feature and will surely be greatly appreciated by many.

Some questions before I look at the code in detail (I am sure I'll think of more things in the coming days):

teg commented 1 year ago

Regarding the draining, it might be worth while to introduce a Freeze and Thaw call and only allow reexec when the broker is frozen. I think this might simplify the proposed reexec logic as an added bonus (as we would still allow calls on the launcher API while frozen).

Freeze would finish any partial dispatch and stop further dispatching. We still need to serialize messages that are queued but not dispatched though. An open question is what to do about misbehaving peers who have stopped reading/worrying from/to their sockets.

Geass-LL commented 1 year ago
  • How do you version the serialized data, and what kinds of upgrade paths do you think we should support? My suggestion: save a major.minor version for the serialized state and increment the minor number when we make a change that can not be rolled back from and change the major number when we make changes that we cant roll forward to (nor back from). Then fail to reexec if we find an incompatible version number.

I didn't think about the version. Currently, I only serialize the peer's fd/id/uid/pid/requested name/match rule/sasl and broker's max_id to know which id we should distribute to the newly connected peer after reexecuting. The serialized string is like this:

max_ids=1234
peer_str=12;5600;91750;996;org.freedesktop.PolicyKit1;[{type='signal',sender='org.freedesktop.DBus',interface='org.freedesktop.DBus',member='NameOwnerChanged',path='/org/freedesktop/DBus'}];[{1}{1}{0}]

These should be enough to recover a peer. But yes, we may have some big changes in the future version, major.minor version sounds a good idea. After rpm -Uvh dbus-broker.aarch64, dbus-broker-launch --reexec can send its version to the running old broker, then broker compares the received version with its own, if the two versions are incompatible, the broker refuses to reexecute and replies an error.

  • How do you think we should deal with in-flight messages? If I understand correctly they are now currently dropped? I think we can't be lossy, so need to come up with something. Ideally we would serialize everything, but we could also possibly consider some sort of drain operation.

I don't close the peer's fd during reexecuting, and write all messages out before reexecuting. After reexecuting, I reepoll these fd and process it. I tested this by the following testcase:

  1. Create a server which can accept two 'x' numbers, and reply the multiply result as 'x';
  2. Create a server which sends method_call to the server in a busy loop, and check the reply;
  3. Reexecuting dbus-broker in a busy loop background for 1000 times;

No message was lost.

Currently, this commit doesn't support create a new connection during reexecuting.

bluca commented 1 year ago

This is really cool - one suggestion, maybe use systemd's FD store? That should allow to avoid needing to manually handle things, and a restart should "just" work?

Geass-LL commented 1 year ago

This is really cool - one suggestion, maybe use systemd's FD store? That should allow to avoid needing to manually handle things, and a restart should "just" work?

Thanks for your suggestion, I'll look into it.