search

liske / needrestart

Restart daemons after library updates.
GNU General Public License v2.0
427 stars 68 forks source link

Special treatment for dbus daemon #44

Closed Vladimir-csp closed 7 years ago

Vladimir-csp commented 8 years ago

Needrestart offers the possibility to restart dbus daemon among other services, but choosing to do so results in long hang, because systemd is left paralyzed when dbus exits. While it is stated by dbus developers that dbus should not be restarted, but rather system reboot is required, it is possible to restard dbus with no ill effects. It is just a matter of correct ordering and systemd reexec. I personally use this sequence on my machine:

# bring down DM, it would crash anyway
systemctl stop display-manager
# restard dbus
systemctl restart dbus
# restart systemd
systemctl daemon-reexec
# restart daemons that directly depend on dbus
systemctl restart systemd-logind systemd-journald NetworkManager
# start DM
systemctl start display-manager

This way nothing seems to hang or break, the process is almost instant.

liske commented 8 years ago

There is already some logic to handle some process special (i.e. Sys-V init). It should be possible to add something to needrestart to handle dbus special, too.

liske commented 7 years ago

I've added a restart.d directory allowing to use scripts for restarting selected services. I've adopted your sequence which is shipped as /etc/needrestart/restart.d/dbus.service. Maybe you could give it a try... thanks!

Vladimir-csp commented 7 years ago

I've tested it, it works. Two caveats though: I once got NetworkManager in stopped state after running needrestart via ssh (just once, did not reproduce since). If running from graphical environment, needrestart process is killed with the environment mid-process. So it would be useful to consider forking the actual restart job into background untying it from any killable terminal.

mtodorov3-69 commented 5 years ago

Vladimir, this makes a lot of sense, restarts ought to be in the order of dependencies and not in the order shown in the blue text box ...

mtodorov3-69 commented 5 years ago

Is there a possibility of live-patching daemons now when Ubuntu is offering canonical-livepatch for the kernel?

Vladimir-csp commented 5 years ago

I've created a more automated script that handles dbus dependencies without any hardcoding: https://github.com/Vladimir-csp/bashlifehacks/blob/master/restart-dbus

But I haven't looked into how to integrate it properly into needrestart to make it unkillable. Currently it works via at hack

Vladimir-csp commented 5 years ago

I've updated the script, it should work when directly executed now. Will test it with needrestart when appropriate upgrades come by.

mtodorov3-69 commented 5 years ago

I hope it works for Ubuntu 18.04.1 LTS which I have at home. It is Debian based, right? I wouldn't think of running it on our production server without thoroughly testing it first ...

mtodorov3-69 commented 5 years ago

I've created a more automated script that handles dbus dependencies without any hardcoding: https://github.com/Vladimir-csp/bashlifehacks/blob/master/restart-dbus

But I haven't looked into how to integrate it properly into needrestart to make it unkillable. Currently it works via at hack

Hi Vladimir, my colleague pointed it out how system logins slow down (on Debian "jessie") if dbus service is restarted. Does your script alleviate this problem? Is it just the problem with the order of the dependencies? Thanks.

Vladimir-csp commented 5 years ago

As far as I remember, login slowdown is due to some logind-related components or logind itself loosing dbus connection and acting weird. In general, this is the point of the script: to restart everything that depends on dbus.

mtodorov3-69 commented 5 years ago

I hope it works for Ubuntu 18.04.1 LTS which I have at home. It is Debian based, right? I wouldn't think of running it on our production server without thoroughly testing it first ...

As far as I remember, login slowdown is due to some logind-related components or logind itself loosing dbus connection and acting weird. In general, this is the point of the script: to restart everything that depends on dbus.

Thus far, Ubuntu 18.04.1 LTS requires a restart almost every week. Your script could help to a great deal if tested properly! :-)

Vladimir-csp commented 5 years ago

I use Debian testing, it has dbus-affecting upgrades regularly. But not in recent couple of days, so I could only test it standalone and it worked fine that way. Have you tried it on Ubuntu 18.04.1?

mtodorov3-69 commented 5 years ago

I use Debian testing, it has dbus-affecting upgrades regularly. But not in recent couple of days, so I could only test it standalone and it worked fine that way. Have you tried it on Ubuntu 18.04.1?

No, not yet I haven't. I have to analyse it first to see what you do with the scripting. I am new to systemd, I am not a systemd expert. I don't even know how to write a systemd script. I had written some init scripts though. Systemd seems fairly too complex. Systemd should be small in footprint and stable regarding features and bugs. We don't need a Swiss knife init, but a stable, reliable init process to start daemons. IMNSHO.

Vladimir-csp commented 5 years ago

No, not yet I haven't. I have to analyse it first to see what you do with the scripting. I am new to systemd, I am not a systemd expert. I don't even know how to write a systemd script. I had written some init scripts though.

There is no such thing as "systemd script", it has mostly declarative units. My script just queries information about units and restarts them in needed order.

I've added more comments in the code.

We don't need a Swiss knife init, but a stable, reliable init process to start daemons. IMNSHO.

Agreed.

lkraav commented 5 years ago

As far as I remember, login slowdown is due to some logind-related components or logind itself loosing dbus connection and acting weird. In general, this is the point of the script: to restart everything that depends on dbus.

Correct, systemctl restart systemd-logind should fix it.

mtodorov3-69 commented 5 years ago

As far as I remember, login slowdown is due to some logind-related components or logind itself loosing dbus connection and acting weird. In general, this is the point of the script: to restart everything that depends on dbus.

Correct, systemctl restart systemd-logind should fix it.

I beg you pardon, shouldn't this IMHO be included in the needrestart when restarting dbus if it is that simple? I don't think people would profit from it anyway until upgrading to Debian stretch at least in backports. Needrestart doesn't select dbus restart by default, restart dbus causes problems, but IMHO doing it right would decrease the need for reboots in the long run and increase availability. I know that most people do not install even backports, so next upgrade of the distribution is the first time we can hope for the main corps of admins to adopt the change ... My $0.02 ...

mtodorov3-69 commented 5 years ago

I've added a restart.d directory allowing to use scripts for restarting selected services. I've adopted your sequence which is shipped as /etc/needrestart/restart.d/dbus.service. Maybe you could give it a try... thanks!

Hi, just wanted to ask in which version of needrestart this will be included in the main needrestrart tree in the dpkg package and shipped with the distributions?

As far as I can see, v2.6 in Ubuntu 16.04.6 LTS doesn't have it, v2.11 in Debian stretch and its jessie backport are first to have it and v3.1 in Ubuntu 18.04.2 LTS has a /etc/needrestart/restart.d/dbus.service ...

In a manner of speaking, I have answered my own question, so I only need authoritative information that this is it and that dbus restart had been solved? Or does it still require system reboot on each library change? Many thanks in forward.

Vladimir-csp commented 5 years ago

Needrestart currently has an older iteration of dbus restart.d hook with static restart sequence and no protection against destruction of sessions. My new script can serve as a drop-in replacement (plus set -x condition to conform to needrestart debugging). I'm planning to make a pull request after I test it in acutal dbus-affecting upgrade.

mtodorov3-69 commented 5 years ago

I've created a more automated script that handles dbus dependencies without any hardcoding: https://github.com/Vladimir-csp/bashlifehacks/blob/master/restart-dbus

Hi, Vladimir, You seem to use a ``sort -u'' to eliminate multiple restarts of same daemons. However, it seems to me that you are then losing the order of dependencies as systemctl gave it. Is it important?

Thanks, MG

Vladimir-csp commented 5 years ago

systemctl lists dependencies alphabetically, so it does not matter.

mtodorov3-69 commented 5 years ago

systemctl lists dependencies alphabetically, so it does not matter.

OK then. I still don't know how to put your script into a needrestart restart.d hook. Thus far I have avoided mixing original OS configurations with customizations and modifications. I would probably have to build a replacement dpkg needrestart package, right? How to install your modification in a seamless way.

But I have a bite on it. I will have to see whether needrestart uses C lines and libraries ... Perhaps if Ubuntu people do not provide a backport I might have to produce one of my own?

Vladimir-csp commented 5 years ago

Just replace restart.d/dbus.service with it.

Vladimir-csp commented 5 years ago

I've tested the script with needrestart in actual dbus restart situation. For some reason it took a minute to bring lightdm back, but otherwise no ill effects were observed. I've added a coupe of sleep commands in the sequence just in case. PR is in.

mtodorov3-69 commented 5 years ago

D-Bus just refused to be restarted:

root@efk:~# /etc/needrestart/restart.d/dbus.service
Failed to restart dbus.service: Operation refused, unit dbus.service may be requested by dependency only (it is configured to refuse manual start/stop).
See system logs and 'systemctl status dbus.service' for details.
root@efk:~# systemctl status dbus.service
● dbus.service - D-Bus System Message Bus
   Loaded: loaded (/lib/systemd/system/dbus.service; static; vendor preset: enabled)
   Active: active (running) since Mon 2019-03-11 09:33:45 CET; 22h ago
     Docs: man:dbus-daemon(1)
 Main PID: 691 (dbus-daemon)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/dbus.service
           └─691 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-a

Mar 12 08:02:29 efk dbus-daemon[691]: [system] Reloaded configuration
Mar 12 08:02:29 efk dbus-daemon[691]: [system] Reloaded configuration
Mar 12 08:02:29 efk dbus-daemon[691]: [system] Reloaded configuration
Mar 12 08:02:29 efk dbus-daemon[691]: [system] Reloaded configuration
Mar 12 08:02:29 efk dbus-daemon[691]: [system] Reloaded configuration
Mar 12 08:02:29 efk dbus-daemon[691]: [system] Reloaded configuration
Mar 12 08:02:29 efk dbus-daemon[691]: [system] Reloaded configuration
Mar 12 08:02:35 efk dbus-daemon[691]: [system] Reloaded configuration
Mar 12 08:03:05 efk dbus-daemon[691]: [system] Activating via systemd: service name='org.freedeskt
Mar 12 08:03:05 efk dbus-daemon[691]: [system] Successfully activated service 'org.freedesktop.Pol
root@efk:~#
Vladimir-csp commented 5 years ago

That is something new to me. Could you provide /lib/systemd/system/dbus.service?

mtodorov3-69 commented 5 years ago

That is something new to me. Could you provide /lib/systemd/system/dbus.service?

It is on a Ubuntu Bionic 18.04.02 LTS host. Needrestart is v3.1 . Here it is:

root@efk:~# cat /lib/systemd/system/dbus.service
[Unit]
Description=D-Bus System Message Bus
Documentation=man:dbus-daemon(1)
Requires=dbus.socket
# we don't properly stop D-Bus (see ExecStop=), thus disallow restart
RefuseManualStart=yes

[Service]
ExecStart=/usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
ExecReload=/usr/bin/dbus-send --print-reply --system --type=method_call --dest=org.freedesktop.DBus / org.freedesktop.DBus.ReloadConfig
ExecStop=/bin/true
KillMode=none
OOMScoreAdjust=-900
root@efk:~#
Vladimir-csp commented 5 years ago

Wow, they sure do not want it to be restarted.

You can try to override this by placing a drop-in into /etc/systemd/system/dbus.service.d/local.conf:

[Unit]
RefuseManualStart=no
[Service]
KillMode=control-group
ExecStop=

and hoping that resetting ExecStop would work.

This seems to be an Ubuntu patch from several years ago, dbus 1.10.6-1ubuntu2, 2016-04-01

zerkms commented 5 years ago

What would be the final conclusion for ubuntu then? To blacklist it?

martux69 commented 4 years ago

Are there any news about this ? Behavior still exists under Ubuntu. Is a blacklisting of dbus the right solution ?

KMTMark commented 1 year ago

Also encountered this issue on a machine where I work. Tried popping in the suggested item above but I think that also requires a reload of dbus to pickup, in which case will need to wait until we reboot the machine anyway :D

Vladimir-csp commented 1 year ago

AFAIR, unit drop-in works, only systemctl daemon-reload is needed to apply.