sosreport / sos

A unified tool for collecting system logs and other debug information
http://sos.rtfd.org
GNU General Public License v2.0
508 stars 542 forks source link

RFE: run systemd-analyze on various 'targets' #3416

Open smooge opened 11 months ago

smooge commented 11 months ago

I have been working with a customer who had problems with systems 'randomly' not starting NetworkManager but not reporting why it was happening. I went through multiple sosreports but could not see anything logged. The issue was found to be due to cyclic dependencies which had been created by custom jobs, and was easily found by systemd-analyze verify default.target. Once that showed there was a loop in their logic, they were able to fix it.

This may be already in an sosreport, and I wasn't aware of it, or I missed a tool to allow me to do this analysis for them, but I thought it might be useful as going through hundreds of systemd service files missed it by hand.

jcastill commented 11 months ago

Hi @smooge . I don't think we capture this command in sos, and it may be a good addition to the plugins. That said, when I run the command in a "good" system, I get no output at all - that's expected, right?

arif-ali commented 11 months ago

Same on my side, the following command gives me empty output, maybe it only outputs info, when there is an issue?

systemctl list-unit-files | grep target | awk '{print $1}' | xargs -I() systemd-analyze verify {}

But, I also like this, so may be worth having this

$ systemd-analyze
Startup finished in 6.473s (firmware) + 14.904s (loader) + 2min 30.833s (kernel) + 1min 4.439s (userspace) = 3min 56.651s 
graphical.target reached after 1min 4.189s in userspace.
jcastill commented 11 months ago

We capture systemd-analyze already, as well as a couple of other commands. @smooge would be useful to have the output (if any) for default.target or for any target present in the system? I'll start preparing a PR to add this command.

smooge commented 11 months ago

It should give no output on verify if everything is working well. However it can start showing things like:

basic.target: Found ordering cycle on sockets.target/start
basic.target: Found dependency on dbus.socket/start
basic.target: Found dependency on sysinit.target/start
basic.target: Found dependency on systemd-tmpfiles-setup.service/start
basic.target: Found dependency on local-fs.target/start
basic.target: Found dependency on data.mount/start
basic.target: Found dependency on user-inserted.service/start
basic.target: Found dependency on basic.target/start
basic.target: Job sockets.target/start deleted to break ordering cycle starting with basic.target/start
basic.target: Found ordering cycle on dbus-broker.service/start
basic.target: Found dependency on dbus.socket/start
basic.target: Found dependency on sysinit.target/start
basic.target: Found dependency on systemd-tmpfiles-setup.service/start
basic.target: Found dependency on local-fs.target/start
basic.target: Found dependency on data.mount/start
basic.target: Found dependency on user-inserted.service/start
basic.target: Found dependency on basic.target/start
basic.target: Job dbus-broker.service/start deleted to break ordering cycle starting with basic.target/start

user-inserted was meant to cover the name. I will look at a plugin to do this