liske / needrestart

Restart daemons after library updates.
GNU General Public License v2.0
426 stars 67 forks source link

Unknown microcode status #229

Open lrosilio opened 2 years ago

lrosilio commented 2 years ago

Hello, so I have needrestart -p configured on Icinga services, and it occasionally yields the following output on some hosts and exits with an unknown state: Info: UNKN - Kernel: 4.19.0-17-amd64, Microcode: unknown, Services: none, Containers: none, Sessions: none

What does "Unknown" state actually mean and how can I fix this to present either OK or Warning/Critical?

liske commented 2 years ago

Could you please provide the output of needrestart -w -v running a affected host? It looks like something went wrong at the microcode check.

lrosilio commented 2 years ago

@liske

lrosilio@host-1:~$ sudo needrestart -w -v
[main] eval /etc/needrestart/needrestart.conf
[main] needrestart v3.5
[main] running in root mode
[Core] Using UI 'NeedRestart::UI::stdio'...
[main] systemd detected
[ucode] using NeedRestart::uCode::AMD
[ucode] using NeedRestart::uCode::Intel
[uCode/AMD] #0 cpu vendor id mismatch
[uCode/Intel] #0 current revision: 0x0012
+ iucode_tool --scan-system
+ grep -oE [^[:space:]]+$
+ sig=found
+ [ -r /sys/devices/system/cpu/cpu0/microcode/processor_flags ]
+ cat /sys/devices/system/cpu/cpu0/microcode/processor_flags
+ filter=-s found,0x2
+ test -r /etc/needrestart/iucode.sh
+ . /etc/needrestart/iucode.sh
+ type bsdtar
+ IUCODE_TOOL_EXTRA_OPTIONS=
+ test -r /etc/default/intel-microcode
+ test  = no
+ [ -r /usr/share/misc/intel-microcode* ]
+ iucode_tool -l -s found,0x2 --ignore-broken -tb /lib/firmware/intel-ucode
+ grep found
[uCode/Intel] #0 did not get available microcode version

Failed to check for processor microcode upgrades.
liske commented 2 years ago

The output sig=found looks confusing - it seems to happen if iucode_tool is not found by the shell:

$ iucode_tool --scan-system 2>&1|grep -oE '[^[:space:]]+$'
found
$ iucode_tool --scan-system 2>&1
bash: iucode_tool: command not found
$ /usr/sbin/iucode_tool --scan-system 2>&1|grep -oE '[^[:space:]]+$'
0x000906e9

Is iucode_tool available? Is sudo configured to keep the PATH env variable?

lrosilio commented 2 years ago

@liske It looks available:

user@host-1:~$ iucode_tool --scan-system 2>&1|grep -oE '[^[:space:]]+$'
found
user@host-1:~$ sudo iucode_tool --scan-system 2>&1|grep -oE '[^[:space:]]+$'
processor
liske commented 2 years ago

This is weird. Could you please add the output of iucode_tool --scan-system? It should look like:

$ /usr/sbin/iucode_tool --scan-system
/usr/sbin/iucode_tool: system has processor(s) with signature 0x000906e9
lrosilio commented 2 years ago

@liske it says iucode_tool: running on a non-Intel processor

which clears the problem, but now sudo needrestart -w -v looks weird...

EDIT: I have another machine with that problem. iucode_tool wasn't installed, but after package installation it looks like that:

user@host-2:~$ sudo needrestart -w -v
[main] eval /etc/needrestart/needrestart.conf
[main] needrestart v3.5
[main] running in root mode
[Core] Using UI 'NeedRestart::UI::stdio'...
[main] systemd detected
[ucode] using NeedRestart::uCode::AMD
[ucode] using NeedRestart::uCode::Intel
[uCode/AMD] #0 cpu vendor id mismatch
[uCode/Intel] #0 current revision: 0x0012
+ iucode_tool --scan-system
+ grep -oE [^[:space:]]+$
+ sig=0x000306c3
+ [ -r /sys/devices/system/cpu/cpu0/microcode/processor_flags ]
+ cat /sys/devices/system/cpu/cpu0/microcode/processor_flags
+ filter=-s 0x000306c3,0x2
+ test -r /etc/needrestart/iucode.sh
+ . /etc/needrestart/iucode.sh
+ type bsdtar
+ IUCODE_TOOL_EXTRA_OPTIONS=
+ test -r /etc/default/intel-microcode
+ test  = no
+ [ -r /usr/share/misc/intel-microcode* ]
+ iucode_tool -l -s 0x000306c3,0x2 --ignore-broken -tb /lib/firmware/intel-ucode
+ grep 0x000306c3
[uCode/Intel] #0 did not get available microcode version

Failed to check for processor microcode upgrades.
user@host-2:~$ sudo iucode_tool --scan-system
iucode_tool: system has processor(s) with signature 0x000306c3
moschlar commented 2 years ago

We do get that temporarily, too. Especially after reboots.

moschlar commented 2 years ago

Ok, scratch that - it does not relate to the reboots.

Actually it's that Cron uses a different PATH than the interactive shell:

[main] eval /etc/needrestart/needrestart.conf
[main] needrestart v3.4
[main] running in root mode
[Core] Using UI 'NeedRestart::UI::stdio'...
[main] systemd detected
Failed to load NeedRestart::uCode::Intel: [uCode/Intel] iucode-tool not available!
[ucode] no supported processor microcode detection

Failed to check for processor microcode upgrades.
[main] eval /etc/needrestart/needrestart.conf
[main] needrestart v3.4
[main] running in root mode
[Core] Using UI 'NeedRestart::UI::stdio'...
[main] systemd detected
Failed to load NeedRestart::uCode::Intel: [uCode/Intel] iucode-tool not available!
[ucode] no supported processor microcode detection

Failed to check for processor microcode upgrades.
moschlar commented 2 years ago

@liske Although this is only a minor issue and I'd give in to the fact that the problem lies in our usage, maybe, instead of querying for the iucode-tool binary by which, you could (at first) look for the binary in some well-known paths, similar to e.g. this check for a python binary:

    return 1 if($bin =~ m@/usr/(local/)?bin/python@);

https://github.com/liske/needrestart/blob/master/perl/lib/NeedRestart/Interp/Python.pm#L45