[RFE] Enable full watchdog support

tbertels commented 3 years ago

Describe the bug

The kernel watchdog is currently enabled on most/all devices (for example on RPi4). However, with the default configuration (/etc/systemd/system.conf), it doesn't reboot the device even on kernel error.

Beyond basic watchdog support (rebooting on kernel panic or freeze), the watchdog service could be installed and configured to restart Kodi when it doesn't respond or crashes. See also /etc/watchdog.conf file

So there seems to be two parts:

[ ] Basic watchdog support (reboot on system crash/freeze)
[ ] Full watchdog support (restart Kodi if crashed/frozen)

Basic support seems like an obvious starting point, but full support could be a strong selling point for LibreElec vs Raspberry Pi OS, which can't have automatic Kodi restart on freeze/crash out of the box. At this time however, only Raspberry Pi OS allows advanced users to install and configure the watchdog.

To Reproduce

Steps to reproduce the behavior:

touch watchdog
- Result: no reboot
sync; sleep 2; sync; echo c > /proc/sysrq-trigger
- Result: kernel panic, no reboot

Informations

LE Version: 9.2.6
Hardware Platform: RPi4

neo-neo1 commented 2 years ago

A configurable watchdog in GUI would be optimal but id settle for anything at this point.

shtrom commented 2 years ago

I have an idea, but it doesn't seem to work so far. I'll leave it here for completeness.

Waiting for upstream support, we could enable the systemd watchdog support in the autostart.sh, by writing an override using RuntimeWatchdogSec in /run/systemd/system.conf.d.

So I have this in /storage/.config/autostart.sh

#!/bin/sh

SYSCONFD=/run/systemd/system.conf.d/

mkdir -p "${SYSCONFD}"

if [ -e /dev/watchdog ]; then
    cat > "${SYSCONFD}/00-watchdog.conf" << EOF
[Manager]
RuntimeWatchdogSec=10
EOF
fi

systemctl daemon-reload

But don't rejoice too early. It doesn't seem to work... If I misspell RuntimeWatchdogSec, I can see systemd complaining about it in the output of journalctl, so I know the file is read.

However, the value still seems to be stuck at 0 after the reload

# systemctl show | grep -i watchdog
RuntimeWatchdogUSec=0
ShutdownWatchdogUSec=10min
ServiceWatchdogs=yes

(It's USec instead of Sec, but I assumed systemd does the conversion internally, and USec is rejected from the config).

shtrom commented 2 years ago

Ah!

Based on https://serverfault.com/questions/805745/reboot-or-systemctl-daemon-reload-for-changes-to-etc-systemd-system-conf: daemon-reload only reloads the units, daemon-reexec reloads the daemon and re-applies all the config. And indeed it does.

Here is an /storage/.config/autostart.sh script that sets-up systemd to use the native RPi4 watchdog:

#!/bin/sh

SYSCONFD=/run/systemd/system.conf.d/

mkdir -p "${SYSCONFD}"

if [ -e /dev/watchdog ]; then
    cat > "${SYSCONFD}/00-watchdog.conf" << EOF
[Manager]
RuntimeWatchdogSec=10
EOF
fi

systemctl daemon-reexec

After running the script, journalctl confirms the watchdog is set-up for use.

# journalctl | grep -i watchdog
Feb 20 21:57:29 libreelec systemd[1]: Hardware watchdog 'Broadcom BCM2835 Watchdog timer', version 0
Feb 20 21:57:29 libreelec systemd[1]: Set hardware watchdog to 10s.
# systemctl show  | grep -i watchdog
RuntimeWatchdogUSec=10s
ShutdownWatchdogUSec=10min
ServiceWatchdogs=yes

sochotnicky commented 2 years ago

I actually have a small SoC computer running on Intel platform and the default x86_64 kernel doesn't seem to have basic watchdog drivers enabled/installed as far as I can see: https://github.com/LibreELEC/LibreELEC.tv/blob/master/projects/Generic/linux/linux.x86_64.conf#L3360

Perhaps that could be added? For example the Intel TCO driver https://www.kernelconfig.io/config_itco_wdt is likely to be present in almost any recent intel-based computer.

heitbaum commented 2 years ago

Hi @sochotnicky - looks like only x86_64 is missing the kernel configuration. But as others have posted. A handler will also need to be put in place. PRs welcome.

$ git grep "CONFIG_WATCHDOG[ =]" projects/
projects/Allwinner/linux/linux.aarch64.conf:CONFIG_WATCHDOG=y
projects/Allwinner/linux/linux.arm.conf:CONFIG_WATCHDOG=y
projects/Amlogic/linux/linux.aarch64.conf:CONFIG_WATCHDOG=y
projects/Generic/linux/linux.x86_64.conf:# CONFIG_WATCHDOG is not set
projects/NXP/devices/iMX6/linux/linux.arm.conf:CONFIG_WATCHDOG=y
projects/NXP/devices/iMX8/linux/linux.aarch64.conf:CONFIG_WATCHDOG=y
projects/Qualcomm/devices/Dragonboard/linux/linux.aarch64.conf:CONFIG_WATCHDOG=y
projects/RPi/devices/RPi/linux/linux.arm.conf:CONFIG_WATCHDOG=y
projects/RPi/devices/RPi2/linux/linux.arm.conf:CONFIG_WATCHDOG=y
projects/RPi/devices/RPi4/linux/linux.aarch64.conf:CONFIG_WATCHDOG=y
projects/Rockchip/devices/RK3288/linux/default/linux.arm.conf:CONFIG_WATCHDOG=y
projects/Rockchip/devices/RK3328/linux/default/linux.aarch64.conf:CONFIG_WATCHDOG=y
projects/Rockchip/devices/RK3399/linux/default/linux.aarch64.conf:CONFIG_WATCHDOG=y
projects/Samsung/linux/linux.arm.conf:CONFIG_WATCHDOG=y

trollkarlen commented 2 weeks ago

Is this enabled or not in later LE ? Can find much information about this.

heitbaum commented 2 weeks ago

This is still not enabled in the Generic kernel .config

Whilst making the single line change in the below is simple

commit 2f7ccefbac4cafe3dd3288c96c384b4881e2bce4
Author: Rudi Heitbaum <rudi@heitbaum.com>
Date:   Wed Oct 16 11:06:45 2024 +0000

    linux (Generic): set CONFIG_WATCHDOG=y inline with all other projects

diff --git a/projects/Generic/linux/linux.x86_64.conf b/projects/Generic/linux/linux.x86_64.conf
index 6666a26cf6..bf04b2872f 100644
--- a/projects/Generic/linux/linux.x86_64.conf
+++ b/projects/Generic/linux/linux.x86_64.conf
@@ -3679,7 +3679,7 @@ CONFIG_INTEL_SOC_DTS_THERMAL=m
 CONFIG_INTEL_HFI_THERMAL=y
 # end of Intel thermal drivers

-# CONFIG_WATCHDOG is not set
+CONFIG_WATCHDOG=y
 CONFIG_SSB_POSSIBLE=y
 CONFIG_SSB=y
 CONFIG_SSB_SPROM=y

What of the below should be included in the image? Then there still is the “improvement” of using the watchdog.

3690,3760d3682
< # CONFIG_WATCHDOG_CORE is not set
< # CONFIG_WATCHDOG_NOWAYOUT is not set
< CONFIG_WATCHDOG_HANDLE_BOOT_ENABLED=y
< CONFIG_WATCHDOG_OPEN_TIMEOUT=0
< # CONFIG_WATCHDOG_SYSFS is not set
< # CONFIG_WATCHDOG_HRTIMER_PRETIMEOUT is not set
< 
< #
< # Watchdog Pretimeout Governors
< #
< 
< #
< # Watchdog Device Drivers
< #
< # CONFIG_SOFT_WATCHDOG is not set
< # CONFIG_LENOVO_SE10_WDT is not set
< # CONFIG_WDAT_WDT is not set
< # CONFIG_XILINX_WATCHDOG is not set
< # CONFIG_ZIIRAVE_WATCHDOG is not set
< # CONFIG_CADENCE_WATCHDOG is not set
< # CONFIG_DW_WATCHDOG is not set
< # CONFIG_MAX63XX_WATCHDOG is not set
< # CONFIG_ACQUIRE_WDT is not set
< # CONFIG_ADVANTECH_WDT is not set
< # CONFIG_ADVANTECH_EC_WDT is not set
< # CONFIG_ALIM1535_WDT is not set
< # CONFIG_ALIM7101_WDT is not set
< # CONFIG_EBC_C384_WDT is not set
< # CONFIG_EXAR_WDT is not set
< # CONFIG_F71808E_WDT is not set
< # CONFIG_SP5100_TCO is not set
< # CONFIG_SBC_FITPC2_WATCHDOG is not set
< # CONFIG_EUROTECH_WDT is not set
< # CONFIG_IB700_WDT is not set
< # CONFIG_IBMASR is not set
< # CONFIG_WAFER_WDT is not set
< # CONFIG_I6300ESB_WDT is not set
< # CONFIG_IE6XX_WDT is not set
< # CONFIG_ITCO_WDT is not set
< # CONFIG_IT8712F_WDT is not set
< # CONFIG_IT87_WDT is not set
< # CONFIG_HP_WATCHDOG is not set
< # CONFIG_SC1200_WDT is not set
< # CONFIG_PC87413_WDT is not set
< # CONFIG_NV_TCO is not set
< # CONFIG_60XX_WDT is not set
< # CONFIG_CPU5_WDT is not set
< # CONFIG_SMSC_SCH311X_WDT is not set
< # CONFIG_SMSC37B787_WDT is not set
< # CONFIG_TQMX86_WDT is not set
< # CONFIG_VIA_WDT is not set
< # CONFIG_W83627HF_WDT is not set
< # CONFIG_W83877F_WDT is not set
< # CONFIG_W83977F_WDT is not set
< # CONFIG_MACHZ_WDT is not set
< # CONFIG_SBC_EPX_C3_WATCHDOG is not set
< # CONFIG_INTEL_MEI_WDT is not set
< # CONFIG_NI903X_WDT is not set
< # CONFIG_NIC7018_WDT is not set
< # CONFIG_MEN_A21_WDT is not set
< 
< #
< # PCI-based Watchdog Cards
< #
< # CONFIG_PCIPCWATCHDOG is not set
< # CONFIG_WDTPCI is not set
< 
< #
< # USB-based Watchdog Cards
< #
< # CONFIG_USBPCWATCHDOG is not set

heitbaum commented 2 days ago

Closing as information shared, can be reopened if further information is available. Needs help from community to design/build if still appropriate

LibreELEC / LibreELEC.tv