search

Glimesh / janus-ftl-plugin

A plugin for the Janus WebRTC gateway to enable relaying of audio/video streams utilizing Mixer's FTL (Faster-Than-Light) protocol.
https://hayden.fyi/posts/2020-08-03-Faster-Than-Light-protocol-engineering-notes.html
GNU Affero General Public License v3.0
46 stars 11 forks source link

⌚🐕 Watchdog support #99

Closed danstiner closed 3 years ago

danstiner commented 3 years ago

Support for optional systemd watchdog integration to automatically restart the service if it is deadlocked. For context see http://0pointer.de/blog/projects/watchdog.html

Compile time support is auto enabled if you have the systemd dev headers available (Ubuntu package libsystemd-dev). Runtime support is auto enabled if you set the WatchdogSec systemd service option, this causes WATCHDOG_USEC to be passed which enables this watchdog integration.

Please be generous with the value for WatchdogSec. Strictly speaking the recommendation is to ping exactly every WATCHDOG_USEC / 2. To keep things simple I instead ping from the metadata reporting thread once per loop it does. This loop can take a variable amount of time as it does a lot of work. It loops through all streams to calculate stats, posts the metadata to Glimesh (including any retries it must do), and then sleeps for FTL_SERVICE_METADATAREPORTINTERVALMS. The benefit of this is if there are any deadlocks the metadata reporting thread basically always locks up so it's a great indicator of service health.

Recommended systemd options:

Type=notify
WatchdogSec=60s
Restart=on-failure

Optional options:

StartLimitInterval=5min
StartLimitBurst=4
StartLimitAction=reboot-force
haydenmc commented 3 years ago

Can you add the awesome context you put in the PR description to a section in the README?

danstiner commented 3 years ago

@haydenmc I can, if it's something we want to advertise supporting :)

haydenmc commented 3 years ago

Don't see why not. If it breaks for folks, they can file issues and we can decide what to do.

Maybe say that it's "experimental" support? :D