REALTIMEKIT Realtime Policy and Watchdog Daemon
GIT: https://github.com/heftig/rtkit
NOTES: RealtimeKit is a D-Bus system service that changes the scheduling policy of user processes/threads to SCHED_RR (i.e. realtime scheduling mode) on request. It is intended to be used as a secure mechanism to allow real-time scheduling to be used by normal user processes.
RealtimeKit enforces strict policies when handing out
real-time security to user threads:
* Only clients with RLIMIT_RTTIME set will get RT scheduling
* RT scheduling will only be handed out to processes with
SCHED_RESET_ON_FORK set to guarantee that the scheduling
settings cannot 'leak' to child processes, thus making sure
that 'RT fork bombs' cannot be used to bypass RLIMIT_RTTIME
and take the system down.
* Limits are enforced on all user controllable resources, only
a maximum number of users, processes, threads can request RT
scheduling at the same time.
* Only a limited number of threads may be made RT in a
specific time frame.
* Client authorization is verified with PolicyKit
RealtimeKit can also be used to hand outh high priority
scheduling (i.e. negative nice level) to user processes.
In addition to this a-priori policy enforcement, RealtimeKit
also provides a-posteriori policy enforcement, i.e. it
includes a canary-based watchdog that automatically demotes
all real-time threads to SCHED_OTHER should the system
overload despite the logic pointed out above. For more
information regarding canary-based RT watchdogs, see the
Acknowledgments section below.
In its duty to manage real-time scheduling *securely*
RealtimeKit runs as unpriviliged user, and uses capabalities,
resource limits and chroot() to minimize its security impact.
RealtimeKit probably has little use in embedded or server use
cases, use RLIMIT_RTPRIO there instead.WHY: If processes that have real-time scheduling privileges enter a busy loop they can freeze the entire the system. To make sure such run-away processes cannot do this RLIMIT_RTTIME has been introduced. Being a per-process limit it is however easily cirumvented by combining a fork bomb with a busy loop.
RealtimeKit hands out RT scheduling to specific threads that
ask for it -- but only to those and due to SCHED_RESET_ON_FORK
it can be sure that this won't 'leak'.
In contrast to RLIMIT_RTPRIO the RealtimeKit logic makes sure
that only a certain number of threads can be made realtime,
per user, per process and per time interval.CLIENTS: To be able to make use of realtime scheduling clients may request so with a small D-Bus interface that is accessible on the interface org.freedesktop.RealtimeKit1 as object /org/freedesktop/RealtimeKit1 on the service org.freedesktop.RealtimeKit1:
void MakeThreadRealtime(u64 thread_id, u32 priority);
void MakeThreadHighPriority(u64 thread_id, s32 priority);
The thread IDs need to be passed as kernel tids as returned by
gettid(), not a pthread_t! (Please
note that gettid() is not available in glibc, you need to
implement that manually using syscall(). Consult the reference
client implementation for details.)
It is possible to promote thread in process to realtime/high
priority from another process, that will make the DBUS call,
using:
void MakeThreadRealtimeWithPID(u64 process, u64 thread_id, u32 priority);
void MakeThreadHighPriorityWithPID(u64 process, u64 thread_id, s32 priority);
where process is the PID of the process that has thread thread_id.
A BSD-licensed reference implementation of the client is
available in rtkit.[ch] as part of the package. You may copy
this into your sources if you wish. However given how simple
the D-Bus interface is you might choose to implement your own
client implementation.
It is advisable to try acquiring realtime scheduling with
sched_setsheduler() first, so that systems where RLIMIT_RTPRIO
is set can be supported.
Here's an example using the reference implementation. Replace
this:
<snip>
struct sched_param p;
memset(&p, 0, sizeof(p));
p.sched_priority = 3;
sched_setscheduler(0, SCHED_RR|SCHED_RESET_ON_FORK, &p);
</snip>
by this:
<snip>
struct sched_param p;
memset(&p, 0, sizeof(p));
p.sched_priority = 3;
if (sched_setscheduler(0, SCHED_RR|SCHED_RESET_ON_FORK, &p) < 0
&& errno == EPERM)
rtkit_make_realtime(system_bus, 0, p.sched_priority);
</snip>
But of course add more appropriate error checking! Also,
falling back to plain SCHED_RR when SCHED_RESET_ON_FORK causes
EINVAL migt be advisable).DAEMON:
The daemon is automatically started on first use via D-Bus
system bus activation.
Currently the daemon does not read on any configuration file,
however it can be configured with command line parameters. You
can edit
/usr/share/dbus-1/system-services/org.freedesktop.RealtimeKit1.service
to set those.
Run
/usr/libexec/rtkit-daemon --help
to get a quick overview on the supported parameters and their
defaults. Many of them should be obvious in their meaning. For
the remaining ones see below:
--max-realtime-priority= may be used to specify the maximum
realtime priority a client can acquire through
RealtimeKit. Please note that this value must be smaller than
the value passed to --our-realtime-priority=.
--our-realtime-priority= may be used to specify the realtime
priority of the daemon itself. Please note that this priority
is only used for a very short time while processing a client
request. Normally the daemon will not be running with a
realtime scheduling policy. The real-time priorities handed
out to the user must be lower than this value. (see above).
--min-nice-level= may be used to specify the minimum nice
level a client can acquire through RealtimeKit.
--our-nice-level= may be used to specify the nice level the
the daemon itself uses most of the time (except when
processing requests, see above). It is probably a good idea to
set this to a small positive value, to make sure that if the
system is overloaded already handing out further RT scheduling
will be delayed a bit.
--rttime-usec-max= may be used to control which RLIMIT_RTTIME
value clients need to have chosen at minimum before they may
acquire RT scheduling through RealtimeKit.
--users-max= specifies how many users may acquire RT
scheduling at the same time for one or multiple of their
processes.
--processes-per-user-max= specifies how many processes per
user may acquire RT scheduling at the same time.
--threads-per-user-max= specifies how many threads per user
may acquire RT scheduling at the same time. Of course this
value should be set higher than --process-per-user-max=.
--actions-burst-sec= may be used to influence the rate
limiting logic in RealtimeKit. The daemon will only pass
realtime scheduling privileges to a maximum number of threads
within this timeframe (see below).
--actions-per-burst-max= may be used to influence the rate
limiting logic in RealtimeKit. The daemon will only pass
realtime scheduling privileges to this number of threads
within the time frame configured via
--actions-burst-sec=. When this limit is reached clients need
to wait until that time passes before requesting RT scheduling
again.
--canary-cheep-msec= may be used to control how often the
canary thread shall cheep.
--canary-watchdog-msec= may be used to control how quickly the
watchdog thread expects to receive a cheep from the canary
thread. This value must be chosen larger than
--canary-cheep-msec=. If the former is set 10s and the latter
to 7s, then the canary thread can trigger and deliver the
cheep with a maximum latency of 3s.ACKNOWLEDGMENTS: The canary watchdog logic is inspired by previous work of Vernon Mauery, Florian Schmidt, Kjetil Matheussen:
http://rt.wiki.kernel.org/index.php/RT_WatchdogLICENSE: GPLv3+ for the daemon BSD for the client reference implementation
AUTHOR: Lennart Poettering
REQUIREMENTS: Linux kernel >= 2.6.31 D-Bus PolicyKit >= 0.92
OPTIONAL DEPENDENCIES: libsystemd - to let rtkit talk to systemd using the sd-daemon API