See resources
for example kube manifests.
apt-get install unattended-upgrades
Disable apt-periodic
unattended-upgrades
:
sed -i 's/APT::Periodic::Unattended-Upgrade .*/APT::Periodic::Unattended-Upgrade "0";/' /etc/apt/apt.conf.d/20auto-upgrades
Alternatively, you can completely disable apt-periodic
, as the upgrade process will also run apt-get update
before the unattended-upgrades
:
systemctl stop apt-daily.timer apt-daily-upgrade.timer
systemctl disable apt-daily.timer apt-daily-upgrade.timer
The set of allowed origins for package upgrades can either be configured via the system /etc/apt/apt.conf.d/50unattended-upgrades
or using a ConfigMap unattended-upgrades.conf
, but the default settings are reasonable.
yum install yum-cron
Ensure that the yum-cron
service is stopped and disabled, as this will interfere with pharos-host-upgrades
:
systemctl stop yum-cron.service
systemctl disable yum-cron.service
The default yum-cron.conf
random_sleep = 360
should also be disabled, either via the default system /etc/yum/yum-cron.conf
file, or using a ConfigMap yum-cron.conf
. The default yum-cron.conf
will only download updates, and requires apply_updates = yes
to actually upgrade the host.
When configured to run as a kube DaemonSet pod (using KUBE_*
envs), the following kube API integrations can be used:
The host upgrades will only run while holding a lock on the kube daemonset, ensuring that only one host upgrades at a time. This lock is also held during a reboot, and released once the pod restarts.
The lock is implemented as a pharos-host-upgrades.kontena.io/lock
annotation on the DaemonSet.
If configured with --reboot --drain
, the kube node will be drained before rebooting, marking the node as unschedulable and evicting pods to move them to other nodes for the duration of the reboot.
The node will be uncordoned once the host-upgrades
pod is restarted, but only if the pharos-host-upgrades.kontena.io/drain
annotation was previously set as a result a drain + reboot triggered by the host-upgrades
pod. If the pod is restarted while the node was otherwise drained, it will not be uncordoned.
The kube node .Status.Conditions
will be updated based on the result of the host upgrades:
HostUpgrades
The HostUpgrades
condition describes the state of the upgrade command itself.
The condition will be True
if the upgrade was run and the node is now up to date, with a message describing any packages upgraded during the last run.
The condition will be False
if the host is not up to date. This will happen in the RebootRequired
case, where the host requires a reboot to finish applying the upgrades.
In case of the upgrade failing, the condition will be Unknown
, with a message describing the error.
HostUpgradesReboot
The HostUpgradesReboot
condition will be True
if the host requires a reboot to finish applying upgrades, and False
otherwise.
The Go binary can be run natively, outside of Docker or Kube:
sudo $GOPATH/bin/pharos-host-upgrades
This mode only requires systemd
DBUS access. Kube locks and configs are not supported.
Example docker run
options:
docker run --rm --name host-upgrades -v $PWD/config:/etc/host-upgrades -v /run/host-upgrades:/run/host-upgrades -v /var/run/dbus:/var/run/dbus -v /run/log/journal:/run/log/journal --privileged kontena/pharos-host-upgrades
See the example kube resources:
kubectl apply -f ./resources
The DaemonSet configures the KUBE_*
envs, enabling the kubernets integrations.
Usage of pharos-host-upgrades:
-alsologtostderr
log to standard error as well as files
-config-path string
Path to configmap dir (default "/etc/host-upgrades")
-drain
Drain kube node before reboot, uncordon after reboot
-host-mount string
Path to shared mount with host. Must be under /run to reset when rebooting! (default "/run/host-upgrades")
-kube-daemonset string
Name of kube DaemonSet (KUBE_DAEMONSET) (default "host-upgrades")
-kube-namespace string
Name of kube Namespace (KUBE_NAMESPACE) (default "kube-system")
-kube-node string
Name of kube Node (KUBE_NODE) (default "ubuntu-xenial")
-log_backtrace_at value
when logging hits line file:N, emit a stack trace
-log_dir string
If non-empty, write log files in this directory
-logtostderr
log to standard error instead of files
-reboot
Reboot if required
-reboot-timeout duration
Wait for system to shutdown when rebooting (default 5m0s)
-schedule string
Scheduled upgrade (cron syntax)
-stderrthreshold value
logs at or above this threshold go to stderr
-v value
log level for V logs
-vmodule value
comma-separated list of pattern=N settings for file-filtered logging
command terminated with exit code 2
--schedule
See https://godoc.org/github.com/robfig/cron#hdr-CRON_Expression_Format (no seconds field).
Standard crontab with five fields (Minutes, Hours, Day of month, Month, Day of week).
Examples:
15 5 * * *
- every day at 05:151 0 * * SUN
- every sunday at 01:00@daily
at midnight--reboot
--reboot-timeout=...
Reboot the host after upgrades, if required.
--drain
Drain the kube node before rebooting, and uncordon once restarted.
The kube DaemonSet also supports an optional ConfigMap with configuration files for the host OS package upgrade tools. The ConfigMap should be mounted at --config-path=/etc/host-upgrades
, and the --host-mount=/run/host-upgrades
path should be bind-mounted from the host.
unattended-upgrades.conf
Refer to the host /etc/apt/apt.conf.d/50unattended-upgrades
config file shipped by the unattended-upgrades
package.
Note overriding the Unattended-Upgrade::Allowed-Origins
list requires the use of a #clear
directive, to have the list items in the config override those in the default host config, rather than merging the two lists together. See the sample unattended-upgrades.conf
for an example:
// Override system /etc/apt/apt.conf.d/50unattended-upgrades
#clear "Unattended-Upgrade::Allowed-Origins";
// Automatically upgrade packages from these (origin:archive) pairs
Unattended-Upgrade::Allowed-Origins {
...
};
yum-cron.conf
Refer to the host /etc/yum/yum-cron.conf
config file shipped by the yum-cron
package.
Note that the default random_sleep
value will delay upgrades across the entire cluster, and should be disabled. See the sample yum-cron.conf
for an example:
[commands]
random_sleep = 0
Using the vagrant machines:
$ vagrant up ubuntu
$ vagrant up centos-7
Note that the centos/7
& debian/stretch64
boxes do not support shared folders... vagrant falls back to rsync, so you must vagrant rsync centos-7
or vagrant rsync debian-9
before building the image after any edits.
$ mkdir .kube
$ sudo cat /etc/kubernetes/admin.conf > .kube/config
/vagrant $ docker build -t kontena/pharos-host-upgrades:dev .
/vagrant $ docker run --rm --name host-upgrades --privileged \
-v $PWD/config:/etc/host-upgrades \
-v /run/host-upgrades:/run/host-upgrades \
-v /var/run/dbus:/var/run/dbus \
-v /run/log/journal:/run/log/journal \
kontena/pharos-host-upgrades:dev