search

kontena / pharos-host-upgrades

Kube DaemonSet for host OS upgrades
Apache License 2.0
41 stars 1 forks source link

Implement --drain support using kubectl #18

Closed SpComb closed 6 years ago

SpComb commented 6 years ago

Bundle kubectl 1.10.3 in the Docker image for kubectl drain ... support.

Implement --drain by running kubectl drain for the node before reboot, setting a pharos-host-upgrades.kontena.io/drain annotation. Uncordon (clear .Spec.Unschedulable) the node after reboot if the pharos-host-upgrades.kontena.io/drain annotation is set.

SpComb commented 6 years ago

Testing with a fake sudo touch /run/reboot-required...

2018/05/29 12:23:04 Reboot required, draining kube node...
2018/05/29 12:23:04 kubectl drain --ignore-daemonsets --force ubuntu-xenial...
2018/05/29 12:24:00 Scheduler is busy, skipping scheduled run
2018/05/29 12:24:37 kubectl drain --ignore-daemonsets --force ubuntu-xenial: node "ubuntu-xenial" cordoned
pod "kube-dns-86f4d74b45-nmx87" evicted
node "ubuntu-xenial" drained
2018/05/29 12:24:37 Kube node drained, rebooting...
2018/05/29 12:24:37 hosts/ubuntu reboot...
2018/05/29 12:24:37 Host is shutting down...
2018/05/29 12:24:37 Leaving kube lock held for reboot, waiting for termination...
2018/05/29 12:25:07 Load config from --config-path=/etc/host-upgrades
2018/05/29 12:25:07 Copying configs to --host-mount=/run/host-upgrades
2018/05/29 12:25:07 hosts/ubuntu boot time: 2018-05-29 12:24:44 +0000 UTC
2018/05/29 12:25:07 hosts/ubuntu probe success: hosts.Info{OperatingSystem:"Ubuntu", OperatingSystemRelease:"16.04.4", Kernel:"Linux", KernelRelease:"4.4.0-127-generic", BootTime:time.Time{wall:0x0, ext:63663193484, loc:(*time.Location)(0x180e020)}}
2018/05/29 12:25:07 Probed host: Ubuntu 16.04.4
2018/05/29 12:25:07 hosts/ubuntu: using host path /run/host-upgrades for output files
2018/05/29 12:25:07 hosts/ubuntu: using copied unattended-upgrades.conf at /run/host-upgrades/unattended-upgrades.conf
2018/05/29 12:25:07 hosts/ubuntu: using generated host-upgrades.sh at /run/host-upgrades/host-upgrades.sh
2018/05/29 12:25:07 Using --kube-namespace=kube-system --kube-daemonset=host-upgrades --kube-node=ubuntu-xenial
2018/05/29 12:25:07 kube/lock kube-system/daemonsets/host-upgrades: get
2018/05/29 12:25:08 kube/lock kube-system/daemonsets/host-upgrades: test pharos-host-upgrades.kontena.io/lock=ubuntu-xenial: acquired
2018/05/29 12:25:08 kube/lock kube-system/daemonsets/host-upgrades: get
2018/05/29 12:25:08 kube/lock kube-system/daemonsets/host-upgrades: release
2018/05/29 12:25:08 kube/lock kube-system/daemonsets/host-upgrades: clear pharos-host-upgrades.kontena.io/lock=ubuntu-xenial
2018/05/29 12:25:08 kube/lock kube-system/daemonsets/host-upgrades: update
2018/05/29 12:25:08 Released kube lock kube-system/daemonsets/host-upgrades (value=ubuntu-xenial)
2018/05/29 12:25:08 Found kube node ubuntu-xenial with existing conditions
2018/05/29 12:25:08 Uncordoned drained kube node ubuntu-xenial (with annotation pharos-host-upgrades.kontena.io/drain)