kontena / pharos-host-upgrades

Kube DaemonSet for host OS upgrades
Apache License 2.0
41 stars 1 forks source link

DaemonSet on CrashLoopBackOff #37

Open aboubeta opened 5 years ago

aboubeta commented 5 years ago

~$ kubectl get pods

NAME                                    READY   STATUS             RESTARTS   AGE
host-upgrades-72m7q                     0/1     CrashLoopBackOff   657        2d17h
host-upgrades-9vvdv                     0/1     CrashLoopBackOff   616        2d17h
host-upgrades-g9d4m                     0/1     CrashLoopBackOff   655        2d17h
host-upgrades-wcbgp                     0/1     CrashLoopBackOff   652        2d16h

~$ kubeclt logs host-upgrades-9vvdv

2019/06/13 07:49:00 Acquiring kube lock...
2019/06/13 07:49:00 kube/lock kube-system/daemonsets/host-upgrades: get
2019/06/13 07:49:00 kube/lock kube-system/daemonsets/host-upgrades: wait
2019/06/13 07:49:00 kube/lock kube-system/daemonsets/host-upgrades: test pharos-host-upgrades.kontena.io/lock=: free
2019/06/13 07:49:00 kube/lock kube-system/daemonsets/host-upgrades: acquire
2019/06/13 07:49:00 kube/lock kube-system/daemonsets/host-upgrades: set pharos-host-upgrades.kontena.io/lock=aks-nodepool1-14334624-1
2019/06/13 07:49:00 kube/lock kube-system/daemonsets/host-upgrades: update
2019/06/13 07:49:00 Running host upgrades...
2019/06/13 07:49:00 hosts/ubuntu upgrade...
2019/06/13 07:49:00 systemd/exec host-upgrades.service: systemd.ExecOptions{Unit:"host-upgrades.service", Cmd:[]string{"/bin/sh", "-x", "/run/host-upgrades/host-upgrades.sh"}, Env:[]string{"HOST_PATH=/run/host-upgrades", "APT_CONFIG=/run/host-upgrades/apt.conf"}}
2019/06/13 07:49:00 systemd/exec host-upgrades.service: reset
2019/06/13 07:49:00 systemd/exec host-upgrades.service: start []dbus.Property{dbus.Property{Name:"Type", Value:dbus.Variant{sig:dbus.Signature{str:"s"}, value:"oneshot"}}, dbus.Property{Name:"Environment", Value:dbus.Variant{sig:dbus.Signature{str:"as"}, value:[]string{"HOST_PATH=/run/host-upgrades", "APT_CONFIG=/run/host-upgrades/apt.conf"}}}, dbus.Property{Name:"ExecStart", Value:dbus.Variant{sig:dbus.Signature{str:"a(sasb)"}, value:[]dbus.execStart{dbus.execStart{Path:"/bin/sh", Args:[]string{"/bin/sh", "-x", "/run/host-upgrades/host-upgrades.sh"}, UncleanIsFailure:false}}}}}
2019/06/13 07:49:00 systemd/exec host-upgrades.service: wait...
2019/06/13 07:49:02 systemd/exec host-upgrades.service: done, status=failed
2019/06/13 07:49:02 systemd/journal: read journal...
2019/06/13 07:49:02 Update kube node aks-nodepool1-14334624-1 condition for status={false 0001-01-01 00:00:00 +0000 UTC  } with error: systemd/exec host-upgrades.service failed: [/bin/sh -x /run/host-upgrades/host-upgrades.sh]
2019/06/13 07:49:02 Upgrade failed, releasing kube lock... (systemd/exec host-upgrades.service failed: [/bin/sh -x /run/host-upgrades/host-upgrades.sh]
)
2019/06/13 07:49:02 Releasing kube lock...
2019/06/13 07:49:02 kube/lock kube-system/daemonsets/host-upgrades: get
2019/06/13 07:49:02 kube/lock kube-system/daemonsets/host-upgrades: release
2019/06/13 07:49:02 kube/lock kube-system/daemonsets/host-upgrades: clear pharos-host-upgrades.kontena.io/lock=aks-nodepool1-14334624-1
2019/06/13 07:49:02 kube/lock kube-system/daemonsets/host-upgrades: update
2019/06/13 07:49:02 systemd/exec host-upgrades.service failed: [/bin/sh -x /run/host-upgrades/host-upgrades.sh]  

On the container:

+ set -ue
+ apt-get update
Get:1 http://security-cdn.debian.org/debian-security stretch/updates InRelease [94.3 kB]
Ign:2 http://cdn-fastly.deb.debian.org/debian stretch InRelease
Get:3 http://cdn-fastly.deb.debian.org/debian stretch-updates InRelease [91.0 kB]
Get:4 http://cdn-fastly.deb.debian.org/debian stretch Release [118 kB]
Get:5 http://security-cdn.debian.org/debian-security stretch/updates/main amd64 Packages [495 kB]
Get:6 http://cdn-fastly.deb.debian.org/debian stretch Release.gpg [2434 B]
Get:7 http://cdn-fastly.deb.debian.org/debian stretch-updates/main amd64 Packages [27.2 kB]
Get:8 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 Packages [7082 kB]
Fetched 7910 kB in 2s (3534 kB/s)
Reading package lists... Done
/run/host-upgrades/host-upgrades.sh: 6: /run/host-upgrades/host-upgrades.sh: HOST_PATH: parameter not set