Open aboubeta opened 5 years ago
~$ kubectl get pods NAME READY STATUS RESTARTS AGE host-upgrades-72m7q 0/1 CrashLoopBackOff 657 2d17h host-upgrades-9vvdv 0/1 CrashLoopBackOff 616 2d17h host-upgrades-g9d4m 0/1 CrashLoopBackOff 655 2d17h host-upgrades-wcbgp 0/1 CrashLoopBackOff 652 2d16h ~$ kubeclt logs host-upgrades-9vvdv 2019/06/13 07:49:00 Acquiring kube lock... 2019/06/13 07:49:00 kube/lock kube-system/daemonsets/host-upgrades: get 2019/06/13 07:49:00 kube/lock kube-system/daemonsets/host-upgrades: wait 2019/06/13 07:49:00 kube/lock kube-system/daemonsets/host-upgrades: test pharos-host-upgrades.kontena.io/lock=: free 2019/06/13 07:49:00 kube/lock kube-system/daemonsets/host-upgrades: acquire 2019/06/13 07:49:00 kube/lock kube-system/daemonsets/host-upgrades: set pharos-host-upgrades.kontena.io/lock=aks-nodepool1-14334624-1 2019/06/13 07:49:00 kube/lock kube-system/daemonsets/host-upgrades: update 2019/06/13 07:49:00 Running host upgrades... 2019/06/13 07:49:00 hosts/ubuntu upgrade... 2019/06/13 07:49:00 systemd/exec host-upgrades.service: systemd.ExecOptions{Unit:"host-upgrades.service", Cmd:[]string{"/bin/sh", "-x", "/run/host-upgrades/host-upgrades.sh"}, Env:[]string{"HOST_PATH=/run/host-upgrades", "APT_CONFIG=/run/host-upgrades/apt.conf"}} 2019/06/13 07:49:00 systemd/exec host-upgrades.service: reset 2019/06/13 07:49:00 systemd/exec host-upgrades.service: start []dbus.Property{dbus.Property{Name:"Type", Value:dbus.Variant{sig:dbus.Signature{str:"s"}, value:"oneshot"}}, dbus.Property{Name:"Environment", Value:dbus.Variant{sig:dbus.Signature{str:"as"}, value:[]string{"HOST_PATH=/run/host-upgrades", "APT_CONFIG=/run/host-upgrades/apt.conf"}}}, dbus.Property{Name:"ExecStart", Value:dbus.Variant{sig:dbus.Signature{str:"a(sasb)"}, value:[]dbus.execStart{dbus.execStart{Path:"/bin/sh", Args:[]string{"/bin/sh", "-x", "/run/host-upgrades/host-upgrades.sh"}, UncleanIsFailure:false}}}}} 2019/06/13 07:49:00 systemd/exec host-upgrades.service: wait... 2019/06/13 07:49:02 systemd/exec host-upgrades.service: done, status=failed 2019/06/13 07:49:02 systemd/journal: read journal... 2019/06/13 07:49:02 Update kube node aks-nodepool1-14334624-1 condition for status={false 0001-01-01 00:00:00 +0000 UTC } with error: systemd/exec host-upgrades.service failed: [/bin/sh -x /run/host-upgrades/host-upgrades.sh] 2019/06/13 07:49:02 Upgrade failed, releasing kube lock... (systemd/exec host-upgrades.service failed: [/bin/sh -x /run/host-upgrades/host-upgrades.sh] ) 2019/06/13 07:49:02 Releasing kube lock... 2019/06/13 07:49:02 kube/lock kube-system/daemonsets/host-upgrades: get 2019/06/13 07:49:02 kube/lock kube-system/daemonsets/host-upgrades: release 2019/06/13 07:49:02 kube/lock kube-system/daemonsets/host-upgrades: clear pharos-host-upgrades.kontena.io/lock=aks-nodepool1-14334624-1 2019/06/13 07:49:02 kube/lock kube-system/daemonsets/host-upgrades: update 2019/06/13 07:49:02 systemd/exec host-upgrades.service failed: [/bin/sh -x /run/host-upgrades/host-upgrades.sh] On the container: + set -ue + apt-get update Get:1 http://security-cdn.debian.org/debian-security stretch/updates InRelease [94.3 kB] Ign:2 http://cdn-fastly.deb.debian.org/debian stretch InRelease Get:3 http://cdn-fastly.deb.debian.org/debian stretch-updates InRelease [91.0 kB] Get:4 http://cdn-fastly.deb.debian.org/debian stretch Release [118 kB] Get:5 http://security-cdn.debian.org/debian-security stretch/updates/main amd64 Packages [495 kB] Get:6 http://cdn-fastly.deb.debian.org/debian stretch Release.gpg [2434 B] Get:7 http://cdn-fastly.deb.debian.org/debian stretch-updates/main amd64 Packages [27.2 kB] Get:8 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 Packages [7082 kB] Fetched 7910 kB in 2s (3534 kB/s) Reading package lists... Done /run/host-upgrades/host-upgrades.sh: 6: /run/host-upgrades/host-upgrades.sh: HOST_PATH: parameter not set
~$ kubectl get pods
NAME READY STATUS RESTARTS AGE host-upgrades-72m7q 0/1 CrashLoopBackOff 657 2d17h host-upgrades-9vvdv 0/1 CrashLoopBackOff 616 2d17h host-upgrades-g9d4m 0/1 CrashLoopBackOff 655 2d17h host-upgrades-wcbgp 0/1 CrashLoopBackOff 652 2d16h
~$ kubeclt logs host-upgrades-9vvdv
2019/06/13 07:49:00 Acquiring kube lock... 2019/06/13 07:49:00 kube/lock kube-system/daemonsets/host-upgrades: get 2019/06/13 07:49:00 kube/lock kube-system/daemonsets/host-upgrades: wait 2019/06/13 07:49:00 kube/lock kube-system/daemonsets/host-upgrades: test pharos-host-upgrades.kontena.io/lock=: free 2019/06/13 07:49:00 kube/lock kube-system/daemonsets/host-upgrades: acquire 2019/06/13 07:49:00 kube/lock kube-system/daemonsets/host-upgrades: set pharos-host-upgrades.kontena.io/lock=aks-nodepool1-14334624-1 2019/06/13 07:49:00 kube/lock kube-system/daemonsets/host-upgrades: update 2019/06/13 07:49:00 Running host upgrades... 2019/06/13 07:49:00 hosts/ubuntu upgrade... 2019/06/13 07:49:00 systemd/exec host-upgrades.service: systemd.ExecOptions{Unit:"host-upgrades.service", Cmd:[]string{"/bin/sh", "-x", "/run/host-upgrades/host-upgrades.sh"}, Env:[]string{"HOST_PATH=/run/host-upgrades", "APT_CONFIG=/run/host-upgrades/apt.conf"}} 2019/06/13 07:49:00 systemd/exec host-upgrades.service: reset 2019/06/13 07:49:00 systemd/exec host-upgrades.service: start []dbus.Property{dbus.Property{Name:"Type", Value:dbus.Variant{sig:dbus.Signature{str:"s"}, value:"oneshot"}}, dbus.Property{Name:"Environment", Value:dbus.Variant{sig:dbus.Signature{str:"as"}, value:[]string{"HOST_PATH=/run/host-upgrades", "APT_CONFIG=/run/host-upgrades/apt.conf"}}}, dbus.Property{Name:"ExecStart", Value:dbus.Variant{sig:dbus.Signature{str:"a(sasb)"}, value:[]dbus.execStart{dbus.execStart{Path:"/bin/sh", Args:[]string{"/bin/sh", "-x", "/run/host-upgrades/host-upgrades.sh"}, UncleanIsFailure:false}}}}} 2019/06/13 07:49:00 systemd/exec host-upgrades.service: wait... 2019/06/13 07:49:02 systemd/exec host-upgrades.service: done, status=failed 2019/06/13 07:49:02 systemd/journal: read journal... 2019/06/13 07:49:02 Update kube node aks-nodepool1-14334624-1 condition for status={false 0001-01-01 00:00:00 +0000 UTC } with error: systemd/exec host-upgrades.service failed: [/bin/sh -x /run/host-upgrades/host-upgrades.sh] 2019/06/13 07:49:02 Upgrade failed, releasing kube lock... (systemd/exec host-upgrades.service failed: [/bin/sh -x /run/host-upgrades/host-upgrades.sh] ) 2019/06/13 07:49:02 Releasing kube lock... 2019/06/13 07:49:02 kube/lock kube-system/daemonsets/host-upgrades: get 2019/06/13 07:49:02 kube/lock kube-system/daemonsets/host-upgrades: release 2019/06/13 07:49:02 kube/lock kube-system/daemonsets/host-upgrades: clear pharos-host-upgrades.kontena.io/lock=aks-nodepool1-14334624-1 2019/06/13 07:49:02 kube/lock kube-system/daemonsets/host-upgrades: update 2019/06/13 07:49:02 systemd/exec host-upgrades.service failed: [/bin/sh -x /run/host-upgrades/host-upgrades.sh]
On the container:
+ set -ue + apt-get update Get:1 http://security-cdn.debian.org/debian-security stretch/updates InRelease [94.3 kB] Ign:2 http://cdn-fastly.deb.debian.org/debian stretch InRelease Get:3 http://cdn-fastly.deb.debian.org/debian stretch-updates InRelease [91.0 kB] Get:4 http://cdn-fastly.deb.debian.org/debian stretch Release [118 kB] Get:5 http://security-cdn.debian.org/debian-security stretch/updates/main amd64 Packages [495 kB] Get:6 http://cdn-fastly.deb.debian.org/debian stretch Release.gpg [2434 B] Get:7 http://cdn-fastly.deb.debian.org/debian stretch-updates/main amd64 Packages [27.2 kB] Get:8 http://cdn-fastly.deb.debian.org/debian stretch/main amd64 Packages [7082 kB] Fetched 7910 kB in 2s (3534 kB/s) Reading package lists... Done /run/host-upgrades/host-upgrades.sh: 6: /run/host-upgrades/host-upgrades.sh: HOST_PATH: parameter not set