search

siderolabs / talos

Talos Linux is a modern Linux distribution built for Kubernetes.
https://www.talos.dev
Mozilla Public License 2.0
6.91k stars 556 forks source link

Ability to force reboot #9625

Open davralin opened 3 weeks ago

davralin commented 3 weeks ago

Bug Report

There should be a way to force a reboot, even if pods/mounts and other things hang.

Description

I had a ceph-rbd-mount that hung, which requires a reboot of the running node. The reboot ended up taking more than 30 mins, ending with me physically rebooting the node instead of it being able to reboot. The API was functional through-out the reboot-stage, so some button or command with similar functionallity as /proc/sysrq-trigger would be nice, the ability to configure a grace period (as mentioned in #4655 ) would also be nice.

Logs

10.0.1.66: user: warning: [2024-11-03T21:51:49.208078219Z]: [talos] reboot via API received. actor id: 7c9e98a2-6447-48bf-adc7-7e64a3130e12
10.0.1.66: user: warning: [2024-11-03T21:51:49.208127219Z]: [talos] reboot sequence: 11 phase(s)
10.0.1.66: user: warning: [2024-11-03T21:51:49.208133219Z]: [talos] phase cleanup (1/11): 1 tasks(s)
10.0.1.66: user: warning: [2024-11-03T21:51:49.208143219Z]: [talos] task stopAllPods (1/1): starting
10.0.1.66: user: warning: [2024-11-03T21:51:49.208148219Z]: [talos] task stopAllPods (1/1): waiting for kubelet lifecycle finalizers
10.0.1.66: user: warning: [2024-11-03T21:51:49.208176219Z]: [talos] task stopAllPods (1/1): shutting down kubelet gracefully
10.0.1.66: kern:    info: [2024-11-03T21:51:50.344500219Z]: EXT4-fs (rbd2): unmounting filesystem cfcc79cc-cd17-4c52-9e48-bc8473f45e83.
10.0.1.66: kern:    info: [2024-11-03T21:51:53.134794219Z]: EXT4-fs (rbd4): unmounting filesystem 5d01b3fd-3247-41ef-a5a4-5774506166be.
10.0.1.66: kern:    info: [2024-11-03T21:51:54.985809219Z]: EXT4-fs (rbd6): unmounting filesystem ebeb3900-8547-4706-a30e-e1a93fee4732.
10.0.1.66: kern:    info: [2024-11-03T21:51:59.919268219Z]: EXT4-fs (rbd8): unmounting filesystem 7381ef6a-7b6a-4a9d-9e6d-9c441f117417.
10.0.1.66: kern:    info: [2024-11-03T21:52:02.453997219Z]: EXT4-fs (rbd5): unmounting filesystem fe8ae95b-ca62-4832-ad1d-ca4acdb6d320.
10.0.1.66: kern: warning: [2024-11-03T21:52:09.230274219Z]: libceph: mon0 (1)10.104.248.190:6789 socket closed (con state OPEN)
10.0.1.66: kern: warning: [2024-11-03T21:52:09.230288219Z]: libceph: mon0 (1)10.104.248.190:6789 socket closed (con state OPEN)
10.0.1.66: kern:    info: [2024-11-03T21:52:09.230340219Z]: libceph: mon0 (1)10.104.248.190:6789 session lost, hunting for new mon
10.0.1.66: kern:    info: [2024-11-03T21:52:09.230342219Z]: libceph: mon0 (1)10.104.248.190:6789 session lost, hunting for new mon
10.0.1.66: kern: warning: [2024-11-03T21:52:09.230349219Z]: libceph: mon0 (1)10.104.248.190:6789 socket closed (con state OPEN)
10.0.1.66: kern:    info: [2024-11-03T21:52:09.230382219Z]: libceph: mon0 (1)10.104.248.190:6789 session lost, hunting for new mon
10.0.1.66: kern:    info: [2024-11-03T21:52:09.233758219Z]: libceph: mon2 (1)10.110.107.1:6789 session established
10.0.1.66: kern:    info: [2024-11-03T21:52:09.233964219Z]: libceph: mon2 (1)10.110.107.1:6789 session established
10.0.1.66: kern:    info: [2024-11-03T21:52:09.235289219Z]: libceph: mon1 (1)10.102.164.207:6789 session established
10.0.1.66: kern: warning: [2024-11-03T21:52:15.221551219Z]: libceph: osd5 (1)10.244.3.101:6801 socket closed (con state OPEN)
10.0.1.66: kern: warning: [2024-11-03T21:52:15.222350219Z]: libceph: osd5 (1)10.244.3.101:6801 socket closed (con state V1_BANNER)
10.0.1.66: kern: warning: [2024-11-03T21:52:15.338686219Z]: libceph: osd4 (1)10.244.3.140:6801 socket closed (con state OPEN)
10.0.1.66: kern: warning: [2024-11-03T21:52:15.474539219Z]: libceph: osd5 (1)10.244.3.101:6801 socket error on write
10.0.1.66: kern: warning: [2024-11-03T21:52:15.974551219Z]: libceph: osd5 (1)10.244.3.101:6801 socket error on write
10.0.1.66: kern: warning: [2024-11-03T21:52:16.998573219Z]: libceph: osd5 (1)10.244.3.101:6801 socket error on write
10.0.1.66: user: warning: [2024-11-03T21:52:19.232657219Z]: [talos] service[kubelet](Stopping): Sending SIGTERM to task kubelet (PID 2755, container kubelet)
10.0.1.66: kern: warning: [2024-11-03T21:52:19.270575219Z]: libceph: osd5 (1)10.244.3.101:6801 socket error on write
10.0.1.66: kern: warning: [2024-11-03T21:52:23.110545219Z]: libceph: osd5 (1)10.244.3.101:6801 socket error on write
10.0.1.66: kern:    info: [2024-11-03T21:52:28.483494219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5066): osd7 weight 0x10000 (in)
10.0.1.66: kern:    info: [2024-11-03T21:52:28.483554219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5081): osd3 weight 0x3333
10.0.1.66: kern:    info: [2024-11-03T21:52:28.483569219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5082): osd6 weight 0x3333
10.0.1.66: kern:    info: [2024-11-03T21:52:28.483599219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5103): osd7 weight 0x1999
10.0.1.66: kern:    info: [2024-11-03T21:52:28.544411219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5118): osd3 down
10.0.1.66: kern:    info: [2024-11-03T21:52:28.544435219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5118): osd4 down
10.0.1.66: kern:    info: [2024-11-03T21:52:28.544446219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5118): osd5 down
10.0.1.66: kern:    info: [2024-11-03T21:52:28.547698219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5118): osd3 down
10.0.1.66: kern:    info: [2024-11-03T21:52:28.547720219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5118): osd4 down
10.0.1.66: kern:    info: [2024-11-03T21:52:28.547731219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5118): osd5 down
10.0.1.66: kern:    info: [2024-11-03T21:52:28.548170219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5118): osd3 down
10.0.1.66: kern:    info: [2024-11-03T21:52:28.548181219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5118): osd4 down
10.0.1.66: kern:    info: [2024-11-03T21:52:28.548189219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5118): osd5 down
10.0.1.66: user: warning: [2024-11-03T21:52:29.238769219Z]: [talos] service[kubelet](Stopping): Sending SIGKILL to task kubelet (PID 2755, container kubelet)
10.0.1.66: kern: warning: [2024-11-03T21:52:29.628134219Z]: rbd: image csi-vol-1b346968-84e6-4077-ac21-64eccb7bd1c1: failed to unwatch: -512
10.0.1.66: kern:    info: [2024-11-03T21:52:30.290287219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5118): osd3 down
10.0.1.66: kern:    info: [2024-11-03T21:52:30.290310219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5118): osd4 down
10.0.1.66: kern:    info: [2024-11-03T21:52:30.290322219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5118): osd5 down
10.0.1.66: kern:    info: [2024-11-03T22:02:30.620681219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5121): osd3 weight 0x0 (out)
10.0.1.66: kern:    info: [2024-11-03T22:02:30.620709219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5121): osd4 weight 0x0 (out)
10.0.1.66: kern:    info: [2024-11-03T22:02:30.620723219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5121): osd5 weight 0x0 (out)
10.0.1.66: kern:    info: [2024-11-03T22:02:30.622488219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5121): osd3 weight 0x0 (out)
10.0.1.66: kern:    info: [2024-11-03T22:02:30.622504219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5121): osd4 weight 0x0 (out)
10.0.1.66: kern:    info: [2024-11-03T22:02:30.622514219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5121): osd5 weight 0x0 (out)
10.0.1.66: kern:    info: [2024-11-03T22:02:30.622913219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5121): osd3 weight 0x0 (out)
10.0.1.66: kern:    info: [2024-11-03T22:02:30.622925219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5121): osd4 weight 0x0 (out)
10.0.1.66: kern:    info: [2024-11-03T22:02:30.622935219Z]: libceph (0433f7b8-197a-41c6-8a5c-bfe10bc76034 e5121): osd5 weight 0x0 (out)
10.0.1.66: kern: warning: [2024-11-03T22:15:19.019314219Z]: libceph: mon1 (1)10.102.164.207:6789 socket closed (con state OPEN)
10.0.1.66: kern:    info: [2024-11-03T22:15:19.019391219Z]: libceph: mon1 (1)10.102.164.207:6789 session lost, hunting for new mon
10.0.1.66: kern: warning: [2024-11-03T22:15:19.019412219Z]: libceph: mon1 (1)10.102.164.207:6789 socket closed (con state OPEN)
10.0.1.66: kern:    info: [2024-11-03T22:15:19.019455219Z]: libceph: mon1 (1)10.102.164.207:6789 session lost, hunting for new mon
10.0.1.66: kern: warning: [2024-11-03T22:15:19.022491219Z]: libceph: mon2 (1)10.110.107.1:6789 socket closed (con state OPEN)
10.0.1.66: kern:    info: [2024-11-03T22:15:19.022565219Z]: libceph: mon2 (1)10.110.107.1:6789 session lost, hunting for new mon
10.0.1.66: kern: warning: [2024-11-03T22:15:19.022594219Z]: libceph: mon2 (1)10.110.107.1:6789 socket closed (con state OPEN)
10.0.1.66: kern:    info: [2024-11-03T22:15:19.022652219Z]: libceph: mon2 (1)10.110.107.1:6789 session lost, hunting for new mon
10.0.1.66: kern: warning: [2024-11-03T22:15:19.022997219Z]: libceph: mon2 (1)10.110.107.1:6789 socket closed (con state OPEN)
10.0.1.66: kern:    info: [2024-11-03T22:15:19.023078219Z]: libceph: mon2 (1)10.110.107.1:6789 session lost, hunting for new mon
10.0.1.66: kern:    info: [2024-11-03T22:15:19.232808219Z]: libceph: mon1 (1)10.110.107.1:6789 session established
10.0.1.66: kern:    info: [2024-11-03T22:15:19.240925219Z]: libceph: mon0 (1)10.102.164.207:6789 session established
10.0.1.66: kern:    info: [2024-11-03T22:15:19.240985219Z]: libceph: mon0 (1)10.102.164.207:6789 session established
10.0.1.66: kern:    info: [2024-11-03T22:15:19.241372219Z]: libceph: mon0 (1)10.102.164.207:6789 session established
10.0.1.66: kern:    info: [2024-11-03T22:15:19.241461219Z]: libceph: mon0 (1)10.102.164.207:6789 session established
10.0.1.66: user: warning: [2024-11-03T22:17:43.684589219Z]: [talos] reboot via API received. actor id: 0d071680-3cb6-4249-b84d-7338920fcea9
10.0.1.66: user: warning: [2024-11-03T22:17:43.684638219Z]: [talos] task stopAllPods (1/1): failed: context canceled
10.0.1.66: user: warning: [2024-11-03T22:17:43.684668219Z]: [talos] phase cleanup (1/11): failed
10.0.1.66: user: warning: [2024-11-03T22:17:43.684678219Z]: [talos] reboot sequence: failed
10.0.1.66: user: warning: [2024-11-03T22:17:43.684709219Z]: [talos] reboot failed: error running phase 1 in reboot sequence: task 1/1: failed, context canceled
10.0.1.66: user: warning: [2024-11-03T22:17:43.684758219Z]: [talos] reboot sequence: 11 phase(s)
10.0.1.66: user: warning: [2024-11-03T22:17:43.684765219Z]: [talos] phase cleanup (1/11): 1 tasks(s)
10.0.1.66: user: warning: [2024-11-03T22:17:43.684781219Z]: [talos] task stopAllPods (1/1): starting
10.0.1.66: user: warning: [2024-11-03T22:17:43.684828219Z]: [talos] task stopAllPods (1/1): waiting for kubelet lifecycle finalizers
10.0.1.66: user: warning: [2024-11-03T22:17:43.684870219Z]: [talos] task stopAllPods (1/1): failed waiting for kubelet lifecycle finalizers: resource KubeletLifecycles.kubernetes.talos.dev(k8s/kubelet@undefined) doesn't exist
10.0.1.66: user: warning: [2024-11-03T22:17:43.684879219Z]: [talos] task stopAllPods (1/1): shutting down kubelet gracefully
10.0.1.66: user: warning: [2024-11-03T22:17:43.684993219Z]: [talos] task stopAllPods (1/1): failed waiting for inhibit shutdown lock: bad file descriptor
10.0.1.66: user: warning: [2024-11-03T22:17:47.221665219Z]: [talos] reboot via API received. actor id: 0082021a-a0ee-497f-a6f6-9023f103409f
10.0.1.66: user: warning: [2024-11-03T22:17:47.221731219Z]: [talos] task stopAllPods (1/1): failed: context canceled
10.0.1.66: user: warning: [2024-11-03T22:17:47.221742219Z]: [talos] phase cleanup (1/11): failed
10.0.1.66: user: warning: [2024-11-03T22:17:47.221747219Z]: [talos] reboot sequence: failed
10.0.1.66: user: warning: [2024-11-03T22:17:47.221754219Z]: [talos] reboot failed: error running phase 1 in reboot sequence: task 1/1: failed, context canceled
10.0.1.66: user: warning: [2024-11-03T22:17:47.221776219Z]: [talos] reboot sequence: 11 phase(s)
10.0.1.66: user: warning: [2024-11-03T22:17:47.221787219Z]: [talos] phase cleanup (1/11): 1 tasks(s)
10.0.1.66: user: warning: [2024-11-03T22:17:47.221816219Z]: [talos] task stopAllPods (1/1): starting
10.0.1.66: user: warning: [2024-11-03T22:17:47.221822219Z]: [talos] task stopAllPods (1/1): waiting for kubelet lifecycle finalizers
10.0.1.66: user: warning: [2024-11-03T22:17:47.221864219Z]: [talos] task stopAllPods (1/1): failed waiting for kubelet lifecycle finalizers: resource KubeletLifecycles.kubernetes.talos.dev(k8s/kubelet@undefined) doesn't exist
10.0.1.66: user: warning: [2024-11-03T22:17:47.221871219Z]: [talos] task stopAllPods (1/1): shutting down kubelet gracefully
10.0.1.66: user: warning: [2024-11-03T22:17:47.221922219Z]: [talos] task stopAllPods (1/1): failed waiting for inhibit shutdown lock: bad file descriptor
10.0.1.66: user: warning: [2024-11-03T22:17:49.371086219Z]: [talos] reboot via API received. actor id: c1253ad3-26f8-4249-9a9b-59032342ae65
10.0.1.66: user: warning: [2024-11-03T22:17:49.371138219Z]: [talos] task stopAllPods (1/1): failed: context canceled
10.0.1.66: user: warning: [2024-11-03T22:17:49.371149219Z]: [talos] phase cleanup (1/11): failed
10.0.1.66: user: warning: [2024-11-03T22:17:49.371154219Z]: [talos] reboot sequence: failed
10.0.1.66: user: warning: [2024-11-03T22:17:49.371162219Z]: [talos] reboot failed: error running phase 1 in reboot sequence: task 1/1: failed, context canceled
10.0.1.66: user: warning: [2024-11-03T22:17:49.371175219Z]: [talos] reboot sequence: 11 phase(s)
10.0.1.66: user: warning: [2024-11-03T22:17:49.371182219Z]: [talos] phase cleanup (1/11): 1 tasks(s)
10.0.1.66: user: warning: [2024-11-03T22:17:49.371195219Z]: [talos] task stopAllPods (1/1): starting
10.0.1.66: user: warning: [2024-11-03T22:17:49.371203219Z]: [talos] task stopAllPods (1/1): waiting for kubelet lifecycle finalizers
10.0.1.66: user: warning: [2024-11-03T22:17:49.371249219Z]: [talos] task stopAllPods (1/1): failed waiting for kubelet lifecycle finalizers: resource KubeletLifecycles.kubernetes.talos.dev(k8s/kubelet@undefined) doesn't exist
10.0.1.66: user: warning: [2024-11-03T22:17:49.371261219Z]: [talos] task stopAllPods (1/1): shutting down kubelet gracefully
10.0.1.66: user: warning: [2024-11-03T22:17:49.371323219Z]: [talos] task stopAllPods (1/1): failed waiting for inhibit shutdown lock: bad file descriptor
10.0.1.66: user: warning: [2024-11-03T22:17:53.356609219Z]: [talos] reboot via API received. actor id: 3039aec8-d0c3-41a5-864e-c515def16777
10.0.1.66: user: warning: [2024-11-03T22:17:53.356659219Z]: [talos] task stopAllPods (1/1): failed: context canceled
10.0.1.66: user: warning: [2024-11-03T22:17:53.356689219Z]: [talos] phase cleanup (1/11): failed
10.0.1.66: user: warning: [2024-11-03T22:17:53.356694219Z]: [talos] reboot sequence: failed
10.0.1.66: user: warning: [2024-11-03T22:17:53.356704219Z]: [talos] reboot failed: error running phase 1 in reboot sequence: task 1/1: failed, context canceled
10.0.1.66: user: warning: [2024-11-03T22:17:53.356718219Z]: [talos] reboot sequence: 11 phase(s)
10.0.1.66: user: warning: [2024-11-03T22:17:53.356722219Z]: [talos] phase cleanup (1/11): 1 tasks(s)
10.0.1.66: user: warning: [2024-11-03T22:17:53.356733219Z]: [talos] task stopAllPods (1/1): starting
10.0.1.66: user: warning: [2024-11-03T22:17:53.356754219Z]: [talos] task stopAllPods (1/1): waiting for kubelet lifecycle finalizers
10.0.1.66: user: warning: [2024-11-03T22:17:53.356779219Z]: [talos] task stopAllPods (1/1): failed waiting for kubelet lifecycle finalizers: resource KubeletLifecycles.kubernetes.talos.dev(k8s/kubelet@undefined) doesn't exist
10.0.1.66: user: warning: [2024-11-03T22:17:53.356788219Z]: [talos] task stopAllPods (1/1): shutting down kubelet gracefully
10.0.1.66: user: warning: [2024-11-03T22:20:02.313849219Z]: [talos] apply config request: mode reboot
10.0.1.66: user: warning: [2024-11-03T22:20:02.314486219Z]: [talos] task stopAllPods (1/1): failed: context canceled
10.0.1.66: user: warning: [2024-11-03T22:20:02.314540219Z]: [talos] phase cleanup (1/11): failed
10.0.1.66: user: warning: [2024-11-03T22:20:02.314565219Z]: [talos] reboot sequence: failed
10.0.1.66: user: warning: [2024-11-03T22:20:02.314608219Z]: [talos] reboot failed: error running phase 1 in reboot sequence: task 1/1: failed, context canceled
10.0.1.66: user: warning: [2024-11-03T22:20:02.314673219Z]: [talos] reboot sequence: 11 phase(s)
10.0.1.66: user: warning: [2024-11-03T22:20:02.314705219Z]: [talos] phase cleanup (1/11): 1 tasks(s)
10.0.1.66: user: warning: [2024-11-03T22:20:02.314782219Z]: [talos] task stopAllPods (1/1): starting
10.0.1.66: user: warning: [2024-11-03T22:20:02.314819219Z]: [talos] task stopAllPods (1/1): waiting for kubelet lifecycle finalizers
10.0.1.66: user: warning: [2024-11-03T22:20:02.314917219Z]: [talos] task stopAllPods (1/1): failed waiting for kubelet lifecycle finalizers: resource KubeletLifecycles.kubernetes.talos.dev(k8s/kubelet@undefined) doesn't exist
10.0.1.66: user: warning: [2024-11-03T22:20:02.314941219Z]: [talos] task stopAllPods (1/1): shutting down kubelet gracefully

10.0.1.66: user: warning: [2024-11-03T22:22:14.835863219Z]: [talos] reboot via API received. actor id: 57f86a95-5a86-4e5a-a27b-faa89505ed0b
10.0.1.66: user: warning: [2024-11-03T22:22:14.836058219Z]: [talos] task stopAllPods (1/1): failed: context canceled
10.0.1.66: user: warning: [2024-11-03T22:22:14.836147219Z]: [talos] phase cleanup (1/11): failed
10.0.1.66: user: warning: [2024-11-03T22:22:14.836192219Z]: [talos] reboot sequence: failed
10.0.1.66: user: warning: [2024-11-03T22:22:14.836242219Z]: [talos] apply configuration failed: error running phase 1 in reboot sequence: task 1/1: failed, context canceled
10.0.1.66: user: warning: [2024-11-03T22:22:14.836319219Z]: [talos] reboot sequence: 11 phase(s)
10.0.1.66: user: warning: [2024-11-03T22:22:14.836354219Z]: [talos] phase cleanup (1/11): 1 tasks(s)
10.0.1.66: user: warning: [2024-11-03T22:22:14.836415219Z]: [talos] task stopAllPods (1/1): starting
10.0.1.66: user: warning: [2024-11-03T22:22:14.836451219Z]: [talos] task stopAllPods (1/1): waiting for kubelet lifecycle finalizers
10.0.1.66: user: warning: [2024-11-03T22:22:14.836585219Z]: [talos] task stopAllPods (1/1): failed waiting for kubelet lifecycle finalizers: resource KubeletLifecycles.kubernetes.talos.dev(k8s/kubelet@undefined) doesn't exist
10.0.1.66: user: warning: [2024-11-03T22:22:14.836608219Z]: [talos] task stopAllPods (1/1): shutting down kubelet gracefully
10.0.1.66: user: warning: [2024-11-03T22:23:00.874688219Z]: [talos] apply config request: mode reboot
10.0.1.66: user: warning: [2024-11-03T22:23:00.875217219Z]: [talos] task stopAllPods (1/1): failed: context canceled
10.0.1.66: user: warning: [2024-11-03T22:23:00.875294219Z]: [talos] phase cleanup (1/11): failed
10.0.1.66: user: warning: [2024-11-03T22:23:00.875337219Z]: [talos] reboot sequence: failed
10.0.1.66: user: warning: [2024-11-03T22:23:00.875389219Z]: [talos] reboot failed: error running phase 1 in reboot sequence: task 1/1: failed, context canceled
10.0.1.66: user: warning: [2024-11-03T22:23:00.875453219Z]: [talos] reboot sequence: 11 phase(s)
10.0.1.66: user: warning: [2024-11-03T22:23:00.875482219Z]: [talos] phase cleanup (1/11): 1 tasks(s)
10.0.1.66: user: warning: [2024-11-03T22:23:00.875554219Z]: [talos] task stopAllPods (1/1): starting
10.0.1.66: user: warning: [2024-11-03T22:23:00.875589219Z]: [talos] task stopAllPods (1/1): waiting for kubelet lifecycle finalizers
10.0.1.66: user: warning: [2024-11-03T22:23:00.875697219Z]: [talos] task stopAllPods (1/1): failed waiting for kubelet lifecycle finalizers: resource KubeletLifecycles.kubernetes.talos.dev(k8s/kubelet@undefined) doesn't exist
10.0.1.66: user: warning: [2024-11-03T22:23:00.875725219Z]: [talos] task stopAllPods (1/1): shutting down kubelet gracefully
10.0.1.66: user: warning: [2024-11-03T22:23:00.875871219Z]: [talos] task stopAllPods (1/1): failed waiting for inhibit shutdown lock: bad file descriptor

Environment