search

nberlee / talos

Friendly fork for Turing RK1 on Talos
https://www.talos.dev
Mozilla Public License 2.0
61 stars 0 forks source link

Upgrade fails #7

Closed huiser closed 3 months ago

huiser commented 4 months ago

Bug Report

Description

I'm trying to upgrade my TuringPi2/RK1 cluster from v1.6.5 to v1.6.7 (before going to v1.7.x), but it fails.

Logs

⨠ talosctl upgrade -i ghcr.io/nberlee/installer:v1.6.7-rk3588 --debug -n 10.76.1.41 
◰ watching nodes: [10.76.1.41]
    * 10.76.1.41: 1 error(s) occurred:
    sequence error: sequence failed: error running phase 11 in upgrade sequence: task 1/1: failed, task "upgrade" failed: exit code 1
console logs for nodes ["10.76.1.41"]:
10.76.1.41: user: warning: [2024-05-06T06:33:04.535372153Z]: [talos] upgrade request received: preserve false, staged false, force false, reboot mode DEFAULT
10.76.1.41: user: warning: [2024-05-06T06:33:04.546523153Z]: [talos] validating "ghcr.io/nberlee/installer:v1.6.7-rk3588"
10.76.1.41: user: warning: [2024-05-06T06:33:22.151084153Z]: [talos] etcd upgrade mutex locked with session ID 35438f4a731b3049
10.76.1.41: user: warning: [2024-05-06T06:33:22.251605153Z]: [talos] upgrade sequence: 15 phase(s)
10.76.1.41: user: warning: [2024-05-06T06:33:22.256962153Z]: [talos] phase drain (1/15): 1 tasks(s)
10.76.1.41: user: warning: [2024-05-06T06:33:22.262503153Z]: [talos] task cordonAndDrainNode (1/1): starting
10.76.1.41: user: warning: [2024-05-06T06:33:22.268916153Z]: [talos] task cordonAndDrainNode (1/1): waiting for node to be cordoned
10.76.1.41: user: warning: [2024-05-06T06:33:22.277534153Z]: [talos] etcd upgrade mutex unlocked and session closed
10.76.1.41: user: warning: [2024-05-06T06:33:22.522347153Z]: [talos] skipping DaemonSet pod rook-ceph/rook-discover-hchlz
10.76.1.41: user: warning: [2024-05-06T06:33:22.530095153Z]: [talos] skipping mirror pod kube-system/kube-scheduler-cp1
10.76.1.41: user: warning: [2024-05-06T06:33:22.537518153Z]: [talos] skipping DaemonSet pod kube-system/cilium-5sm5c
10.76.1.41: user: warning: [2024-05-06T06:33:22.544659153Z]: [talos] skipping mirror pod kube-system/kube-apiserver-cp1
10.76.1.41: user: warning: [2024-05-06T06:33:22.552073153Z]: [talos] skipping mirror pod kube-system/kube-controller-manager-cp1
10.76.1.41: user: warning: [2024-05-06T06:33:22.560326153Z]: [talos] skipping DaemonSet pod rook-ceph/csi-rbdplugin-7q2hw
10.76.1.41: user: warning: [2024-05-06T06:33:22.567909153Z]: [talos] skipping DaemonSet pod loki/loki-canary-9kj7c
10.76.1.41: user: warning: [2024-05-06T06:33:22.574849153Z]: [talos] skipping DaemonSet pod loki/promtail-fp4v8
10.76.1.41: user: warning: [2024-05-06T06:33:22.581969153Z]: [talos] skipping DaemonSet pod prometheus/kube-prometheus-stack-prometheus-node-exporter-mxbvt
10.76.1.41: user: warning: [2024-05-06T06:33:22.592937153Z]: [talos] skipping DaemonSet pod rook-ceph/csi-cephfsplugin-xh87d
10.76.1.41: user: warning: [2024-05-06T06:34:22.609958153Z]: [talos] WARNING: failed to evict pod: failed waiting on pod rook-ceph/rook-ceph-exporter-cp1-7cbc6447dd-p2hv4 to be deleted: 2 error(s) occurred:
10.76.1.41: user: warning: [2024-05-06T06:34:22.625971153Z]:  pod is still running on the node
10.76.1.41: user: warning: [2024-05-06T06:34:22.630997153Z]:  timeout
10.76.1.41: user: warning: [2024-05-06T06:34:22.634210153Z]: [talos] task cordonAndDrainNode (1/1): done, 1m0.375860217s
10.76.1.41: user: warning: [2024-05-06T06:34:22.641880153Z]: [talos] phase drain (1/15): done, 1m0.388986407s
10.76.1.41: user: warning: [2024-05-06T06:34:22.648294153Z]: [talos] phase cleanup (2/15): 1 tasks(s)
10.76.1.41: user: warning: [2024-05-06T06:34:22.653878153Z]: [talos] task removeAllPods (1/1): starting
10.76.1.41: user: warning: [2024-05-06T06:34:22.659633153Z]: [talos] task removeAllPods (1/1): waiting for kubelet lifecycle finalizers
10.76.1.41: user: warning: [2024-05-06T06:34:22.677348153Z]: [talos] task removeAllPods (1/1): shutting down kubelet gracefully
10.76.1.41: user: warning: [2024-05-06T06:34:52.693757153Z]: [talos] service[kubelet](Stopping): Sending SIGTERM to task kubelet (PID 5041, container kubelet)
10.76.1.41: user: warning: [2024-05-06T06:34:52.865332153Z]: [talos] service[kubelet](Finished): Service finished successfully
10.76.1.41: user: warning: [2024-05-06T06:34:52.915614153Z]: [talos] removing pod loki/loki-canary-9kj7c with network mode "POD"
10.76.1.41: user: warning: [2024-05-06T06:34:52.924194153Z]: [talos] removing pod loki/promtail-fp4v8 with network mode "POD"
10.76.1.41: user: warning: [2024-05-06T06:34:52.932183153Z]: [talos] removing pod rook-ceph/rook-discover-hchlz with network mode "POD"
10.76.1.41: user: warning: [2024-05-06T06:34:52.941098153Z]: [talos] removing pod rook-ceph/rook-ceph-exporter-cp1-7cbc6447dd-p2hv4 with network mode "POD"
10.76.1.41: user: warning: [2024-05-06T06:34:52.951969153Z]: [talos] removing container loki/promtail-fp4v8:promtail
10.76.1.41: user: warning: [2024-05-06T06:34:52.965844153Z]: [talos] removed container loki/promtail-fp4v8:promtail
10.76.1.41: user: warning: [2024-05-06T06:35:13.075928153Z]: [talos] removed pod loki/loki-canary-9kj7c
10.76.1.41: user: warning: [2024-05-06T06:35:13.084004153Z]: [talos] removed pod rook-ceph/rook-discover-hchlz
10.76.1.41: user: warning: [2024-05-06T06:35:13.125148153Z]: [talos] removed pod rook-ceph/rook-ceph-exporter-cp1-7cbc6447dd-p2hv4
10.76.1.41: user: warning: [2024-05-06T06:35:13.185509153Z]: [talos] removed pod loki/promtail-fp4v8
10.76.1.41: user: warning: [2024-05-06T06:35:13.196790153Z]: [talos] removing pod rook-ceph/csi-cephfsplugin-xh87d with network mode "NODE"
10.76.1.41: user: warning: [2024-05-06T06:35:13.206358153Z]: [talos] removing pod kube-system/kube-scheduler-cp1 with network mode "NODE"
10.76.1.41: user: warning: [2024-05-06T06:35:13.215693153Z]: [talos] removing pod rook-ceph/csi-rbdplugin-7q2hw with network mode "NODE"
10.76.1.41: user: warning: [2024-05-06T06:35:13.225637153Z]: [talos] removing pod kube-system/kube-controller-manager-cp1 with network mode "NODE"
10.76.1.41: user: warning: [2024-05-06T06:35:13.235822153Z]: [talos] removing pod kube-system/cilium-5sm5c with network mode "NODE"
10.76.1.41: user: warning: [2024-05-06T06:35:13.244498153Z]: [talos] removing pod kube-system/kube-apiserver-cp1 with network mode "NODE"
10.76.1.41: user: warning: [2024-05-06T06:35:13.253723153Z]: [talos] removing pod prometheus/kube-prometheus-stack-prometheus-node-exporter-mxbvt with network mode "NODE"
10.76.1.41: user: warning: [2024-05-06T06:35:13.266069153Z]: [talos] removing container kube-system/kube-scheduler-cp1:kube-scheduler
10.76.1.41: user: warning: [2024-05-06T06:35:13.274987153Z]: [talos] removed pod rook-ceph/csi-cephfsplugin-xh87d
10.76.1.41: user: warning: [2024-05-06T06:35:13.282085153Z]: [talos] removing container kube-system/kube-controller-manager-cp1:kube-controller-manager
10.76.1.41: user: warning: [2024-05-06T06:35:13.293155153Z]: [talos] removed pod rook-ceph/csi-rbdplugin-7q2hw
10.76.1.41: user: warning: [2024-05-06T06:35:13.299709153Z]: [talos] removing container kube-system/kube-apiserver-cp1:kube-apiserver
10.76.1.41: user: warning: [2024-05-06T06:35:13.308638153Z]: [talos] removed pod kube-system/cilium-5sm5c
10.76.1.41: user: warning: [2024-05-06T06:35:13.315565153Z]: [talos] removed pod prometheus/kube-prometheus-stack-prometheus-node-exporter-mxbvt
10.76.1.41: user: warning: [2024-05-06T06:35:13.325642153Z]: [talos] removed container kube-system/kube-scheduler-cp1:kube-scheduler
10.76.1.41: user: warning: [2024-05-06T06:35:13.335000153Z]: [talos] removed container kube-system/kube-controller-manager-cp1:kube-controller-manager
10.76.1.41: user: warning: [2024-05-06T06:35:13.345684153Z]: [talos] removed container kube-system/kube-apiserver-cp1:kube-apiserver
10.76.1.41: user: warning: [2024-05-06T06:35:13.355247153Z]: [talos] removed pod kube-system/kube-scheduler-cp1
10.76.1.41: user: warning: [2024-05-06T06:35:13.361943153Z]: [talos] removed pod kube-system/kube-controller-manager-cp1
10.76.1.41: user: warning: [2024-05-06T06:35:13.470614153Z]: [talos] removed pod kube-system/kube-apiserver-cp1
10.76.1.41: user: warning: [2024-05-06T06:35:13.477807153Z]: [talos] task removeAllPods (1/1): done, 50.827400351s
10.76.1.41: user: warning: [2024-05-06T06:35:13.484669153Z]: [talos] phase cleanup (2/15): done, 50.839866746s
10.76.1.41: user: warning: [2024-05-06T06:35:13.491128153Z]: [talos] phase dbus (3/15): 1 tasks(s)
10.76.1.41: user: warning: [2024-05-06T06:35:13.496432153Z]: [talos] task stopDBus (1/1): starting
10.76.1.41: user: warning: [2024-05-06T06:35:13.502189153Z]: [talos] task stopDBus (1/1): done, 5.719684ms
10.76.1.41: user: warning: [2024-05-06T06:35:13.508517153Z]: [talos] phase dbus (3/15): done, 17.347771ms
10.76.1.41: user: warning: [2024-05-06T06:35:13.514673153Z]: [talos] phase leave (4/15): 1 tasks(s)
10.76.1.41: user: warning: [2024-05-06T06:35:13.520286153Z]: [talos] task leaveEtcd (1/1): starting
10.76.1.41: user: warning: [2024-05-06T06:35:13.559613153Z]: [talos] service[etcd](Stopping): Sending SIGTERM to task etcd (PID 5076, container etcd)
10.76.1.41: user: warning: [2024-05-06T06:35:13.570877153Z]: [talos] removed static pod {"component": "controller-runtime", "controller": "k8s.StaticPodServerController", "id": "kube-scheduler"}
10.76.1.41: user: warning: [2024-05-06T06:35:13.585706153Z]: [talos] removed static pod {"component": "controller-runtime", "controller": "k8s.StaticPodServerController", "id": "kube-apiserver"}
10.76.1.41: user: warning: [2024-05-06T06:35:13.600403153Z]: [talos] removed static pod {"component": "controller-runtime", "controller": "k8s.StaticPodServerController", "id": "kube-controller-manager"}
10.76.1.41: user: warning: [2024-05-06T06:35:14.720085153Z]: [talos] service[etcd](Finished): Service finished successfully
10.76.1.41: user: warning: [2024-05-06T06:35:14.865642153Z]: [talos] task leaveEtcd (1/1): done, 1.345468732s
10.76.1.41: user: warning: [2024-05-06T06:35:14.872214153Z]: [talos] phase leave (4/15): done, 1.357664733s
10.76.1.41: user: warning: [2024-05-06T06:35:14.878498153Z]: [talos] phase stopServices (5/15): 1 tasks(s)
10.76.1.41: user: warning: [2024-05-06T06:35:14.884784153Z]: [talos] task stopServicesForUpgrade (1/1): starting
10.76.1.41: user: warning: [2024-05-06T06:35:14.891567153Z]: [talos] service[udevd](Stopping): Sending SIGTERM to Process(["/sbin/udevd" "--resolve-names=never"])
10.76.1.41: user: warning: [2024-05-06T06:35:14.903110153Z]: [talos] service[cri](Stopping): Sending SIGTERM to Process(["/bin/containerd" "--address" "/run/containerd/containerd.sock" "--config" "/etc/cri/containerd.toml"])
10.76.1.41: user: warning: [2024-05-06T06:35:14.920808153Z]: [talos] service[trustd](Stopping): Sending SIGTERM to task trustd (PID 4992, container trustd)
10.76.1.41: user: warning: [2024-05-06T06:35:14.931725153Z]: [talos] service[udevd](Finished): Service finished successfully
10.76.1.41: user: warning: [2024-05-06T06:35:14.939560153Z]: [talos] service[cri](Finished): Service finished successfully
10.76.1.41: user: warning: [2024-05-06T06:35:15.040910153Z]: [talos] service[trustd](Finished): Service finished successfully
10.76.1.41: user: warning: [2024-05-06T06:35:15.048876153Z]: [talos] task stopServicesForUpgrade (1/1): done, 164.101859ms
10.76.1.41: user: warning: [2024-05-06T06:35:15.056569153Z]: [talos] phase stopServices (5/15): done, 178.076274ms
10.76.1.41: user: warning: [2024-05-06T06:35:15.063443153Z]: [talos] phase unmountUser (6/15): 1 tasks(s)
10.76.1.41: user: warning: [2024-05-06T06:35:15.069445153Z]: [talos] task unmountUserDisks (1/1): starting
10.76.1.41: user: warning: [2024-05-06T06:35:15.075542153Z]: [talos] task unmountUserDisks (1/1): done, 6.098293ms
10.76.1.41: user: warning: [2024-05-06T06:35:15.082409153Z]: [talos] phase unmountUser (6/15): done, 18.973049ms
10.76.1.41: user: warning: [2024-05-06T06:35:15.089047153Z]: [talos] phase unmount (7/15): 2 tasks(s)
10.76.1.41: user: warning: [2024-05-06T06:35:15.094630153Z]: [talos] task unmountPodMounts (2/2): starting
10.76.1.41: user: warning: [2024-05-06T06:35:15.100820153Z]: [talos] task unmountOverlayFilesystems (1/2): starting
10.76.1.41: user: warning: [2024-05-06T06:35:15.108138153Z]: [talos] task unmountPodMounts (2/2): unmounting /var/lib/kubelet/pods/7c1b1e50-adf5-413f-ab45-f0c9b3192cc6/volumes/kubernetes.io~secret/config
10.76.1.41: user: warning: [2024-05-06T06:35:15.124296153Z]: [talos] task unmountPodMounts (2/2): unmounting /var/lib/kubelet/pods/7c1b1e50-adf5-413f-ab45-f0c9b3192cc6/volumes/kubernetes.io~projected/kube-api-access-p5d6d
10.76.1.41: user: warning: [2024-05-06T06:35:15.141972153Z]: [talos] task unmountPodMounts (2/2): done, 47.329448ms
10.76.1.41: user: warning: [2024-05-06T06:35:15.150447153Z]: [talos] task unmountOverlayFilesystems (1/2): done, 55.548595ms
10.76.1.41: user: warning: [2024-05-06T06:35:15.158433153Z]: [talos] phase unmount (7/15): done, 69.380374ms
10.76.1.41: user: warning: [2024-05-06T06:35:15.164734153Z]: [talos] phase unmountBind (8/15): 1 tasks(s)
10.76.1.41: user: warning: [2024-05-06T06:35:15.170866153Z]: [talos] task unmountSystemDiskBindMounts (1/1): starting
10.76.1.41: user: warning: [2024-05-06T06:35:15.178297153Z]: [talos] task unmountSystemDiskBindMounts (1/1): unmounting /system/state
10.76.1.41: kern:  notice: [2024-05-06T06:35:15.187330153Z]: XFS (mmcblk0p5): Unmounting Filesystem a41c083d-f8b8-40e1-b017-1475b399a125
10.76.1.41: user: warning: [2024-05-06T06:35:15.203359153Z]: [talos] task unmountSystemDiskBindMounts (1/1): unmounting /var
10.76.1.41: kern:  notice: [2024-05-06T06:35:15.447600153Z]: XFS (mmcblk0p6): Unmounting Filesystem 2c949d80-249e-4f7e-b427-081031015ed9
10.76.1.41: user: warning: [2024-05-06T06:35:15.490590153Z]: [talos] task unmountSystemDiskBindMounts (1/1): done, 319.74368ms
10.76.1.41: user: warning: [2024-05-06T06:35:15.498640153Z]: [talos] phase unmountBind (8/15): done, 333.940236ms
10.76.1.41: user: warning: [2024-05-06T06:35:15.506266153Z]: [talos] phase unmountSystem (9/15): 2 tasks(s)
10.76.1.41: user: warning: [2024-05-06T06:35:15.512554153Z]: [talos] task unmountStatePartition (2/2): starting
10.76.1.41: user: warning: [2024-05-06T06:35:15.519468153Z]: [talos] task unmountEphemeralPartition (1/2): starting
10.76.1.41: user: warning: [2024-05-06T06:35:15.526933153Z]: [talos] task unmountStatePartition (2/2): done, 7.220119ms
10.76.1.41: user: warning: [2024-05-06T06:35:15.534368153Z]: [talos] task unmountEphemeralPartition (1/2): done, 14.983651ms
10.76.1.41: user: warning: [2024-05-06T06:35:15.542312153Z]: [talos] phase unmountSystem (9/15): done, 36.062385ms
10.76.1.41: user: warning: [2024-05-06T06:35:15.549226153Z]: [talos] phase verifyDisk (10/15): 1 tasks(s)
10.76.1.41: user: warning: [2024-05-06T06:35:15.555302153Z]: [talos] task verifyDiskAvailability (1/1): starting
10.76.1.41: user: warning: [2024-05-06T06:35:15.562864153Z]: [talos] task verifyDiskAvailability (1/1): done, 7.560517ms
10.76.1.41: user: warning: [2024-05-06T06:35:15.570390153Z]: [talos] phase verifyDisk (10/15): done, 21.167699ms
10.76.1.41: user: warning: [2024-05-06T06:35:15.577088153Z]: [talos] phase upgrade (11/15): 1 tasks(s)
10.76.1.41: user: warning: [2024-05-06T06:35:15.582900153Z]: [talos] task upgrade (1/1): starting
10.76.1.41: user: warning: [2024-05-06T06:35:15.612788153Z]: [talos] task upgrade (1/1): performing upgrade via "ghcr.io/nberlee/installer:v1.6.7-rk3588"
10.76.1.41: user: warning: [2024-05-06T06:35:15.628704153Z]: [talos] pulling extension "ghcr.io/nberlee/rk3588:v1.6.5"
10.76.1.41: user: warning: [2024-05-06T06:35:18.205612153Z]: 2024/05/06 06:35:21 running Talos installer v1.6.7
10.76.1.41: user: warning: [2024-05-06T06:35:18.212183153Z]: 2024/05/06 06:35:21 WARNING: config validation:
10.76.1.41: kern:  notice: [2024-05-06T06:35:18.215701153Z]: XFS (mmcblk0p3): Mounting V5 Filesystem 832c26ad-0e47-4d3d-afdd-7bb231b93a87
10.76.1.41: user: warning: [2024-05-06T06:35:18.218438153Z]: 2024/05/06 06:35:21   .machine.install.extensions is deprecated, please see https://www.talos.dev/latest/talos-guides/install/boot-assets/
10.76.1.41: kern:    info: [2024-05-06T06:35:18.267291153Z]: XFS (mmcblk0p3): Ending clean mount
10.76.1.41: kern:  notice: [2024-05-06T06:35:18.275582153Z]: XFS (mmcblk0p3): Unmounting Filesystem 832c26ad-0e47-4d3d-afdd-7bb231b93a87
10.76.1.41: user: warning: [2024-05-06T06:35:18.297333153Z]: 2024/05/06 06:35:21 running pre-flight checks
10.76.1.41: user: warning: [2024-05-06T06:35:18.304430153Z]: 2024/05/06 06:35:21 host Talos version: v1.6.5
10.76.1.41: user: warning: [2024-05-06T06:35:18.321734153Z]: 2024/05/06 06:35:21 host Kubernetes versions: kubelet: 1.29.3, kube-apiserver: 1.29.3, kube-scheduler: 1.29.3, kube-controller-manager: 1.29.3
10.76.1.41: user: warning: [2024-05-06T06:35:18.337297153Z]: 2024/05/06 06:35:21 all pre-flight checks successful
10.76.1.41: user: warning: [2024-05-06T06:35:18.344035153Z]: 2024/05/06 06:35:21 discovered system extensions:
10.76.1.41: user: warning: [2024-05-06T06:35:18.350491153Z]: 2024/05/06 06:35:21 NAME             VERSION   AUTHOR
10.76.1.41: user: warning: [2024-05-06T06:35:18.357332153Z]: 2024/05/06 06:35:21 rk3588-drivers   v1.6.5    Nico Berlee
10.76.1.41: user: warning: [2024-05-06T06:35:18.364657153Z]: 2024/05/06 06:35:21 validating system extensions
10.76.1.41: user: warning: [2024-05-06T06:35:18.371032153Z]: 2024/05/06 06:35:21 preparing to run depmod to generate kernel modules dependency tree
10.76.1.41: user: warning: [2024-05-06T06:35:23.847023153Z]: Error: copying kernel modules from /system/extensions/000.ghcr.io-nberlee-rk3588-v1.6.5/rootfs/lib/modules failed: stat /system/extensions/000.ghcr.io-nberlee-rk3588-v1.6.5/rootfs/lib/modules/6.6.22-talos: no such file or directory
10.76.1.41: user: warning: [2024-05-06T06:35:23.871241153Z]: Usage:
10.76.1.41: user: warning: [2024-05-06T06:35:23.873506153Z]:   installer install [flags]
10.76.1.41: user: warning: [2024-05-06T06:35:23.877822153Z]: 
10.76.1.41: user: warning: [2024-05-06T06:35:23.879557153Z]: Flags:
10.76.1.41: user: warning: [2024-05-06T06:35:23.881809153Z]:   -h, --help   help for install
10.76.1.41: user: warning: [2024-05-06T06:35:23.886492153Z]: 
10.76.1.41: user: warning: [2024-05-06T06:35:23.888160153Z]: Global Flags:
10.76.1.41: user: warning: [2024-05-06T06:35:23.891098153Z]:       --arch string                    The target architecture (default "arm64")
10.76.1.41: user: warning: [2024-05-06T06:35:23.900545153Z]:       --board string                   The value of talos.board (default "none")
10.76.1.41: user: warning: [2024-05-06T06:35:23.909996153Z]:       --bootloader                     Deprecated: no op (default true)
10.76.1.41: user: warning: [2024-05-06T06:35:23.918570153Z]:       --config string                  The value of talos.config
10.76.1.41: user: warning: [2024-05-06T06:35:23.926478153Z]:       --disk string                    The path to the disk to install to
10.76.1.41: user: warning: [2024-05-06T06:35:23.935252153Z]:       --extra-kernel-arg stringArray   Extra argument to pass to the kernel
10.76.1.41: user: warning: [2024-05-06T06:35:23.944214153Z]:       --force                          Indicates that the install should forcefully format the partition
10.76.1.41: user: warning: [2024-05-06T06:35:23.956000153Z]:       --meta metaValueSlice            A key/value pair for META (default [])
10.76.1.41: user: warning: [2024-05-06T06:35:23.965172153Z]:       --platform string                The value of talos.platform
10.76.1.41: user: warning: [2024-05-06T06:35:23.973265153Z]:       --upgrade                        Indicates that the install is being performed by an upgrade
10.76.1.41: user: warning: [2024-05-06T06:35:23.984455153Z]:       --zero                           Indicates that the install should write zeros to the disk before installing
10.76.1.41: user: warning: [2024-05-06T06:35:23.997206153Z]: 
10.76.1.41: user: warning: [2024-05-06T06:35:23.998881153Z]: copying kernel modules from /system/extensions/000.ghcr.io-nberlee-rk3588-v1.6.5/rootfs/lib/modules failed: stat /system/extensions/000.ghcr.io-nberlee-rk3588-v1.6.5/rootfs/lib/modules/6.6.22-talos: no such file or directory
10.76.1.41: user: warning: [2024-05-06T06:35:24.081845153Z]: [talos] task upgrade (1/1): failed: task "upgrade" failed: exit code 1
10.76.1.41: user: warning: [2024-05-06T06:35:24.090616153Z]: [talos] phase upgrade (11/15): failed
10.76.1.41: user: warning: [2024-05-06T06:35:24.096109153Z]: [talos] upgrade sequence: failed

Environment

hagak commented 4 months ago

If you only have a single control node you need to add --preserve=true

huiser commented 4 months ago

If you only have a single control node you need to add --preserve=true

Sorry, forgot to mention: I have 4 nodes, 3 control-plane nodes and 1 worker. And cluster.allowSchedulingOnControlPlanes set to true

nberlee commented 4 months ago

@huiser The update wants to install the wrong extension. Is 1.6.5 of the rk3588 by any chance in your machine config (machine.install.extensions)? If so, either update it, or if it is the only extension, remove it from the machine config

huiser commented 4 months ago

@huiser The update wants to install the wrong extension. Is 1.6.5 of the rk3588 by any chance in your machine config (machine.install.extensions)? If so, either update it, or if it is the only extension, remove it from the machine config

I removed the extension from the machine config, that fixed my problem. Thanks!