search

siderolabs / talos

Talos Linux is a modern Linux distribution built for Kubernetes.
https://www.talos.dev
Mozilla Public License 2.0
6.63k stars 531 forks source link

Unable to install system extensions via upgrade #9354

Closed asianpianoman closed 3 weeks ago

asianpianoman commented 3 weeks ago

Bug Report

Description

I'm running a homelab cluster with 3 control planes, each on a separate physical Proxmox node. I'm trying to install some system extensions (iscsi-tools, tailscale, util-linux-tools) on them and having success on 2 of them (node_ids 102 and 103) but not on 1 of them (node_id 101). And of course, 101 is the one node I wanted these extensions on the most :face_with_diagonal_mouth: .

For each node, I ran this same command: talosctl upgrade --image factory.talos.dev/installer/708747e350d604ae9e57227d8dcf274091453ddb1097b765d4ea8884f1992c1f:v1.7.6 -m powercycle -n 10.69.100.<node_id>

and am using talosctl get extensions -n 10.69.100.<node_id> to verify:

> talosctl get extensions -n 10.69.100.101 NODE NAMESPACE TYPE ID VERSION NAME VERSION 10.69.100.101 runtime ExtensionStatus 0 1 qemu-guest-agent 8.2.2 10.69.100.101 runtime ExtensionStatus 1 1 schematic ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515

> talosctl get extensions -n 10.69.100.102 NODE NAMESPACE TYPE ID VERSION NAME VERSION 10.69.100.102 runtime ExtensionStatus 0 1 iscsi-tools v0.1.4 10.69.100.102 runtime ExtensionStatus 1 1 tailscale 1.68.1 10.69.100.102 runtime ExtensionStatus 2 1 util-linux-tools 2.39.3 10.69.100.102 runtime ExtensionStatus 3 1 schematic 708747e350d604ae9e57227d8dcf274091453ddb1097b765d4ea8884f1992c1f

> talosctl get extensions -n 10.69.100.103 NODE NAMESPACE TYPE ID VERSION NAME VERSION 10.69.100.103 runtime ExtensionStatus 0 1 iscsi-tools v0.1.4 10.69.100.103 runtime ExtensionStatus 1 1 tailscale 1.68.1 10.69.100.103 runtime ExtensionStatus 2 1 util-linux-tools 2.39.3 10.69.100.103 runtime ExtensionStatus 3 1 schematic 708747e350d604ae9e57227d8dcf274091453ddb1097b765d4ea8884f1992c1f

I'm not seeing any obvious errors in the dmesg logs but if any expert around here would be so kind as to take a look, I would super appreciate it. I don't have a HA control plane set up yet so the logs are split up between pre and post reboot.

Logs

pre_reboot.txt post_reboot.txt

Environment

smira commented 3 weeks ago

Most probably your VM is booting not from disk, but from some other source - e.g. ISO.

If upgrade fails, you would see a message.

asianpianoman commented 3 weeks ago

You were exactly right. Rookie mistake... fitting for a rookie like me. Sorry for the bother. Once I switched to the right drive and rebooted, extensions are now there.

The problem is even worse. Since I posted this I realized I'm doubly an idiot and I really wanted the iscsi tools extension on my worker node, not my control plane node.