mysticaltech
commented
1 year ago Thanks @Viktor-Osika for letting us know, there is a more official package that we can use, I will fix it.
Closed Viktor-Osika closed 1 year ago
mysticaltech
commented
1 year ago Thanks @Viktor-Osika for letting us know, there is a more official package that we can use, I will fix it.
mysticaltech
commented
1 year ago @Viktor-Osika I understood what is happening, we forgot to add a update lock for the package k3s-selinux, I am pushing new version now. But if your cluster is already deployed, just ssh into each node (see readme), and run:
transactional-update --continue shell <<< "zypper addlock k3s-selinux"
touch /var/run/reboot-requiredAfter that, your nodes will start updating normally again.
bvehse
commented
11 months ago @mysticaltech Thanks for the solution and your work in general!
On two separate clusters created with kube-hetzner we're experiencing HostKernelVersionDeviations alerts because Micro OS transactional updates haven't been successful one some nodes recently.
~ % kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
abc-agent-nbg1-1-vla Ready <none> 113d v1.27.8+k3s2 10.2.0.101 PLACEHOLDER openSUSE MicroOS 6.6.6-1-default containerd://1.7.7-k3s1.27
abc-agent-nbg1-2-nxw Ready <none> 113d v1.27.8+k3s2 10.3.0.101 PLACEHOLDER openSUSE MicroOS 6.6.6-1-default containerd://1.7.7-k3s1.27
abc-control-plane-nbg1-1-qbu Ready control-plane,etcd,master 515d v1.27.8+k3s2 10.255.0.101 PLACEHOLDER openSUSE MicroOS 6.5.9-1-default containerd://1.7.7-k3s1.27
abc-control-plane-nbg1-2-keh Ready control-plane,etcd,master 515d v1.27.8+k3s2 10.254.0.101 PLACEHOLDER openSUSE MicroOS 6.5.9-1-default containerd://1.7.7-k3s1.27
abc-control-plane-nbg1-3-cal Ready control-plane,etcd,master 515d v1.27.8+k3s2 10.253.0.101 PLACEHOLDER openSUSE MicroOS 6.5.9-1-default containerd://1.7.7-k3s1.27As you can see the agent nodes were recreated at a later point and have successfully updated to kernel version 6.6.6.-1-default while the control plane nodes are stuck at 6.5.9-1-default.
I encountered the same error as @Viktor-Osika and followed the procedure described above with the following result:
~ # transactional-update --continue shell <<< "zypper addlock k3s-selinux"
Checking for newer version.
Repository 'openSUSE-Tumbleweed-Non-Oss' is invalid.
[repo-non-oss|http://download.opensuse.org/tumbleweed/repo/non-oss/] Valid metadata not found at specified URL
History:
- Signature verification failed for repomd.xml
- Can't provide /repodata/repomd.xml
Please check if the URIs defined for this repository are pointing to a valid repository.
Repository 'openSUSE-Tumbleweed-Oss' is invalid.
[repo-oss|http://download.opensuse.org/tumbleweed/repo/oss/] Valid metadata not found at specified URL
History:
- Signature verification failed for repomd.xml
- Can't provide /repodata/repomd.xml
Please check if the URIs defined for this repository are pointing to a valid repository.
Some of the repositories have not been refreshed because of an error.
transactional-update 4.4.0 started
[...]As a result the kernel on these nodes is not updated. I can reach http://download.opensuse.org/tumbleweed/repo/non-oss/repodata/repomd.xml locally and on the nodes though.
Here are the configured repos:
~ # zypper lr -d
# | Alias | Name | Enabled | GPG Check | Refresh | Priority | Type | URI | Service
--+---------------------------+-----------------------------+---------+-----------+---------+----------+--------+----------------------------------------------------------+--------
1 | rancher-k3s-common-stable | Rancher K3s Common (stable) | Yes | ( p) Yes | No | 99 | rpm-md | https://rpm.rancher.io/k3s/stable/common/microos/noarch |
2 | repo-debug | openSUSE-Tumbleweed-Debug | No | ---- | ---- | 99 | NONE | http://download.opensuse.org/debug/tumbleweed/repo/oss/ |
3 | repo-non-oss | openSUSE-Tumbleweed-Non-Oss | Yes | (r ) Yes | Yes | 99 | rpm-md | http://download.opensuse.org/tumbleweed/repo/non-oss/ |
4 | repo-oss | openSUSE-Tumbleweed-Oss | Yes | (r ) Yes | Yes | 99 | rpm-md | http://download.opensuse.org/tumbleweed/repo/oss/ |
5 | repo-source | openSUSE-Tumbleweed-Source | No | ---- | ---- | 99 | NONE | http://download.opensuse.org/source/tumbleweed/repo/oss/ |
6 | repo-update | openSUSE-Tumbleweed-Update | Yes | (r ) Yes | Yes | 99 | rpm-md | http://download.opensuse.org/update/tumbleweed/ | Because of this I executed zypper -v ref:
~ # zypper -v ref
Verbosity: 2
Initializing Target
Specified repositories:
Checking whether to refresh metadata for Rancher K3s Common (stable)
Retrieving: repomd.xml .........................................................................................................................................................................[done (2.9 KiB/s)]
Repository 'Rancher K3s Common (stable)' is up to date.
Skipping disabled repository 'openSUSE-Tumbleweed-Debug'
Checking whether to refresh metadata for openSUSE-Tumbleweed-Non-Oss
Retrieving: repomd.xml .........................................................................................................................................................................[done (9.8 KiB/s)]
Retrieving: media ................................................................................................................................................................................[done (109 B/s)]
Retrieving: repomd.xml.asc .......................................................................................................................................................................[done (827 B/s)]
Retrieving: repomd.xml.key .....................................................................................................................................................................[done (1.6 KiB/s)]
Retrieving: repomd.xml .....................................................................................................................................................................................[done]
New repository or package signing key received:
Repository: openSUSE-Tumbleweed-Non-Oss
Key Fingerprint: AD48 5664 E901 B867 051A B15F 35A2 F86E 29B7 00A4
Key Name: openSUSE Project Signing Key <opensuse@opensuse.org>
Key Algorithm: RSA 4096
Key Created: Mon Jun 20 14:03:14 2022
Key Expires: Fri Jun 19 14:03:14 2026
Rpm Name: gpg-pubkey-29b700a4-62b07e22
Note: Signing data enables the recipient to verify that no modifications occurred after the data
were signed. Accepting data with no, wrong or unknown signature can lead to a corrupted system
and in extreme cases even to a system compromise.
Note: A GPG pubkey is clearly identified by its fingerprint. Do not rely on the key's name. If
you are not sure whether the presented key is authentic, ask the repository provider or check
their web site. Many providers maintain a web page showing the fingerprints of the GPG keys they
are using.
Do you want to reject the key, trust temporarily, or trust always? [r/t/a/?] (r): a
Subprocess failed. Error: Failed to import public key [35A2F86E29B700A4-62b07e22] [openSUSE Project Signing Key <opensuse@opensuse.org>] [expires: 2026-06-19]
History:
- Command exited with status 1.
- error: /var/tmp/zypp.ogavgi/pubkey-35A2F86E29B700A4-TeKC1n: key 1 import failed.
- error: can't create transaction lock on /usr/lib/sysimage/rpm/.rpm.lock (Read-only file system)
Repository: openSUSE-Tumbleweed-Non-Oss
Key Fingerprint: AD48 5664 E901 B867 051A B15F 35A2 F86E 29B7 00A4
Key Name: openSUSE Project Signing Key <opensuse@opensuse.org>
Key Algorithm: RSA 4096
Key Created: Mon Jun 20 14:03:14 2022
Key Expires: Fri Jun 19 14:03:14 2026
Rpm Name: gpg-pubkey-29b700a4-62b07e22
Retrieving: 40048a4bf062c8bbce9474b8ff0b63f42af9b6fa0a1df7cebd119398b64233647af9026832c78bff4e9c3901d168a18dc264f65529da08b5b4384a6dc2050307-primary.xml.zst ...............................................[done]
Retrieving: b5abb7b269a511ba13fff889514a0fb5861b57f08a3512e74583d87c894c5d02-susedata.xml.gz ...............................................................................................................[done]
Retrieving repository 'openSUSE-Tumbleweed-Non-Oss' metadata ...............................................................................................................................................[done]
Building repository 'openSUSE-Tumbleweed-Non-Oss' cache ....................................................................................................................................................[done]
Checking whether to refresh metadata for openSUSE-Tumbleweed-Oss
Retrieving: repomd.xml ........................................................................................................................................................................[done (10.5 KiB/s)]
Retrieving: media .................................................................................................................................................................................[done (93 B/s)]
Retrieving: repomd.xml.asc .......................................................................................................................................................................[done (827 B/s)]
Retrieving: repomd.xml.key .....................................................................................................................................................................[done (1.6 KiB/s)]
Retrieving: repomd.xml .....................................................................................................................................................................................[done]
Repository: openSUSE-Tumbleweed-Oss
Key Fingerprint: AD48 5664 E901 B867 051A B15F 35A2 F86E 29B7 00A4
Key Name: openSUSE Project Signing Key <opensuse@opensuse.org>
Key Algorithm: RSA 4096
Key Created: Mon Jun 20 14:03:14 2022
Key Expires: Fri Jun 19 14:03:14 2026
Rpm Name: gpg-pubkey-29b700a4-62b07e22
Retrieving: 5c34e521e18b22b8e35bbe8c985e3bfc0ccd12291633d64f6a81f49a05ce3aeb-appdata.xml.gz ...................................................................................................[done (23.2 MiB/s)]
Retrieving: 2c968f93e08ae0391e453db2c201670a5d73042c9d2e9a2d3f505446057e78f7-appdata-icons.tar.gz .............................................................................................[done (31.1 MiB/s)]
Retrieving: af2b85770531511b7339ec1fd148d40d631a06fedf96c1289a70ef8ee76883d868defd55740c56a831529e1ff20bd9a0922e1dca576abc926ebf87f1207ded6a-primary.xml.zst ..................................[done (98.0 MiB/s)]
Retrieving: d3d5d7ad2ee16fff407ce9e4d781ce4922351ed80c461216facb7fe375ead4be-susedata.xml.gz ..................................................................................................[done (44.6 MiB/s)]
Retrieving repository 'openSUSE-Tumbleweed-Oss' metadata ...................................................................................................................................................[done]
Building repository 'openSUSE-Tumbleweed-Oss' cache ........................................................................................................................................................[done]
Skipping disabled repository 'openSUSE-Tumbleweed-Source'
Checking whether to refresh metadata for openSUSE-Tumbleweed-Update
Retrieving: repomd.xml .........................................................................................................................................................................[done (3.5 KiB/s)]
Repository 'openSUSE-Tumbleweed-Update' is up to date.
All repositories have been refreshed.After verifying the new key's fingerprint via https://en.opensuse.org/openSUSE:Signing_Keys I accepted it. I presume the resulting error Failed to import public key was caused by the read-only filesystem. On subsequent calls of zypper -v ref no new keys or errors were encountered.
Running transactional-update again led to no errors but also no updated kernel after reboot:
~ # transactional-update --continue shell <<< "zypper addlock k3s-selinux"
Checking for newer version.
New version found - updating...
Loading repository data...
Reading installed packages...
Retrieving: transactional-update-4.5.0-1.1.x86_64 (openSUSE-Tumbleweed-Oss) (1/1), 73.2 KiB
Retrieving: transactional-update-4.5.0-1.1.x86_64.rpm ..........................................................................................................................................[done (2.7 KiB/s)]
(1/1) /tmp/transactional-update.hDj4EkXhiw/repo-oss/x86_64/transactional-update-4.5.0-1.1.x86_64.rpm .......................................................................................................[done]
Loading repository data...
Reading installed packages...
Retrieving: libtukit4-4.5.0-1.1.x86_64 (openSUSE-Tumbleweed-Oss) (1/2), 166.3 KiB
Retrieving: libtukit4-4.5.0-1.1.x86_64.rpm .................................................................................................................................................................[done]
(1/2) /tmp/transactional-update.hDj4EkXhiw/repo-oss/x86_64/libtukit4-4.5.0-1.1.x86_64.rpm ..................................................................................................................[done]
Retrieving: tukit-4.5.0-1.1.x86_64 (openSUSE-Tumbleweed-Oss) (2/2), 70.3 KiB
Retrieving: tukit-4.5.0-1.1.x86_64.rpm .....................................................................................................................................................................[done]
(2/2) /tmp/transactional-update.hDj4EkXhiw/repo-oss/x86_64/tukit-4.5.0-1.1.x86_64.rpm ......................................................................................................................[done]
transactional-update 4.5.0 started
Options: --continue shell
Separate /var detected.
2023-12-22 11:17:35 tukit 4.5.0 started
2023-12-22 11:17:35 Options: -c177 open
2023-12-22 11:17:35 Using snapshot 177 as base for new snapshot 178.
2023-12-22 11:17:35 /var/lib/overlay/177/etc
2023-12-22 11:17:35 Syncing /etc of previous snapshot 176 as base into new snapshot "/.snapshots/178/snapshot"
2023-12-22 11:17:35 SELinux is enabled.
/var/lib/kubelet/pods not reset as customized by admin to unconfined_u:object_r:container_file_t:s0
ID: 178
2023-12-22 11:17:38 Transaction completed.
Opening chroot in snapshot 178, continue with 'exit'
2023-12-22 11:17:38 tukit 4.5.0 started
2023-12-22 11:17:38 Options: call 178 bash
/var/lib/kubelet/pods not reset as customized by admin to unconfined_u:object_r:container_file_t:s0
2023-12-22 11:17:39 Executing `bash`:
2023-12-22 11:17:40 Application returned with exit status 0.
2023-12-22 11:17:40 Transaction completed.
2023-12-22 11:17:40 tukit 4.5.0 started
2023-12-22 11:17:40 Options: close 178
/var/lib/kubelet/pods not reset as customized by admin to unconfined_u:object_r:container_file_t:s0
2023-12-22 11:17:42 New default snapshot is #178 (/.snapshots/178/snapshot).
2023-12-22 11:17:42 Transaction completed.
Please reboot your machine to activate the changes and avoid data loss.
New default snapshot is #178 (/.snapshots/178/snapshot).
transactional-update finished
~ # touch /var/run/reboot-required~ # uname -r
6.5.9-1-defaultI'm thinking either there is something wrong with the mirrors I am redirected to (see https://github.com/openSUSE/zypper/issues/478) or I misconfigured something during my manual attempts at using transactional-update. Could using different (current at the time) versions of the kube-hetzner provider for the initial creation of agent and control plane nodes be a problem?
Thanks a lot in advance!
Description
I've been running a cluster for two weeks and noticed that daily automatic MicroOS updates are not working on all the nodes with the following logs:
And this is the cause:
As a sidenote - is there an easy way to expose microos updates metrics so they can be monitored?
Kube.tf file