Switch to repos to maintaned pkgs.k8s.io

deric commented 1 year ago

Summary

New Kubernetes releases will be published to community owned repost pkgs.k8s.io instead of Google hosted repos.

up to v1.23 available only in packages.cloud.google.com
v1.24 to v1.27 can be installed from both repos
v1.28 onward should be installed from pkgs.k8s.io

Added tests to ensure backwards compatibility.

Related Issues (if any)

655

Checklist

[x] 🟢 Spec tests.
[ ] 🟢 Acceptance tests.
[x] Manually verified. (For example puppet apply)

deric commented 1 year ago

The apt source must have following definition:

deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.25/deb /

Because otherwise we'd get following errors (as CDN is being used):

Notice: /Stage[main]/Apt::Update/Exec[apt_update]/returns: W: GPG error: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.25/deb  InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 234654DA9A296436
Notice: /Stage[main]/Apt::Update/Exec[apt_update]/returns: E: The repository 'https://pkgs.k8s.io/core:/stable:/v1.25/deb  InRelease' is not signed.

The absence of $repos for Debian is weird, because packages are reported to come from an unknown source.

The param key

apt::source { 'kubernetes':
   ...
   key => {}
}

and keyring are mutually exclusive.

apt::source { 'kubernetes':
   ...
   keyring => '',
}

I'm not sure if we can write this in a better, backward compatible way :thinking:

XSpielinbox commented 1 year ago

The apt source must have following definition:
deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.25/deb /
Because otherwise we'd get following errors (as CDN is being used):
Notice: /Stage[main]/Apt::Update/Exec[apt_update]/returns: W: GPG error: https://prod-cdn.packages.k8s.io/repositories/isv:/kubernetes:/core:/stable:/v1.25/deb  InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 234654DA9A296436
Notice: /Stage[main]/Apt::Update/Exec[apt_update]/returns: E: The repository 'https://pkgs.k8s.io/core:/stable:/v1.25/deb  InRelease' is not signed.
The absence of $repos for Debian is weird, because packages are reported to come from an unknown source.

The param key
apt::source { 'kubernetes':
   ...
   key => {}
}
and keyring are mutually exclusive.
apt::source { 'kubernetes':
   ...
   keyring => '',
}
I'm not sure if we can write this in a better, backward compatible way 🤔

The signed-by option should be backwards compatible with the very old Debian 9 and Ubuntu 16.04 - that should be more than enough in my opinion. The newest versions of Debian and Ubuntu don't work with the old apt-key. Therefore I would definitely opt for the newer variant. See also: https://github.com/puppetlabs/puppetlabs-apt/issues/1034

danifr commented 11 months ago

Thanks for working on this, I am also interested in having it merged as soon as possible.

For RedHat based OS, would it be possible to also add exclude in the repo definition?

Like in the official example provided here: https://kubernetes.io/blog/2023/08/15/pkgs-k8s-io-introduction/#how-to-migrate-rpm

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

Thank you

deric commented 11 months ago

@danifr Sure. Just curious, what does the exclude mean? See baea6f9232c6d81f86e43b9950f92aaef0431336

XSpielinbox commented 11 months ago

@danifr Sure. Just curious, what does the exclude mean? See baea6f9

As far as I can find and understand it, this excludes updates or installs of these packages from this repository. It does not really make sense to me, however, as this would be the main packages you want to install from this repo or what am I missing? I haven't tried the effect of this option, but the official documentation and man page for Fedora Linux 38 / RHEL 9 don't reference this option at all.

Sources I found about this option: 1, 2, 3, 4

danifr commented 11 months ago

Yeah disregard my previous comment. My idea was to exclude them from the repo definition so they won't be accidentally upgraded during a regular OS update.

But it seems that we exclude them there, puppet cannot installed them either.

Not sure why it is part of the configuration example...

deric commented 11 months ago

@danifr You might be able to use something like yum versionlock kubelet. But I think the whole point of having minor version in repo url:

https://pkgs.k8s.io/core:/stable:/v1.28/rpm/

is that the regular update could upgrade only to next patch version. Which also not ideal because the upgrade should be applied to control-plane nodes first. That's probably the reason why k8s packages are excluded, and when performing manual upgrade you should do this:

yum list --showduplicates kubeadm --disableexcludes=kubernetes

treydock commented 9 months ago

I think the recommendation by Kubernetes to exclude the Kubernetes packages is counterintuitive when configuration management is involved. To use the excludes the only real option with Puppet is install_options on the Package resources to force passing the --disableexcludes=kubernetes flag on install. Generally if someone is doing yum updates they need to do yum update --disablerepo=kubernetes unless they do intend to update Kubernetes since Kubernetes upgrades, even patch releases, require many extra steps.

jadestorm commented 7 months ago

It looks like this has developed into a more serious issue now -- the google cloud paths are no longer providing content. I imagine they got cleaned up.

XSpielinbox commented 6 months ago

It looks like this has developed into a more serious issue now -- the google cloud paths are no longer providing content. I imagine they got cleaned up.

As announced back in August, the legacy repos have been frozen for month now and all Kubernetes version that where available through the old repos are long end of life now. As the end of February has passed the old repos are no longer available and if you are still running any version that was available in the old repos you should be looking into upgrading to a supported version as soon as possible.

[1] https://kubernetes.io/blog/2023/08/15/pkgs-k8s-io-introduction/ [2] https://kubernetes.io/blog/2023/08/31/legacy-package-repository-deprecation/ [3] https://kubernetes.io/releases/patch-releases/

jadestorm commented 6 months ago

Just wanted to add a couple of notes --

It is possible to pass in (via hiera or direct) some reasonable settings to get the latest release, as-is, to use pkgs.k8s.io -- we are actively using that right now because ...
I can't explain why but when I used your fork for this PR as-is @deric , I got the following dependency loop which frankly makes zero sense to me after looking through the code: Error: Found 1 dependency cycle: (Exec[Install cni network provider] => Exec[Install cni network provider])\nTry the '--graph' option and opening the resulting '.dot' file in OmniGraffle or GraphViz Error: Failed to apply catalog: One or more resource dependency cycles detected in graph

I could not track down specifically what I might have been doing to cause this, if anything, but I just wanted to mention it. It also did not occur on ALL of our hosts using this module -- so there's a good chance it's something we caused. None-the-less, it does not occur in 8.0.0 so I wanted to bring it up not as a problem with this PR, but as a note. I believe your fork only has work-since-8.0.0 plus pkgs changes.

I can share the hiera method of working around this issue with 8.0.0 if anyone needs it.

XSpielinbox commented 6 months ago

1. It is possible to pass in (via hiera or direct) some reasonable settings to get the latest release, as-is, to use pkgs.k8s.io -- we are actively using that right now because ...

@jadestorm from looking at the code something like setting

kubernetes::kubernetes_apt_location: "https://pkgs.k8s.io/core:/stable:/v${minor_version}/deb"
kubernetes::kubernetes_apt_release: '/'
kubernetes::kubernetes_version: 1.29.2
kubernetes::kubernetes_package_version: 1.29.2-1.1
kubernetes::kubernetes_apt_repos: ''
kubernetes::kubernetes_key_id: '<the new fingerprint>'
kubernetes::kubernetes_key_source: "https://pkgs.k8s.io/core:/stable:/v${minor_version}/deb/Release.key"

in the hiera should work (given that your containerd, systemd, etc. config is updated already) , but haven't tested it directly. ${minor_version} of course must be substituted either directly or by doing something like

$kubernetes_version = lookup('kubernetes::kubernetes_version')

$parts = split($kubernetes_version, '[.]')
$minor_version = "${parts[0]}.${parts[1]}"

puppetlabs / puppetlabs-kubernetes