search

cri-o / cri-o

Open Container Initiative-based implementation of Kubernetes Container Runtime Interface
https://cri-o.io
Apache License 2.0
5.24k stars 1.06k forks source link

Upgrade cri-o from 1.29 to 1.30 deletes and creates all running pods on ubuntu #8739

Open ogre0403 opened 2 weeks ago

ogre0403 commented 2 weeks ago

What happened?

I have a test 1.29 k8s cluster with 1.29 cri-o runtime. Because this is a test cluster, there is no any workload on it. I will upgrade the k8s cluster to 1.30, then I need to upgrade cri-o to 1.30 first.

Before I run the cri-o upgrade procedure, I have stopped cri-o service multiple times, the pods still running and not re-created. which is the expected behavior. Then https://github.com/cri-o/cri-o/issues/2479 should be not my case.

However, I notice that the existing pods (flannel and kube-proxy) will be recreated after the cri-o upgrade to new version and restart cri-o service.

I'm not sure the pod recreation is the expected behavior when cri-o upgrade, or I have some not correct configuration to make the pod recreated.

In addition to upgrade cri-o from 1.29 to 1.30, I also have tried downgrade cri-o from 1.30 to 1.29. Once the major version of cri-o has changed, the pod will be re-created.

Following is the commands I used to upgrade the cri-o.

Step for upgrade cri-o

CRIO_VERSION=v1.30

curl -fsSL https://pkgs.k8s.io/addons:/cri-o:/stable:/$CRIO_VERSION/deb/Release.key | gpg --dearmor -o /etc/apt/keyrings/cri-o-apt-keyring.gpg

echo "deb [signed-by=/etc/apt/keyrings/cri-o-apt-keyring.gpg] https://pkgs.k8s.io/addons:/cri-o:/stable:/$CRIO_VERSION/deb/ /" | tee /etc/apt/sources.list.d/cri-o.list

apt-get update
apt-cache madison cri-o

# systemctl stop crio
apt-get install cri-o

systemctl daemon-reload
systemctl restart crio

What did you expect to happen?

When I upgrade crio, the existing pod should not re-created

How can we reproduce it (as minimally and precisely as possible)?

  1. install k8s use kubeadm with cri-o runtime
  2. upgrade the cri-o on some node.
  3. When cri-o upgrade and restart new version cri-o, the running pod will re-created.

Anything else we need to know?

No response

CRI-O and Kubernetes version

old cri-o version ```console $ crio --version crio version 1.29.9 Version: 1.29.9 GitCommit: 933bdd28b54ead4dfbbb1b0cc5ff46e8942b08a1 GitCommitDate: 2024-10-01T19:08:39Z GitTreeState: clean BuildDate: 1970-01-01T00:00:00Z GoVersion: go1.21.7 Compiler: gc Platform: linux/amd64 Linkmode: static BuildTags: static netgo osusergo exclude_graphdriver_btrfs exclude_graphdriver_devicemapper seccomp apparmor selinux LDFlags: unknown SeccompEnabled: true AppArmorEnabled: true ``` new cri-o version ```console $ crio --version crio version 1.30.7 Version: 1.30.7 GitCommit: 2391edcb98c15a5557fa899a143501d2608dc044 GitCommitDate: 2024-11-01T10:19:56Z GitTreeState: clean BuildDate: 1970-01-01T00:00:00Z GoVersion: go1.22.0 Compiler: gc Platform: linux/amd64 Linkmode: static BuildTags: static netgo osusergo exclude_graphdriver_btrfs exclude_graphdriver_devicemapper seccomp apparmor selinux LDFlags: unknown SeccompEnabled: true AppArmorEnabled: true ``` ```console $ kubectl version --output=json { "clientVersion": { "major": "1", "minor": "29", "gitVersion": "v1.29.10", "gitCommit": "f0c1ea863533246b6d3fda3e6addb7c13c8a6359", "gitTreeState": "clean", "buildDate": "2024-10-22T20:35:55Z", "goVersion": "go1.22.8", "compiler": "gc", "platform": "linux/amd64" }, "kustomizeVersion": "v5.0.4-0.20230601165947-6ce0bf390ce3", "serverVersion": { "major": "1", "minor": "29", "gitVersion": "v1.29.10", "gitCommit": "f0c1ea863533246b6d3fda3e6addb7c13c8a6359", "gitTreeState": "clean", "buildDate": "2024-10-22T20:27:06Z", "goVersion": "go1.22.8", "compiler": "gc", "platform": "linux/amd64" } } ```

OS version

```console # On Linux: $ cat /etc/os-release PRETTY_NAME="Ubuntu 22.04.5 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.5 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy $ uname -a Linux k8s-03 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux ```

Additional environment details (AWS, VirtualBox, physical, etc.)

Virtual machine on Proxmox