search

apache / cloudstack

Apache CloudStack is an opensource Infrastructure as a Service (IaaS) cloud computing platform
https://cloudstack.apache.org/
Apache License 2.0
2.06k stars 1.1k forks source link

Problem with self-generated kubernetes iso #7542

Closed ugenk closed 1 year ago

ugenk commented 1 year ago
ISSUE TYPE
 * Bug Report
COMPONENT NAME
Kubernetes
CLOUDSTACK VERSION
4.18.0.0
CONFIGURATION
N/A
OS / ENVIRONMENT
Ubuntu 22.04
SUMMARY

We've created kubernetes 1.27.2 iso with create-kubernetes-binaries-iso.sh and tried to update existing 1.26.0 cluster installed from ISO provided on your website. Upgrade fails with error in logs

STEPS TO REPRODUCE
/usr/share/cloudstack-common/scripts/util/create-kubernetes-binaries-iso.sh ./ 1.27.2 1.1.2 1.27.0 https://github.com/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s.yaml https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml
EXPECTED RESULTS
Create ISO that support latest version of Kubernetes
ACTUAL RESULTS
In /var/log/cloudstack/management/management-server.log on management node:

2023-05-22 17:37:43,644 ERROR [c.c.u.s.SshHelper] (API-Job-Executor-109:ctx-aec42508 job-1928 ctx-4cfc9215) (logid:d33e6f48) SSH execution of command sudo ./upgrade-kubernetes.sh4216874459906370299.sh 1.27.2 true false false has an error status code in return. Result output: Installing binaries from /mnt/k8sdisk/
unpacking docker.io/apache/cloudstack-kubernetes-autoscaler:latest (sha256:1c8a22c342daa5884f622f078be70ff913599aada0c4f859fe42ed28413afe98)...done
unpacking docker.io/apache/cloudstack-kubernetes-provider:v1.0.0 (sha256:8c2408eccd30eef24cfa72cfbdd912aa4261bc032406d2d078a85dd7246b368d)...done
unpacking registry.k8s.io/coredns/coredns:v1.10.1 (sha256:a0ead06651cf580044aeb0a0feba63591858fb2e43ade8c9dea45a6a89ae7e5e)...done
unpacking docker.io/kubernetesui/dashboard:v2.7.0 (sha256:2e500d29e9d5f4a086b908eb8dfe7ecac57d2ab09d65b24f588b1d449841ef93)...done
unpacking registry.k8s.io/etcd:3.5.7-0 (sha256:51eae8381dcb1078289fa7b4f3df2630cdc18d09fb56f8e56b41c40e191d6c83)...done
unpacking registry.k8s.io/kube-apiserver:v1.27.2 (sha256:94e48585629fde3c1d06c6ae8f62885d3052f12a1072ffd97611296525eff5b9)...done
unpacking registry.k8s.io/kube-controller-manager:v1.27.2 (sha256:b0990ef7c9ce9edd0f57355a7e4cb43a71e864bfd2cd55bc68e4998e00213b56)...done
unpacking registry.k8s.io/kube-proxy:v1.27.2 (sha256:1e4f13f5f5c215813fb9c9c6f56da1c0354363f2a69bd12732658f79d585864f)...done
unpacking registry.k8s.io/kube-scheduler:v1.27.2 (sha256:89e8c591cc58bd0e8651dddee3de290399b1ae5ad14779afe84779083fe05177)...done
unpacking docker.io/kubernetesui/metrics-scraper:v1.0.8 (sha256:76049887f07a0476dc93efc2d3569b9529bf982b22d29f356092ce206e98765c)...done
unpacking registry.k8s.io/pause:3.9 (sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097)...done
unpacking docker.io/weaveworks/weave-kube:latest (sha256:35827a9c549c095f0e9d1cf8b35d8f27ae2c76e31bc6f7f3c0bc95911d5accea)...done
unpacking docker.io/weaveworks/weave-npc:latest (sha256:062832fd25b5e9e16650e618f26bba1409a7b3bf2c3903e1b369d788abc63aef)...done
registry.k8s.io/pause:3.9

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now

2023-05-22 17:37:43,645 ERROR [c.c.k.c.a.KubernetesClusterActionWorker] (API-Job-Executor-109:ctx-aec42508 job-1928 ctx-4cfc9215) (logid:d33e6f48) Failed to upgrade Kubernetes cluster : testupdates, unable to upgrade Kubernetes node on VM : testupdates-control-18841cd6ee3
kiranchavala commented 1 year ago

Hi @ugenk

I was able to successfully create a k8s cluster of the latest k8s release 1.27.2

1.27.2 kubernetes version, see https://github.com/kubernetes/kubernetes/releases 1.3.0 CNI version, see https://github.com/containernetworking/plugins/releases 1.27.0 cri-tools version, see https://github.com/kubernetes-sigs/cri-tools/releases 1.11 weave addon for kubernetes, see https://github.com/weaveworks/weave/tree/master/prog/weave-kube 2.7.0 kubernetes dashboard version, see https://github.com/kubernetes/dashboard/release

./create-kubernetes-binaries-iso.sh ./ 1.27.2 1.3.0 1.27.0 https://raw.githubusercontent.com/weaveworks/weave/master/prog/weave-kube/weave-daemonset-k8s-1.11.yaml https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml setup-1.27.2

NAME                     STATUS   ROLES           AGE    VERSION
k4-control-188471a62c6   Ready    control-plane   115s   v1.27.2
k4-node-188471ab5e9      Ready    <none>          101s   v1.27.2
kiranchavala commented 1 year ago

@ugenk was also able to upgrade the k8s cluster from 1.26.0 to 1.27.2

ugenk commented 1 year ago

Yeah, just tested - it works. Looks like i had a problem with CNI version. Would be nice, if you will describe that better in docs, and will update provided ISOs to latest versions (for now mostly you have x.y.0 versions)

kiranchavala commented 1 year ago

Closing the issue as its resolved