search

opendevstack / ods-core

The core of OpenDevStack - infrastructure setup based on Atlassian tools, Jenkins, Nexus, SonarQube and shared images
Apache License 2.0
47 stars 34 forks source link

Preparing to upgrade to OpenShift Container Platform 4.9 [Kubernetes 1.22] APIs Deprecated #1053

Open stephenkehoe opened 2 years ago

stephenkehoe commented 2 years ago

The next OpenShift Container Platform release, currently planned to be OpenShift Container Platform 4.9, is expected to use Kubernetes 1.22. Kubernetes 1.22 removed a significant number of deprecated v1beta1 APIs.

OpenShift Container Platform 4.8.14 introduced a requirement that an administrator must provide a manual acknowledgment before the cluster can be upgraded from OpenShift Container Platform 4.8 to 4.9. This is to help prevent issues after upgrading to OpenShift Container Platform 4.9, where APIs that have been removed are still in use by workloads, tools, or other components running on or interacting with the cluster. Administrators must evaluate their cluster for any APIs in use that will be removed and migrate the affected components to use the appropriate new API version. After this is done, the administrator can provide the administrator acknowledgment.

All clusters require this administrator acknowledgment before they can be upgraded to OpenShift Container Platform 4.9.

Removed Kubernetes APIs

Kubernetes 1.22 removed the following deprecated v1beta1 APIs. If your cluster, or any idle workloads or tools use any of these APIs, you must migrate them to the appropriate new version before upgrading to OpenShift Container Platform 4.9.

Resource | API | Notable changes -- | -- | -- APIService | apiregistration.k8s.io/v1beta1 | No CertificateSigningRequest | certificates.k8s.io/v1beta1 | Yes ClusterRole | rbac.authorization.k8s.io/v1beta1 | No ClusterRoleBinding | rbac.authorization.k8s.io/v1beta1 | No CSIDriver | storage.k8s.io/v1beta1 | No CSINode | storage.k8s.io/v1beta1 | No CustomResourceDefinition | apiextensions.k8s.io/v1beta1 | Yes Ingress | extensions/v1beta1 | Yes Ingress | networking.k8s.io/v1beta1 | Yes IngressClass | networking.k8s.io/v1beta1 | No Lease | coordination.k8s.io/v1beta1 | No LocalSubjectAccessReview | authorization.k8s.io/v1beta1 | Yes MutatingWebhookConfiguration | admissionregistration.k8s.io/v1beta1 | Yes PriorityClass | scheduling.k8s.io/v1beta1 | No Role | rbac.authorization.k8s.io/v1beta1 | No RoleBinding | rbac.authorization.k8s.io/v1beta1 | No SelfSubjectAccessReview | authorization.k8s.io/v1beta1 | Yes StorageClass | storage.k8s.io/v1beta1 | No SubjectAccessReview | authorization.k8s.io/v1beta1 | Yes TokenReview | authentication.k8s.io/v1beta1 | No ValidatingWebhookConfiguration | admissionregistration.k8s.io/v1beta1 | Yes VolumeAttachment | storage.k8s.io/v1beta1 | No
stephenkehoe commented 2 years ago

https://kubernetes.io/blog/2021/07/14/upcoming-changes-in-kubernetes-1-22/

michaelsauter commented 2 years ago

@stephenkehoe Thanks a lot for this! One question: where did you get the above list? In the linked blog post, I do not see any mention of changes to e.g. RoleBinding. Is there another resource to read up on those changes?

The changes listed in the blog post sound safe to me, nothing comes to mind that affects what ODS is doing.

gerardcl commented 2 years ago

hi! I have some updates regarding this topic:

  1. See "Preparing to upgrade to OpenShift Container Platform 4.9" knowledge page from redhat here

  2. From (1), I have checked on our OSD4.8 cluster what k8s API resources might be affected, by running command oc get apirequestcounts, and filtering by existing REMOVEDINRELEASE we see:

$ oc get apirequestcount
NAME                                                                           REMOVEDINRELEASE   REQUESTSINCURRENTHOUR   REQUESTSINLAST24H
certificatesigningrequests.v1beta1.certificates.k8s.io                         1.22               0                       0
clusterrolebindings.v1beta1.rbac.authorization.k8s.io                          1.22               0                       0
customresourcedefinitions.v1beta1.apiextensions.k8s.io                         1.22               12                      292
flowschemas.v1alpha1.flowcontrol.apiserver.k8s.io                              1.21               0                       0
ingresses.v1beta1.extensions                                                   1.22               18                      437
ingresses.v1beta1.networking.k8s.io                                            1.22               0                       0
mutatingwebhookconfigurations.v1beta1.admissionregistration.k8s.io             1.22               0                       0
rolebindings.v1beta1.rbac.authorization.k8s.io                                 1.22               0                       0
roles.v1beta1.rbac.authorization.k8s.io                                        1.22               0                       0
validatingwebhookconfigurations.v1beta1.admissionregistration.k8s.io           1.22               0                       0
  1. Based on results in (2) we have checked what is doing requests to those used resources (we have checked two days in a row already):

  1. From this solution article by RedHat, we can be pretty sure that we are not affected by changes in ingresses.v1beta1.extensions, and we believe regarding customresourcedefinitions.v1beta1.apiextensions.k8s.io we are also in a good shape since we might get a newest/working openshift-pipelines-operator version.

  2. Nevertheless, we have an open ticket to double check such assumptions with RedHat, and once we have all fully verified we will be able to upgrade.

Finally, please also take into account we do use both ods-core and ods-pipeline ODS frameworks in that cluster (and some other operators). Hence we can use all the information provided here as a baseline for all clusters using ODS that require upgrading.

michaelsauter commented 2 years ago

Regarding OpenShift Pipelines:

See https://docs.openshift.com/container-platform/4.9/cicd/pipelines/op-release-notes.html#compatibility-support-matrix_op-release-notes. Since we are on 4.8, we use OpenShift Pipelines 1.5, which is based on Tekton 0.24.x. According to Tekton 0.27.3 is the first release with support for Kubernetes 1.22 (https://github.com/tektoncd/pipeline/releases/tag/v0.27.3). So the update to OpenShift Pipelines 1.6 should resolve the issue as it is based on 0.28.x.

cschweikert commented 2 years ago

Regarding Kubernetes API there might be relevant dependencies towards Kubernetes-related Jenkins plugins:

I suspect those plugins to be responsible for https://github.com/opendevstack/ods-core/issues/1123. Maybe this is also relevant here.

cschweikert commented 2 years ago

Jenkins plugins kubernetes of version 1.30.6+ requires Kubernetes version 1.19+. See also https://github.com/jenkinsci/kubernetes-plugin/releases/tag/kubernetes-1.30.6

What version of Kubernetes are we running on right now?

michaelsauter commented 2 years ago

@cschweikert the cluster you used is OpenShift 4.8, which uses Kubernetes 1.21. You can always check by clicking on the "question mark" icon in the header, and then selecting "About" from the menu.