Test all Operator bundles within an index for use of deprecated APIs in OCP 4.12/K8 1.25 & OCP 4.13/K8 1.26

[madorn]

Context

OCP 4.12 is scheduled for GA in Jan 2023.

The following API versions will be deprecated in OCP 4.12 / K8s 1.25:

Group	Version	Resource	How to upgrade
batch	v1beta1	cronjobs	use v1 instead of v1beta1. v1 has been available since k8s v1.21/Openshift 4.8. No notable changes
discovery.k8s.io	v1beta1	endpointslices	use v1 instead of v1beta1. v1 has been available since k8s v1.21/Openshift 4.8. (See the guide to check the notable changes)
events.k8s.io	v1beta1	events	use v1 instead of v1beta1. v1 has been available since k8s v1.19/Openshift 4.6. (See the guide to check the notable changes)
autoscaling	v2beta1	horizontalpodautoscalers	use v2 instead of v2beta1. v2 has been available since k8s v1.23/Openshift 4.10. (No notable changes)
policy	v1beta1	poddisruptionbudgets	use v1 instead of v1beta1. v1 has been available since k8s v1.21/Openshift 4.8. (See the guide to check the notable changes.)
policy	v1beta1	podsecuritypolicies	Migrated to 3rd-party admission webhooks.
node.k8s.io	v1beta1	runtimeclasses	use v1 instead of v1beta1. v1 has been available since k8s v1.20/Openshift 4.7. (No notable changes)

The following API versions will be deprecated in OCP 4.13 / K8s 1.26:

Group	Version	Resource	How to upgrade
autoscaling	v2beta2	horizontalpodautoscalers	use v2 instead of v2beta2. v2 has been available since k8s v1.23/Openshift 4.10(No notable changes)
flowcontrol.apiserver.k8s.io	v1beta1	flowschemas	use v1beta2 instead of v1beta1. v1beta2 has been available since k8s v1.23/Openshift 4.10(No notable changes)
flowcontrol.apiserver.k8s.io	v1beta1	prioritylevelconfigurations	use v1beta2 instead of v1beta1. v1beta2 has been available since  k8s v1.23/Openshift 4.10(No notable changes)

Goal

Automate the discovery of Operators utilizing deprecated APIs on a 4.11 cluster so that we can anticipate the required modifications for 4.12 compatibility.

How?

Camila Macedo has a thorough announcement covering all the methods for determining use of deprecated APIs.

It appears that the most comprehensive method for determining deprecated API use would be via ApiRequestCount.

On OpenShift, the ApiRequestCount resource has the ability to return the list of service accounts/users (10 by default) who have made requests to deprecated endpoints within the past hour. We would only need to cross reference an Operator's service account with the list of all users across all deprecated endpoints, i.e. ApiRequest.status.currentHour.byNode.byUser.username.

Upon investigating ApiRequestCount, it appears that the record of a service account/user's API call can take anywhere from one to ten minutes to appear within the apiRequestCount object.

We have yet to research the internals of how this time period is determined.

Existing tools

The existing audit tool does contain a deprecated API audit check for all bundles in a given index.

It's important to note that this check only uses an operator's ClusterServiceVersion.spec.install.permissions to determine its use of deprecated APIs. These are referenced against a current list of deprecated API groups.

This list lacks specific deprecated API versions since Roles/ClusterRoles are only required to specify the API group. The tool could only serve as a potential warning to those using a deprecated group/version.

[madorn]

After reviewing the use of ApiRequestCount API, it appears too unreliable due to the time it takes to populate with origin user account.

We were thinking we could tie into PR #334 and check controller logs.

A few observations while testing on 4.11/k8s 1.24

When watching soon to be deprecated resources via Owns, controller output will look similar to the following:

W1219 22:26:54.991614   90358 warnings.go:70] batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob
W1219 22:26:54.991545   90358 warnings.go:70] discovery.k8s.io/v1beta1 EndpointSlice is deprecated in v1.21+, unavailable in v1.25+; use discovery.k8s.io/v1 EndpointSlice
W1219 22:26:55.012682   90358 warnings.go:70] events.k8s.io/v1beta1 Event is deprecated in v1.22+, unavailable in v1.25+
W1219 22:26:54.938174   90358 warnings.go:70] autoscaling/v2beta1 HorizontalPodAutoscaler is deprecated in v1.22+, unavailable in v1.25+; use autoscaling/v2 HorizontalPodAutoscaler
W1219 22:53:31.741209   93315 warnings.go:70] autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
W1219 22:26:54.991662   90358 warnings.go:70] policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
W1219 22:26:54.938025   90358 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W1219 22:26:54.991440   90358 warnings.go:70] node.k8s.io/v1beta1 RuntimeClass is deprecated in v1.22+, unavailable in v1.25+
W1219 22:26:54.991042   90358 warnings.go:70] flowcontrol.apiserver.k8s.io/v1beta1 FlowSchema is deprecated in v1.23+, unavailable in v1.26+; use flowcontrol.apiserver.k8s.io/v1beta2 FlowSchema
W1219 22:26:54.991075   90358 warnings.go:70] flowcontrol.apiserver.k8s.io/v1beta1 PriorityLevelConfiguration is deprecated in v1.23+, unavailable in v1.26+; use flowcontrol.apiserver.k8s.io/v1beta2 PriorityLevelConfiguration

Afer initial controller startup, triggering the reconciler via CR creation and sending an API call to a soon to be deprecated endpoint will output following:

1.671511503243319e+09 INFO KubeAPIWarningLogger batch/v1beta1 CronJob is deprecated in v1.21+, unavailable in v1.25+; use batch/v1 CronJob

Note that (client.Reader).Get will not produce deprecation warnings. These warnings only appear on Create, Update, and Delete calls to these endpoints.

[madorn]

Just a note that the above was tested with operator-sdk version v1.25.3.

@yashoza19 has discovered that errors could appear differently depending on controller version - although they all should all return is deprecated in.